relpipe-data/examples-asn1-x509.xml
author František Kučera <franta-hg@frantovo.cz>
Mon, 21 Feb 2022 00:43:11 +0100
branchv_0
changeset 329 5bc2bb8b7946
permissions -rw-r--r--
Release v0.18
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
329
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     1
<stránka
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     2
	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     3
	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     4
	
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     5
	<nadpis>Exploring content of X.509 certificates</nadpis>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     6
	<perex>open and query common SSL/TLS certificates or other ASN.1 data encoded in BER/DER/CER</perex>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     7
	<m:pořadí-příkladu>05000</m:pořadí-příkladu>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     8
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     9
	<text xmlns="http://www.w3.org/1999/xhtml">
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    10
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    11
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    12
			X.509 certificates and keys used for SSL/TLS (HTTPS, POP3S, IMAPS etc.) are usually distributed as files either with <code>.pem</code> or <code>.der</code> extension.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    13
			Or bundled together in a PKCS#12 container as a <code>.p12</code> file.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    14
			The „text“ PEM format is often considered more „accessible“ or „friendly“ than the binary DER.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    15
			However PEM is just Base64 encoded DER original and is actually less legible to the naked eye than DER,
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    16
			because in DER we can spot at least some strings like common and domain names or validity/expiration dates or recognize certain data structures in a HEX editor.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    17
			Base64 just obfuscates everything. PEM can be easily copied through clipboard, which is probably the only advantage of this format (but it can also more likely leak).
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    18
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    19
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    20
		<!-- Indeed, Base64 is evil. Use hexadecimal encoding where ASCII representaion of binary data is necessary. -->
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    21
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    22
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    23
			So our first step is to get rid of the annoying Base64 pseudo-plain-text encoding – we use one of these commands:
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    24
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    25
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    26
		<m:pre jazyk="text"><![CDATA[cat certificate.pem | grep -v ^--- | base64 -d              > certificate.der
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    27
cat certificate.pem | openssl x509 -inform PEM -outform DER > certificate.der]]></m:pre>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    28
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    29
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    30
			Telco veterans could now start reading the DER file with <code>hd</code> or <code>xxd</code>, jumping over the offsets and traversing the sequences and sets…
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    31
			However most people would appreciate some software that helps them parsing the ASN.1 BER encoding (the superset of DER and CER).
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    32
			Such software is e.g. Wireshark or dumpasn1. These programs are good for ad-hoc inspection or quick check.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    33
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    34
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    35
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    36
			In <m:name/> <m:a href="release-v0.18">v0.18</m:a> we have (early and bit raw) support for ASN.1 BER encoding and thus we can get the structured data in a machine-readable form
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    37
			– which is good for further processing, conversion to other formats or use in scripts.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    38
			Because the ASN.1 data model is not relational – actually it is a tree – this format is supported in the <code>relpipe-in-asn1table</code>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    39
			command that is modelled after the well-known <code>XMLTable()</code> database function that allows translating arbitrary tree structures to relations using the XPath expressions.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    40
			So in <code>relpipe-in-asn1table</code> we can write XPath expressions to query the ASN.1 tree data structures and extract relations, records and attributes
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    41
			from X.509 certificates, keys or other cryptographic artifacts, LDAP or SNMP packets or any other ASN.1 BER data.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    42
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    43
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    44
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    45
			But how do we know what XPath expressions should we run?
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    46
			It is useful to see the XML representation of whole source data.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    47
			There is a simple trick to do this – use <code>"/"</code> as the XPath for selecting records (is always selects the single record, single node – the root)
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    48
			and use <code>"."</code> as the XPath to select a single attribute (it always select the root element)
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    49
			and add <code>--mode raw-xml</code>, so we get the raw XML source instead of the text content of given elements.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    50
			We do not have to write this routine by hand – just create a symlink to the example script:
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    51
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    52
		<m:pre jazyk="bash"><![CDATA[ln -s …/relpipe-in-xmltable.cpp/examples/2xml.sh asn12xml # in ~/bin or somewhere]]></m:pre>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    53
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    54
			This example is generic and works also for other formats supported by the <code>relpipe-in-*table</code> commands.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    55
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    56
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    57
		<p> 
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    58
			Then we can analyze X.509 DER certificates stored on our disk or we can fetch some from live servers.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    59
			The <code>openssl</code> command helps us with that:
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    60
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    61
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    62
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    63
		<m:pre jazyk="bash"><![CDATA[fetch_x509_certificate() {
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    64
	echo \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    65
		| openssl s_client -connect $1:${2:-443} 2>/dev/null \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    66
		| openssl x509 -inform PEM -outform DER;
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    67
}]]></m:pre>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    68
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    69
		<p>Now put both commands together in a pipeline:</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    70
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    71
		<m:pre jazyk="bash"><![CDATA[fetch_x509_certificate "gnu.org" | asn12xml # HTTPS port (443) is used as default]]></m:pre>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    72
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    73
		<p>and get this XML representation of the ASN.1 X.509 tree:</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    74
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    75
		<m:pre jazyk="xml" src="examples/x509-gnu.org.xml"/>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    76
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    77
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    78
		<p>Once we know the structure, we can easily hack together a function that extracts parts of the tree as relations:</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    79
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    80
		<m:pre jazyk="bash"><![CDATA[parse_x509_certificate() {
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    81
	relpipe-in-asn1table \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    82
		--relation 'validity' \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    83
			--records '//sequence[date-time][1]' \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    84
			--attribute 'from'  string 'date-time[1]' \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    85
			--attribute 'to'    string 'date-time[2]' \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    86
		--relation 'alternative_name' \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    87
			--records '//sequence[oid="2.5.29.17"][1]/encapsulated/sequence/specific' \
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    88
			--attribute 'name'  string '.';
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    89
}]]></m:pre>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    90
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    91
		<p>Everything put together:</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    92
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    93
		<m:pre jazyk="bash"><![CDATA[fetch_x509_certificate "gnu.org" | parse_x509_certificate | relpipe-out-tabular]]></m:pre>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    94
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    95
		<p>will print:</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    96
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    97
		<m:pre jazyk="text" src="examples/x509-gnu.org.txt"/>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    98
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    99
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   100
			The function above is just a „hello world“ example.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   101
			Please note that the XPath expressions need to be carefully crafted with respect to the given format in order to match exactly what we want.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   102
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   103
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   104
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   105
			Instead of printing a table, we can use the <code>relpipe-out-nullbyte</code> tool + the <code>read_nullbyte</code> function
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   106
			and shell loop over the records (alternative names) and e.g. <code>ping</code> each domain or fetch given root web page using <code>wget</code> or <code>curl</code>.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   107
			We can also write a simple script that checks the validity of our own certificates and notifies us in advance when some of them are going to expire.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   108
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   109
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   110
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   111
			Later versions of <code>relpipe-in-asn1table</code> will probably support OID names, so it will not be necessary to use the numeric object identifiers.
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   112
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   113
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   114
		<p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   115
			n.b. there is also the <code>relpipe-in-asn1</code> – this tool reads data generated by its counterpart, the <code>relpipe-out-asn1</code> (or other ASN.1 BER capable software)
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   116
			i.e. it is not as universal as <code>relpipe-in-asn1table</code>, it has simpler interface, needs no configuration and expects certain ASN.1 structures (relations serialized in BER format).
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   117
		</p>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   118
		
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   119
	</text>
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   120
5bc2bb8b7946 Release v0.18
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   121
</stránka>