--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-reading-querying-uniform-way.xml Mon Feb 21 00:43:11 2022 +0100
@@ -0,0 +1,170 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way</nadpis>
+ <perex>run XPath queries and turn data from various sources to relations</perex>
+ <m:pořadí-příkladu>04600</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>
+ Data come in different shapes and formats.
+ We can distinguish several main logical models:
+ relational,
+ tree
+ and graph
+ (a tree is an undirected graph with no cycles)
+ Arbitrary trees or even graphs are more flexible, but they are also harder to comprehend and work with.
+ Relational model is somehow limited and easier to grasp, however still flexible enough to describe almost anything.
+ (actually it can describe anything, it is just a question of how nice and native it should look)
+ Unsurprisingly, <m:name/> are build around the relational model.
+ However, sometimes we have to interact with the tree/graph world and deal with data that have other than relational shape.
+ So we need to bridge the gap between trees/graphs and relations.
+ </p>
+
+ <p>
+ While we have just few logical models, there is abundance of serialization formats i.e. mappings of given logical model to a sequence of octets (bytes).
+ Relations might be serialized as CSV, ODS, tables in a database, Recfiles etc.
+ Trees might be serialized as XML, YAML, ASN.1, CBOR, JSON etc.
+ </p>
+
+ <p>
+ Why reinvent the wheel and repeat the same work for each format?
+ </p>
+
+ <p>
+ We already have reusable code for relational data – this is given by the design of <m:name/>, because it separates: <em>inputs</em>, <em>transformations</em> and <em>outputs</em>.
+ Once the data (e.g. CSV) passes through the input filter, it becomes relational data and can be processed in a uniform way by any transformation(s) or output filter.
+ </p>
+
+ <p>
+ But what about the tree data? We have created a set of tools (input filters) that support various serialization formats, in <m:a href="release-v0.18">v0.18</m:a>:
+ </p>
+
+ <ul>
+ <li>XML: <code>relpipe-in-xmltable</code></li>
+ <li>ASN.1: <code>relpipe-in-asn1table</code></li>
+ <li>CBOR: <code>relpipe-in-cbortable</code></li>
+ <li>HTML: <code>relpipe-in-htmltable</code></li>
+ <li>INI: <code>relpipe-in-initable</code></li>
+ <li>MIME: <code>relpipe-in-mimetable</code></li>
+ <li>YAML: <code>relpipe-in-yamltable</code></li>
+ </ul>
+
+ <p>
+ These tools follow the same design principle and offer the same user interface.
+ So once the user learns one tool, he can use this knowledge also while working with other formats.
+ The principle is:
+ </p>
+
+ <ul>
+ <li>We are converting the tree structure to one or more relations.</li>
+ <li>For each relation, define the expression that selects record nodes from the tree.</li>
+ <li>For each attribute, define the expression (relative to the record node) that selects the attribute value.</li>
+ <li>If anything can not (or is not desired to) be mapped to relations, keep is as a tree, so we can process it later – these (sub)trees might be embedded in normal records or reside in a separate relation.</li>
+ <li>We may do a full (lossless) conversion, but we may also extract just a single value from the whole tree (generate a single relation with single record and single attribute). Or anything in between. Anyway, the tool and the logic used is still the same.</li>
+ </ul>
+
+ <p>
+ This is nothing new – and experienced SQL users should already know where the inspiration comes from:
+ the <code>XMLTable()</code> SQL function that converts XML tree to a result set (relation).
+ We just implemented the same functionality as a separate CLI tool, without dependency on any SQL engine and with support for not only XML but also for alternative serialization formats.
+ And for all of them, we use the same query language: XPath.
+ </p>
+
+ <p>
+ Despite this sounds so <i>XML-ish</i>, we do not translate the alternative formats to the XML markup. There is no <i>text full of angle brackets and ampersands</i> in the middle of the process.
+ In our case, we should see XML not as a markup text (meta)format, but rather as an in-memory model – a generic tree of node objects stored in the RAM that allows us doing various tree operations (queries, modifications).
+ </p>
+
+
+ <h2 id="yamlToRelations">Converting a YAML tree to a set of relations</h2>
+
+ <p>
+ Flat key-value lists become sooner or later insufficient for software configuration and it is necessary to somehow manage trees of configuration items (or relations, of course).
+ YAML is quite good tree-serialization format.
+ It is used e.g. for configuring Java Spring applications or for Netplan network configuration in the Ubuntu GNU/Linux distribution:
+ </p>
+
+ <m:pre jazyk="yaml" src="examples/netplan-1.yaml"/>
+
+ <p>We can use following command to convert the tree to a set of relations:</p>
+
+ <m:pre jazyk="bash" src="examples/netplan-1.sh"/>
+
+ <p>
+ So we can do a full relational conversion of the original tree structure or extract just few desired values (e.g. the gateway IP address).
+ We can also pipe a relation to a shell loop and execute some command for each record (e.g. DNS server or IP address).
+ </p>
+
+ <m:img src="img/wmaker-yaml-xml-tabular-1.png"/>
+
+ <p>
+ n.b. YAML is considered to be a superset of JSON, thus tools that can read YAML, can also read JSON.
+ In current version (v0.18) of <m:name/> the <code>relpipe-in-json</code> and <code>relpipe-in-jsontable</code> are just symbolic links to their YAML counterparts.
+ </p>
+
+ <p>
+ There is also similar example: <m:a href="examples-in-xmltable-libvirt">Reading Libvirt XML files using XMLTable</m:a>
+ where we build relations from a XML tree.
+ The principles are the same for all input formats.
+ </p>
+
+ <h2 id="htmlTagSoup">Dealing with the HTML tagsoup</h2>
+
+ <p>
+ With <code>relpipe-in-htmltable</code> we can extract structured information from poor HTML pages.
+ And unlike <code>relpipe-in-xmltable</code>, this tool does not require valid XML/XHTML, so it is good for the dirty work.
+ Processing such invalid data is always bit unreliable, but still better than nothing.
+ </p>
+
+ <m:pre jazyk="bash" src="examples/html-tagsoup-1.sh"/>
+
+ <p>Although Mr. Ryszczyks is unable to create a valid document, this script will print:</p>
+
+ <m:pre jazyk="text" src="examples/html-tagsoup-1.txt"/>
+
+ <p>
+ And thanks to the terminal autodetection in the <code>format_result()</code> function,
+ we can even pipe the result of this script to any <code>relpipe-tr-*</code> or <code>relpipe-out-*</code>
+ and get machine-readable data instead of the ANSI-colored tables –
+ so we can do some further processing or conversion to a different format (XHTML, GUI, ODS, Recfile etc.).
+ </p>
+
+ <h2 id="the2xmlTool">The <code>2xml</code> helper script: <code>yaml2xml</code>, <code>json2xml</code>, <code>asn12xml</code>, <code>mime2xml</code> etc.</h2>
+
+ <p>
+ Mapping from the original syntax to the tree structure is usually quite intuitive and straightforward.
+ However, sometimes it is useful to see the XML serialization of this in-memory model.
+ In the <code>relpipe-in-xmltable.cpp</code> repository we have a helper script called <code>
+ <a href="http://hg.globalcode.info/relpipe/relpipe-in-xmltable.cpp/file/tip/examples/2xml.sh">2xml</a>
+ </code>
+ – this script is not intended to be called directly – instead the user should create a symlink e.g. <code>ini2xml</code>, <code>yaml2xml</code>, <code>asn12xml</code> etc.
+ The <code>2xml</code> script choses the right input filter according to the symlink name and uses it for conversion from the source tree-serialization format to the XML tree-serialization format.
+ </p>
+
+ <p>
+ If we want to do the same thing without the helper script, it is quite simple.
+ We use appropriate <code>relpipe-in-*table</code> tool and extract a single relation with single attribute and single record.
+ The <code>--records</code> expression is <code>'/'</code> i.e. the root node.
+ The <code>--attribute</code> expression is <code>'.'</code> i.e. still the root node.
+ And then we just add the <code>--mode raw-xml</code> to this attribute, so we get the XML serialization of given node (root) instead of the text content.
+ </p>
+
+ <p>
+ In addition to this, the <code>2xml</code> script does also formatting/indentation and syntax highlighting,
+ if given tools (<code>xmllint</code> and <code>pygmentize</code>) are available and the STDOUT is a terminal.
+ </p>
+
+ <p>
+ This script is useful when writing the expressions for <code>relpipe-in-*table</code>,
+ but also as a pipeline filter that allows us to use the whole XML ecosystem also for other formats.
+ We can read YAML, JSON, INI, MIME or even some binary formats etc. and apply a XSLT transformation on such data and generate e.g. some XHTML report or a DocBook document,
+ or validate such structures using XSD or Relax NG schema or we can process such data using XQuery functional language.
+ </p>
+
+
+ </text>
+
+</stránka>