<stránka
xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
<nadpis>Release v0.18</nadpis>
<perex>new public release of Relational pipes</perex>
<m:release>v0.18</m:release>
<text xmlns="http://www.w3.org/1999/xhtml">
<p>
We are pleased to introduce you the new development version of <m:name/>.
This release brings several new tools of each type (inputs, outputs, transformations and streamlet) and some smaller improvements.
</p>
<ul>
<li>
New <strong>input tools</strong> for reading various formats:
read <strong>ASN.1</strong>, <strong>YAML</strong> and <strong>JSON</strong>; see details below
</li>
<li>
New <strong>XMLTable-like input tools</strong> for converting arbitrary tree structures to relations:
read <strong>ASN.1</strong>, <strong>INI</strong>, <strong>MIME</strong>, <strong>YAML</strong>, <strong>JSON</strong>, <strong>CBOR</strong> and <strong>HTML</strong>; see details below
</li>
<li>
New <strong>input tool and streamlet</strong> for reading <strong>barcodes</strong> and <strong>QR codes</strong>: see details below
</li>
<li>
New <strong>input and output tool</strong> for the <strong>INI</strong> format: see details below
</li>
<li>
New <strong>input and output tool</strong> for interacting with <strong>X11</strong>: see details below
</li>
<li>
New <strong>XPath transformation</strong>: see details below
</li>
<li>
<strong>Data types</strong> support on <strong>CSV</strong> input and output: strings, integers and booleans; see details below
</li>
<li>
The <strong>grep</strong>, <strong>cut</strong> and <strong>sed</strong> transformations have new CLI interface: see details below
</li>
</ul>
<p>
See the (<a href="#newExamples">new</a>) <m:a href="examples">examples</m:a> and (<a href="#newScreenshots">new</a>) <m:a href="screenshots">screenshots</m:a> pages for details.
</p>
<p>
Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change.
Any suggestions, ideas and bug reports are welcome in our <m:a href="contact">mail box</m:a>.
</p>
<h2 id="standardInputModules">New standard input modules</h2>
<p>
These tools allows serialization of arbitrary relational data
and then reading resulting file or stream back to the relational data.
An of course, we can read data created or generated somewhere else.
Since this release, we can newly serialize our relations to YAML and ASN.1, store, edit or transfer these files or streams and then deserialize them and pipe them through relational transformations or outputs.
Formats like XML or Recfiles were already supported in previous releases.
Full lossless round-trip is now possible with: XML, YAML, ASN.1 and Recfile formats.
Partial lossless round-trip is possible with CSV – these input and output tools are limited to a single relation.
Lossy round-trip is possible with <code>relpipe-out-nullbyte</code> and <code>relpipe-in-cli</code> –
the <i>nullbyte</i> stream may contain only a single relation and its data types need to be passed as CLI arguments to the input tool.
Other formats (e.g. JACK, INI or X11) may also have corresponding inputs and outputs, but they produce and consume some specific structures – they are not universal and usable with arbitrary relational data.
</p>
<h3 id="YAML-JSON">YAML and JSON</h3>
<p>
YAML is a text format for serializing tree structures into text (like XML).
It can be sometimes tricky, but basic scenarios are intuitive and straightforward.
We can use YAML to describe relational data and load them using the <code>relpipe-in-yaml</code> input filter
and we can produce YAML files using the <code>relpipe-out-yaml</code> output filter.
</p>
<p>More details in the example: <m:a href="examples-yaml">Reading and writing YAML</m:a></p>
<h3>ASN.1</h3>
<p>
There are actually more ASN.1 applications around us than most people realize.
ASN.1 is almost everywhere – in cryptography (X.509 / TLS / HTTP, S/MIME etc.), LDAP, payment cards, telecommunications (including GSM, GPRS, EDGE, UMTS, LTE, 5G), Kerberos authentication etc.
So it is very useful to be able to access these data.
Because ASN.1 is abstract, there are several ways how such data can be serialized into a stream of octets (bytes).
Most common way is BER (Basic Encoding Rules), so we support BER (and thus its variants DER and CER) in the first version of our tools.
Later other encoding might be supported and we could also parse the ASN.1 syntax (the schema language).
</p>
<p>
ASN.1 BER output was added several releases ago.
Now we have also ASN.1 BER input (<code>relpipe-in-asn1</code>) so we can read data in this format generated earlier.
Of course, BER data can be generated or read also by other ASN.1 capable software.
However, maybe more interesting than <code>relpipe-in-asn1</code> is <code>relpipe-in-asn1table</code> (see <a href="#XMLTable">below</a>) that reads arbitrary ASN.1 BER data (not only serialized relational data).
</p>
<h2 id="INI">Reading and writing INI</h2>
<p>
INI is very common simple text format used for configuration files.
It extends classic <i>key=value</i> config files and adds one more level – sections – for structuring.
In this release we got input and output filter for INI and similar formats (Java .properties, MANIFEST.MF, key=value configs).
Because comments and whitespace are also supported, we can do (almost) loss-less conversions and transformations (e.g. change value of an entry).
</p>
<p>More details in the example: <m:a href="examples-ini">Reading and writing INI and unix configs</m:a></p>
<h2>Reading barcodes and QR codes</h2>
<p>
Either ubiquitous 1D barcodes (procudct labels, ISBN in form of EAN-13 etc.) or 2D barcodes (QR containing hyperlinks, vCards etc.)
encode numbers, texts or other data into images that can be printed and scanned or photographed later.
Now we can interact with this technology – read barcodes using the
<code>relpipe-in-barcode</code> tool or <code>barcode-reader</code> streamlet.
</p>
<p>More details in the example: <m:a href="examples-barcode">Reading barcodes and QR</m:a></p>
<h2 id="XMLTable">New XMLTable-like input modules: ASN.1, INI, MIME, YAML, JSON, CBOR, HTML</h2>
<p>
More details in the examples:
</p>
<ul>
<li><m:a href="examples-reading-querying-uniform-way">Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way</m:a></li>
<li><m:a href="examples-asn1-x509">Exploring content of X.509 certificates</m:a></li>
</ul>
<h2 id="X11">X11 input and output modules</h2>
<p>
The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland).
This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice.
Since this release of <m:name/> we can interact with this wonderful technology through the <code>relpipe-in-x11</code> and <code>relpipe-out-x11</code> tools.
In the following example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements).
</p>
<p>More details in the example: <m:a href="examples-x11-basics">Exploring X11 windows and devices and emulating mouse movements and keystrokes</m:a></p>
<h2 id="XPath">XPath transformation</h2>
<p>
We got a new powerful language for filtering and transformations: XPath.
It is now part of the toolset consisting of SQL, AWK, Scheme and others.
However XPath is originally a language designed for XML, in <m:name/> we can use it for relational data coming from various sources, not only XML,
and also for data that violates the rules of normal forms.
We can process quite complex tree structures entangled in records but we can also write simple and intuitive expressions like <code>x = "a" or y = 123</code>.
</p>
<p>More details in the example: <m:a href="examples-xpath-filtering-transforming">Filtering and transforming relational data with XPath</m:a></p>
<h2 id="csvDataTypes">Data types in CSV + generic relpipe-tr-infertypes tool</h2>
<p>
CSV input and output filters now support data types i.e. CSV can now carry not only text strings, but also booleans and integers (or more types in future releases).
We also have a new tool <code>relpipe-tr-infertypes</code> that can automatically recognize the types in data that came from CSV or any other source.
</p>
<p>More details in the example: <m:a href="examples-csv-data-types">CSV and data types</m:a></p>
<h2>Feature overview</h2>
<h3>Data types</h3>
<ul>
<li m:since="v0.8">boolean</li>
<li m:since="v0.15">variable-length signed integer (SLEB128)</li>
<li m:since="v0.8">string in UTF-8</li>
</ul>
<h3>Inputs</h3>
<ul>
<li m:since="v0.11">Recfile</li>
<li m:since="v0.9">XML</li>
<li m:since="v0.13">XMLTable</li>
<li m:since="v0.9">CSV</li>
<li m:since="v0.9">file system</li>
<li m:since="v0.8">CLI</li>
<li m:since="v0.8">fstab</li>
<li m:since="v0.14">SQL script</li>
<li m:since="v0.16">JACK</li>
<li m:since="v0.18">barcode</li>
<li m:since="v0.18">X11</li>
<li m:since="v0.18">ASN.1 BER</li>
<li m:since="v0.18">ASN.1 BER Table</li>
<li m:since="v0.18">INI</li>
<li m:since="v0.18">INITable</li>
<li m:since="v0.18">MIMETable</li>
<li m:since="v0.18">YAML</li>
<li m:since="v0.18">YAMLTable</li>
<li m:since="v0.18">JSON</li>
<li m:since="v0.18">JSONTable</li>
<li m:since="v0.18">CBORTable</li>
<li m:since="v0.18">HTMLTable</li>
</ul>
<h3>Transformations</h3>
<ul>
<li m:since="v0.13">sql: filtering and transformations using the SQL language</li>
<li m:since="v0.18">xpath: filtering and transformations using the XPath language</li>
<li m:since="v0.12">awk: filtering and transformations using the classic AWK tool and language</li>
<li m:since="v0.10">scheme: filtering and transformations defined in the Scheme language using GNU Guile</li>
<li m:since="v0.8">grep: regular expression filter, removes unwanted records from the relation</li>
<li m:since="v0.8">cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)</li>
<li m:since="v0.8">sed: regular expression replacer</li>
<li m:since="v0.8">validator: just a pass-through filter that crashes on invalid data</li>
<li m:since="v0.18">infertypes: derive data types from attribute values</li>
<li m:since="v0.8">python: highly experimental</li>
</ul>
<h3>Streamlets</h3>
<ul>
<li m:since="v0.15">xpath (example, unstable)</li>
<li m:since="v0.15">hash (example, unstable)</li>
<li m:since="v0.15">jar_info (example, unstable)</li>
<li m:since="v0.15">mime_type (example, unstable)</li>
<li m:since="v0.15">exiftool (example, unstable)</li>
<li m:since="v0.15">pid (example, unstable)</li>
<li m:since="v0.15">cloc (example, unstable)</li>
<li m:since="v0.15">exiv2 (example, unstable)</li>
<li m:since="v0.15">inode (example, unstable)</li>
<li m:since="v0.15">lines_count (example, unstable)</li>
<li m:since="v0.15">pdftotext (example, unstable)</li>
<li m:since="v0.15">pdfinfo (example, unstable)</li>
<li m:since="v0.15">tesseract (example, unstable)</li>
<li m:since="v0.18">barcode (example, unstable)</li>
</ul>
<h3>Outputs</h3>
<ul>
<li m:since="v0.11">ASN.1 BER</li>
<li m:since="v0.11">Recfile</li>
<li m:since="v0.9">CSV</li>
<li m:since="v0.8">tabular</li>
<li m:since="v0.8">XML</li>
<li m:since="v0.8">nullbyte</li>
<li m:since="v0.8">GUI in Qt</li>
<li m:since="v0.8">ODS (LibreOffice)</li>
<li m:since="v0.17">JACK</li>
<li m:since="v0.18">X11</li>
<li m:since="v0.18">INI</li>
<li m:since="v0.18">YAML</li>
</ul>
<h2 id="newExamples">New examples</h2>
<ul>
<li><m:a href="examples-reading-querying-uniform-way">Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way</m:a></li>
<li><m:a href="examples-xpath-filtering-transforming">Filtering and transforming relational data with XPath</m:a></li>
<li><m:a href="examples-csv-data-types">CSV and data types</m:a></li>
<li><m:a href="examples-x11-basics">Exploring X11 windows and devices and emulating mouse movements and keystrokes</m:a></li>
<li><m:a href="examples-asn1-x509">Exploring content of X.509 certificates</m:a></li>
<li><m:a href="examples-csv-sql-join">Running SQL JOINs on multiple CSV files</m:a></li>
<li><m:a href="examples-yaml">Reading and writing YAML</m:a></li>
<li><m:a href="examples-ini">Reading and writing INI and unix configs</m:a></li>
<li><m:a href="examples-barcode">Reading barcodes and QR</m:a></li>
<!--
<li><m:a href="examples-x11-kvm">Software replacement of KVM switch</m:a></li>
-->
</ul>
<h2 id="newScreenshots">New screenshots</h2>
<m:img src="img/wmaker-yaml-xml-tabular-1.png"/>
<h2 id="incompatibleChanges">Backward incompatible changes</h2>
<p>
In <code>relpipe-tr-sql</code> and <code>relpipe-in-sql</code> the <code>--list-data-sources</code> option has now boolean parameter
i.e. it is not the presence of the option but the <code>true</code> or <code>false</code> value what matters.
We can also list the data sources while we simultaneously run some SQL.
</p>
<p>
The <code>relpipe-tr-grep</code>, <code>relpipe-tr-cut</code> and <code>relpipe-tr-sed</code>
were the last tools with the obsolete positional CLI interface.
In this release, they were updated and now share the same style of CLI arguments as other tools.
Besides consistent and predictable user interface, these tools are now more powerful because they can transforma more relations at once.
</p>
<p>Instead of e.g.</p>
<pre>relpipe-tr-grep 'a' 'b' 'c'</pre>
<p>we now write:</p>
<pre>relpipe-tr-grep --relation 'a' --attribute 'b' --value 'c'</pre>
<p>Bash completion helps us while writing such commands and result is much more readable than original cryptic version.</p>
<p>See updated examples:</p>
<ul>
<li><m:a href="examples-rename-vg-fstab">Renaming VG in /etc/fstab using relpipe-tr-sed</m:a></li>
<li><m:a href="examples-rename-groups-backreferences">Using relpipe-tr-sed with groups and backreferences</m:a></li>
<li><m:a href="examples-grep-fstab">Filtering /etc/fstab using relpipe-tr-grep</m:a></li>
<li><m:a href="examples-grep-cut-fstab">Doing projection and restriction using cut and grep</m:a></li>
<li><m:a href="examples-apt">Reading apt (Debian package system) results</m:a></li>
<li><m:a href="examples-in-xmltable-ssm-gui">Generating statistics and charts using XMLTable</m:a></li>
<li><m:a href="streamlets-preview">Streamlets preview</m:a></li>
<li><m:a href="examples-csv-sql-join">Running SQL JOINs on multiple CSV files</m:a></li>
<li><m:a href="examples-runnable-jars">Finding runnable JARs</m:a></li>
</ul>
<h2 id="installation">Installation</h2>
<p>
Instalation was tested on Debian GNU/Linux 10.2.
The process should be similar on other distributions.
</p>
<m:pre src="examples/release-v0.18.sh" jazyk="bash" odkaz="ano"/>
<p>
<m:name/> are modular thus you can download and install only parts you need (the libraries are needed always).
Tools <code>out-gui.qt</code> and <code>tr-python</code> require additional libraries and are not built by default.
</p>
</text>
</stránka>