<stránka
xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
<nadpis>Release v0.15</nadpis>
<perex>new public release of Relational pipes</perex>
<m:release>v0.15</m:release>
<text xmlns="http://www.w3.org/1999/xhtml">
<p>
We are pleased to introduce you the new development version of <m:name/>.
This release brings two big new features: streamlets and parallel processing + several smaller improvements.
</p>
<ul>
<li>
<strong>SLEB128</strong>: variable-length integers are now signed (i.e. can be even negative!) and encoded as SLEB128</li>
<li>
<strong>streamlets in relpipe-in-filesystem</strong>: see details below</li>
<li>
<strong>parallel processing in relpipe-in-filesystem</strong>: see details below</li>
<li>
<strong>multiple modes in relpipe-in-xmltable</strong>: see details below</li>
<li>
<strong>XInclude in relpipe-in-xmltable</strong>: use <code>--xinclude true</code> to process XIncludes before converting XML to relations</li>
<li>
<strong>relpipe-lib-protocol → relpipe-lib-common</strong>: this module was renamed and converted to a shared library, it will contain some common functions instead of just the header files</li>
</ul>
<p>
See the <m:a href="examples">examples</m:a> and <m:a href="screenshots">screenshots</m:a> pages for details.
</p>
<p>
Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change.
Any suggestions, ideas and bug reports are welcome in our <m:a href="contact">mail box</m:a>.
</p>
<h2>Streamlets</h2>
<p>
<em>Streamlet</em> is a small stream that inflows into the main stream, fuse with it and (typically) brings new attributes.
</p>
<p>
From the technical point of view, streamlets are something between classic <m:a href="classic-example">filters</m:a> and functions.
Unlike a function, the streamlet can be written in any programming language and runs as a separate process.
Unlike a filter, the streamlet does not relplace whole stream with a new one, but reads certain attributes from the original stream and adds some new ones back.
Common feature of filters and streamlets is that both continually read the input and continually deliver outputs, so the memory requirements are usually constant and „infinite“ streams might be processed.
And unlike ordinary commands (executed e.g. using <code>xargs</code> or a shell loop over a set of files), the streamlet does not <code>fork()</code> and <code>exec()</code> on each input file – the single streamlet process is reused for all records in the stream which is much more efficient (especially if there is some expensive initialization phase).
</p>
<p>
Because streamlets are small scripts or compiled programs, they can be used for extending <m:name/> with minimal effort.
A streamlet can be e.g. few-lines Bash script – or on the other hand: a more powerful C++ or Java program.
Currently we have templates/examples written in Bash, C++ and Java. But it is possible to use any scripting or programming language.
The streamlet communicates with its parent (who manages the whole stream) through a simple <a href="img/streamlet-release-v0.15.png">message-based protocol</a>.
Full documentation will be published when stable (before v1.0.0) as a part of the public API.
</p>
<p>
The first module where streamlets have been implemented is <code>relpipe-in-filesystem</code>.
Streamlets in this module get a single input attribute (the file path) and add various file metadata to the stream.
We have e.g. streamlets that compute hashes (SHA-256 etc.), extract metadata from image files (PNG, JPEG etc.) or PDF documents (title, author… or even the full content in plain-text), <m:a href="streamlets-preview">OCR</m:a>-recognized text from images,
count lines of code, extract portions of XML files using XPath or some metadata from JAR/ZIP files.
The streamlets are a way how to keep <code>relpipe-in-filesystem</code> simple with small code footpring while making it extensible and thus powerful.
We are not going to face the question: „Should we add this nice feature (+) and thus also this library dependency (-)? Would it be bloatware or not?“.
We (or the users) can add any feature through streamlets while the core <code>relpipe-in-filesystem</code> will stay simple and nobody (who does not need that feature) will not suffer from the growing complexity.
</p>
<p>
But streamlets are not limited to <code>relpipe-in-filesystem</code> – they are a general concept and there will be <code>relpipe-tr-streamlet</code> module.
Such streamlets will get any set of input attributes (not only file names) defined by the user and compute values based on them.
Such streamlet can e.g. modify a text attribute, compute a sum of numeric attributes, encrypt or decrypt values or interact with some external systems.
Writing a streamlet is easier than writing a transformation (like <code>relpipe-tr-*</code>) and it is more than OK to write simple single-purpose ad-hoc streamlets.
It is like writing simple shell scripts or functions.
Examples of really simple streamlets are: <code>inode</code> (Bash) and <code>pid</code> (C++).
It requires implementing only two functions: first one returns names and types of the output attributes and the second one returns that attributes for a record.
However, the streamlets might be parametrized through options, might return dynamic number of output attributes and might provide complex logic.
Some streamlets will become a stable part of the <m:name/> specification and API (<code>xpath</code> and <code>hash</code> seems to be such ones).
</p>
<p>
One of open questions is whether to have streamlets in <code>relpipe-in-filesystem</code> when we have <code>relpipe-tr-streamlet</code>.
<em>One tool should do one thing</em> and <em>we should not duplicate the effort</em>…
But it still makes some sense because the file streamlets are specific kind of streamlets and e.g. Bash completion should suggest them if we work with files but not with other data.
And it is also nice to have all metadata collecting on the same level in a single command (i.e. <code>--streamlet</code> beside <code>--file</code> and <code>--xattr</code>)
than having to collect basic and extended file attributes using single command and collect other file metadata using different command.
</p>
<h2>Parallel processing</h2>
<p>
There are two kinds of parallelism: over attributes and over records.
</p>
<p>
Because streamlets are forked processes, they are quite naturally parallelized over attributes.
We can e.g. compute SHA-1 hash in one streamlet and SHA-256 hash in another streamlet and we will utilize two CPU cores (or we can ask one streamlet to compute both SHA-1 and SHA-256 hashes and then we will utilize only one CPU core).
The <code>relpipe-in-filesystem</code> tool simply 1) feeds all streamlet instances with the current file name, 2) streamlets work in parallel and then 3) the tool collects results from all streamlets.
</p>
<p>
But it would not be enough. Today, we usually have more CPU cores than heavy attributes (like hashes).
So we need to process multiple records in parallel.
The first design proposal (not implemented) was that the tool will simply distribute the file names to STDINs of particular streamlet processes in the round-robin fashion and processes will write to the common STDOUT (just with a lock for synchronization to keep the records atomic – the <m:name/> data format is specifically designed for such use).
This will be really simple and somehow helpful (better than nothing).
But this design has a significant flaw: the tool is not aware of how busy particular streamlet processes are and will feed them with tasks (file names) equally.
So it will work satisfactorily only in case that all tasks have similar difficultness.
This is unfortunately not the usual case because e.g. computing a hash of a big file takes much more time than computing a hash of a small file.
Thus some streamlet processes will be overloaded while other will be idle and in the end whole group will be waiting for the overloaded ones (and only one or few CPU cores will be utilized).
So this is not a good way to go.
</p>
<p>
The solution is using a queue. The tool will feed the tasks (file names in the <code>relpipe-in-filesystem</code> case) to the queue
and the streamlet processes will fetch them from the queue as soon as they are idle.
So we will utilize all the CPU cores all the time (obviously if we have more records than CPU cores, which is usually true).
Because our target platform are POSIX operating systems (and primary one is GNU/Linux), we choose POSIX MQ as the queue.
POSIX MQ is a nice and simple technology, it is standardized and really classic. It does not require any broker process or any third-party library so it does not bring additional dependencies – it is provided directly by the OS.
However, fallback is still possible:
a) if we set <code>--parallel 1</code> (which is default behavior), it will run directly in a single process without the queue;
b) the POSIX MQ have quite simple API so it is possible to write an adapter and port the tool to another system that does not have POSIX MQ and still enjoy the parallelism (or simply reimplement this API using shared memory and a semaphore).
</p>
<p>
We could add another queue to the output side and use it for serialization of the stream (which flows to the single STDOUT/FD).
But it is not necessary (thanks to the <m:name/> format design) and would add just some overhead.
So on the output side, we use just a POSIX semaphore (and a lock/guard based on it).
Thus the tool still has no other dependencies than the standard library and the operating system.
</p>
<p>
If we still have idle CPU cores or machines and need even more parallelism, streamlets can fork their own sub-processes, use threads or some technology like MPI or OpenMP.
However, simple parallel processing of records (<code>--parallel N</code>) is usually more than suitable and efficiently utilize our hardware.
</p>
<h2>XPath modes</h2>
<p>
Both <code>relpipe-in-xmltable</code> and the <code>xpath</code> streamlet uses XPath language to extract values from XML documents.
There are several modes of value extraction:
</p>
<ul>
<li>
<code>string</code>: this is default option, simply the text content
</li>
<li>
<code>boolean</code>: the value converted to a boolean in the XPath fashion;
<!--
can be used also to check whether given file is valid XML:
<code>- -streamlet xpath - -option attribute . - -option mode boolean - -as valid_xml</code>
-->
</li>
<li>
<code>raw-xml</code>: a portion of original XML document;
this is a way to put multiple values or any structured data in a single attribute;
if the XPath points to multiple nodes, it can still be returned as a valid XML document using a configurable wrapper node (so we can return e.g. all headlines from a document)
</li>
<li>
<code>line-number</code>: number of the line where given node was found;
this can be used for referencing particular place in the document</li>
<li>
<code>xpath</code>: XPath pointing to particular node;
it would be a different XPath expression than the original one (which might point to a set of nodes);
this can also be used for referencing particular place in the document</li>
</ul>
<p>
Both tools share the naming convention and are configured in a similar way – using e.g. <code>relpipe-in-xmltable --mode raw-xml</code> or <code>--streamlet xpath --option mode raw-xml</code>.
</p>
<h2>Feature overview</h2>
<h3>Data types</h3>
<ul>
<li m:since="v0.8">boolean</li>
<li m:since="v0.15">variable-length signed integer (SLEB128)</li>
<li m:since="v0.8">string in UTF-8</li>
</ul>
<h3>Inputs</h3>
<ul>
<li m:since="v0.11">Recfile</li>
<li m:since="v0.9">XML</li>
<li m:since="v0.13">XMLTable</li>
<li m:since="v0.9">CSV</li>
<li m:since="v0.9">file system</li>
<li m:since="v0.8">CLI</li>
<li m:since="v0.8">fstab</li>
<li m:since="v0.14">SQL script</li>
</ul>
<h3>Transformations</h3>
<ul>
<li m:since="v0.13">sql: filtering and transformations using the SQL language</li>
<li m:since="v0.12">awk: filtering and transformations using the classic AWK tool and language</li>
<li m:since="v0.10">guile: filtering and transformations defined in the Scheme language using GNU Guile</li>
<li m:since="v0.8">grep: regular expression filter, removes unwanted records from the relation</li>
<li m:since="v0.8">cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)</li>
<li m:since="v0.8">sed: regular expression replacer</li>
<li m:since="v0.8">validator: just a pass-through filter that crashes on invalid data</li>
<li m:since="v0.8">python: highly experimental</li>
</ul>
<h3>Streamlets</h3>
<ul>
<li m:since="v0.15">xpath (example, unstable)</li>
<li m:since="v0.15">hash (example, unstable)</li>
<li m:since="v0.15">jar_info (example, unstable)</li>
<li m:since="v0.15">mime_type (example, unstable)</li>
<li m:since="v0.15">exiftool (example, unstable)</li>
<li m:since="v0.15">pid (example, unstable)</li>
<li m:since="v0.15">cloc (example, unstable)</li>
<li m:since="v0.15">exiv2 (example, unstable)</li>
<li m:since="v0.15">inode (example, unstable)</li>
<li m:since="v0.15">lines_count (example, unstable)</li>
<li m:since="v0.15">pdftotext (example, unstable)</li>
<li m:since="v0.15">pdfinfo (example, unstable)</li>
<li m:since="v0.15">tesseract (example, unstable)</li>
</ul>
<h3>Outputs</h3>
<ul>
<li m:since="v0.11">ASN.1 BER</li>
<li m:since="v0.11">Recfile</li>
<li m:since="v0.9">CSV</li>
<li m:since="v0.8">tabular</li>
<li m:since="v0.8">XML</li>
<li m:since="v0.8">nullbyte</li>
<li m:since="v0.8">GUI in Qt</li>
<li m:since="v0.8">ODS (LibreOffice)</li>
</ul>
<h2>New examples</h2>
<ul>
<li><m:a href="examples-parallel-hashes">Computing hashes in parallel</m:a></li>
<li><m:a href="examples-runnable-jars">Finding runnable JARs</m:a></li>
<li><m:a href="examples-xhtml-filesystem-xpath">Collecting statistics from XHTML pages</m:a></li>
</ul>
<h2>Backward incompatible changes</h2>
<p>
The data format has changed: SLEB128 is now used for encoding numbers.
If the data format was used only on-thy-fly, no additional steps are required during upgrade.
If the data format was used for persistence (streams redirected to files), recommended upgrade procedure is:
convert files to XML using old version of <code>relpipe-out-xml</code> and then convert it from XML back using new version of <code>relpipe-in-xml</code>.
</p>
<h2>Installation</h2>
<p>
Instalation was tested on Debian GNU/Linux 10.2.
The process should be similar on other distributions.
</p>
<m:pre src="examples/release-v0.15.sh" jazyk="bash" odkaz="ano"/>
<p>
<m:name/> are modular thus you can download and install only parts you need (the libraries are needed always).
Tools <code>out-gui.qt</code> and <code>tr-python</code> require additional libraries and are not built by default.
</p>
</text>
</stránka>