<stránka

	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"

	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">

	<perex>new public release of Relational pipes</perex>

	<text xmlns="http://www.w3.org/1999/xhtml">

		<p>

			We are pleased to introduce you the new development version of <m:name/>.

			This release improves and standardizes the CLI interface of several tools, brings a new output module + several smaller improvements.

		</p>

		<ul>

			<li>

				<strong>The <code>relpipe-in-cli</code> module has changed CLI interface</strong>: see details below

			</li>

			<li>

				<strong>The <code>relpipe-in-csv</code> module has changed CLI interface</strong>: see details below

			</li>

			<li>

				<strong>The <code>relpipe-out-csv</code> module has new CLI interface</strong>: see details below

			</li>

			<li>

				<strong>The <code>relpipe-out-tabular</code> module has new CLI interface</strong>: see details below

			</li>

			<li>

				<strong>The <code>relpipe-tr-guile</code> was renamed to <code>relpipe-tr-scheme</code></strong>: see details below

			</li>

			<li>

				<strong>New tool <code>relpipe-out-jack</code> for generating MIDI through JACK</strong>: see details below

			</li>

		</ul>

		<p>

			See the <m:a href="examples">examples</m:a> and <m:a href="screenshots">screenshots</m:a> pages for details.

		</p>

		<p>

			Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change.

			Any suggestions, ideas and bug reports are welcome in our <m:a href="contact">mailing list</m:a>.

		</p>

		<h2>New CLI interfaces</h2>

		<p>

			Several earlier tools have non-standard CLI interface and does not conform with <m:a href="principles">principle</m:a> „Use --long-options“.

			Posistional parameters might be shorter but they are also cryptic and less flexible.

			In this release the <code>relpipe-in-cli</code>, <code>relpipe-in-csv</code> tools dropped their old CLI interface

			and got a new one based on named parameters and long options.

			The <code>relpipe-out-csv</code> and <code>relpipe-out-tabular</code> had no parameters in previous versions and now have standard CLI interface.

		</p>

		<p>

			In <code>relpipe-out-tabular</code> we can now optionally omit several parts:

			<code>--write-record-count false</code>,

			<code>--write-types false</code>,

			<code>--write-relation-name false</code>

			and thus simplify the output if we do not need given information.

		</p>

		<pre><![CDATA[ ╭───────┬────────╮

 │ first │ second │

 ├───────┼────────┤

 │     1 │ abc    │

 │     2 │ def    │

 ╰───────┴────────╯]]></pre>

		<p>

			These options may be set globally or for particular relation using the <code>--relation</code> option with pattern parameter.

			Thus we may e.g. print some relations with data types and some without them.

		</p>

		<p>

			Remaining tools that will be transformed in the same way: <code>relpipe-tr-grep</code>, <code>relpipe-tr-cut</code>, <code>relpipe-tr-sed</code>

			and will get standard CLI interface.

		</p>

		<h2>JACK (MIDI) output module</h2>

		<p>

			A powerful audio system called <a href="https://jackaudio.org/">JACK</a> allows us to

			build pipelines consisting of audio interfaces, players, recorders, filters and effects…

			and route sound streams (both PCM and MIDI) through them.

			MIDI messages can come from keyboards or other hardware MIDI controllers or from MIDI players and other software.

		</p>

		<p>

			In the <m:a href="release-v0.16">v0.16</m:a> release, the <code>relpipe-in-jack</code> was introduced.

			This tools allowed capturing the MIDI events from JACK and sending them to the relational pipeline.

			Now, in v0.17, we introduce the counterpart tool <code>relpipe-out-jack</code> that reads relational data

			and sends SysEx messages, MIDI notes and other events to the JACK pipeline.

		</p>

		<m:img src="img/jack-connections-2.png"/>

		<p>

			We can join the JACK graph with <code>relpipe-out-jack</code> command.

			It does not produce anything on STDOUT, it sends MIDI events to JACK instead, so no other output data are needed.

		</p>

		<p>

			Besides capturing and playing the MIDI, we can use the <code>relpipe-in-jack</code> and <code>relpipe-out-jack</code>

			to tak a snapshot of the JACK graph (connections between ports) and recreate it later.

			Or establish any new connections or break existing ones.

			Either through CLI commands of <code>relpipe-in-jack</code> or through relational data (list of connection and disconnection commands)

			sent to <code>relpipe-out-jack</code>.

		</p>

		<p>

			Both commands have also the <code>--required-connections</code> and <code>--connect-to</code> parameters,

			so they can react on reaching particular connection count or automatically connect their port to another one specified on the CLI.

			The bash-completion suggests available port names.

		</p>

		<p>

			More details in the examples:

			<m:a href="examples-jack-ports">Working with JACK ports</m:a>

			and <m:a href="examples-jack-midi-generating-1">Generating and sending MIDI messages</m:a>.

		</p>

		<h2>Feature overview</h2>

		<h3>Data types</h3>

		<ul>

			<li m:since="v0.8">boolean</li>

			<li m:since="v0.15">variable-length signed integer (SLEB128)</li>

			<li m:since="v0.8">string in UTF-8</li>

		</ul>

		<h3>Inputs</h3>

		<ul>

			<li m:since="v0.11">Recfile</li>

			<li m:since="v0.9">XML</li>

			<li m:since="v0.13">XMLTable</li>

			<li m:since="v0.9">CSV</li>

			<li m:since="v0.9">file system</li>

			<li m:since="v0.8">CLI</li>

			<li m:since="v0.8">fstab</li>

			<li m:since="v0.14">SQL script</li>

			<li m:since="v0.16">JACK</li>

		</ul>

		<h3>Transformations</h3>

		<ul>

			<li m:since="v0.13">sql: filtering and transformations using the SQL language</li>

			<li m:since="v0.12">awk: filtering and transformations using the classic AWK tool and language</li>

			<li m:since="v0.10">scheme: filtering and transformations defined in the Scheme language using GNU Guile</li>

			<li m:since="v0.8">grep: regular expression filter, removes unwanted records from the relation</li>

			<li m:since="v0.8">cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)</li>

			<li m:since="v0.8">sed: regular expression replacer</li>

			<li m:since="v0.8">validator: just a pass-through filter that crashes on invalid data</li>

			<li m:since="v0.8">python: highly experimental</li>

		</ul>

		<h3>Streamlets</h3>

		<ul>

			<li m:since="v0.15">xpath (example, unstable)</li>

			<li m:since="v0.15">hash (example, unstable)</li>

			<li m:since="v0.15">jar_info (example, unstable)</li>

			<li m:since="v0.15">mime_type (example, unstable)</li>

			<li m:since="v0.15">exiftool (example, unstable)</li>

			<li m:since="v0.15">pid (example, unstable)</li>

			<li m:since="v0.15">cloc (example, unstable)</li>

			<li m:since="v0.15">exiv2 (example, unstable)</li>

			<li m:since="v0.15">inode (example, unstable)</li>

			<li m:since="v0.15">lines_count (example, unstable)</li>

			<li m:since="v0.15">pdftotext (example, unstable)</li>

			<li m:since="v0.15">pdfinfo (example, unstable)</li>

			<li m:since="v0.15">tesseract (example, unstable)</li>

		</ul>

		<h3>Outputs</h3>

		<ul>

			<li m:since="v0.11">ASN.1 BER</li>

			<li m:since="v0.11">Recfile</li>

			<li m:since="v0.9">CSV</li>

			<li m:since="v0.8">tabular</li>

			<li m:since="v0.8">XML</li>

			<li m:since="v0.8">nullbyte</li>

			<li m:since="v0.8">GUI in Qt</li>

			<li m:since="v0.8">ODS (LibreOffice)</li>

			<li m:since="v0.17">JACK</li>

		</ul>

		<h2>New examples</h2>

		<ul>

			<li><m:a href="examples-jack-ports">Working with JACK ports</m:a></li>

			<li><m:a href="examples-jack-midi-generating-1">Generating and sending MIDI messages</m:a></li>

		</ul>

		<h2>Backward incompatible changes</h2>

		<p>

			The <code>relpipe-tr-guile</code> has been renamed to <code>relpipe-tr-scheme</code>, so it is necessary to modify existing scripts.

			However the CLI interface has not been changed, thus it can be simply updated e.g. by:

		</p>

		<m:pre jazyk="bash">sed s/relpipe-tr-guile/relpipe-tr-scheme/g -i script.sh</m:pre>

		<p>

			The reason behind this rename is that Scheme is a programming language and Guile is one of implementations of this language.

			And we prefer <a href="https://sane-software.globalcode.info/v_0/#a0f42ec9-5032-4f6d-a50a-4b7bddde77f0">depending on interfaces</a> rather than on particular implementations.

			Similar situation is <code>relpipe-tr-sql</code>: this module is also named after the language (SQL), not after the default implementation (SQLite)

			and this module can use <m:a href="examples-tr-sql-odbc">any SQL engine</m:a>.

		</p>

		<p>

			The CLI interface of the <code>relpipe-in-cli</code> tool has been standardized and made consistent with other tools.

			Now, it uses named parameters instead of cryptic positional ones.

			The format of the STDIN stream remains unchanged (values separated by <code>\0</code> byte).

		</p>

		<m:pre jazyk="bash"><![CDATA[# Old syntax before v0.17:

relpipe-in-cli generate "relation_from_cli" 3 \

	"a" "integer" \

	"b" "string" \

	"c" "boolean" \

	"1" "Hello" "true" \

	"2" "World!" "false"

# New syntax:

relpipe-in-cli \

	--relation "relation_from_cli" \

		--attribute "a" integer \

		--attribute "b" string \

		--attribute "c" boolean \

		--record "1" "Hello" "true" \

		--record "2" "World!" "false"

]]></m:pre>

		<p>

			We can repeat these parameters and generate several relations (that was not possibel in the older version).

			One relation may be filled with values from STDIN: <code>--records-on-stdin true</code> (replaces the old <code>relpipe-in-cli generate-from-stdin</code> syntax).

			And the last relation may be filled with values from remaining CLI arguments: <code>--records "1" "Hello" "true" "2" "World!" "false"</code>.

			The demo (<code>relpipe-in-cli demo</code>) has been removed (sample data can be easily generated by setting the CLI arguments).

		</p>

		<p>

			The CLI interface of the <code>relpipe-in-csv</code> tool has also been standardized and made consistent with other tools.

			If we have attribute names on the first line of the CSV file (which is quite common), the usage is same as in previous versions:

		</p>

		<m:pre jazyk="bash"><![CDATA[cat file.csv | relpipe-in-csv]]></m:pre>

		<p>If we have only data in the CSV file (no headers) and thus want to specify them on CLI, the syntax is different:</p>

		<m:pre jazyk="bash"><![CDATA[# Old syntax before v0.17:

cat file.csv | relpipe-in-csv "my_relation" "a" integer "b" boolean

# New syntax:

cat file.csv | relpipe-in-csv \

	--relation "my_relation" \

		--attribute "a" integer \

		--attribute "b" boolean

]]></m:pre>

		<p>Bash-completion scripts for all tools are provided.</p>

		<p>

			Future versions should allow optionally storing the attribute data types in the CSV header and parsing them back (together with the attribute names).

		</p>

		<h2>Installation</h2>

		<p>

			Instalation was tested on Debian GNU/Linux 10.2.

			The process should be similar on other distributions.

		</p>

		<p>

			<m:name/> are modular thus you can download and install only parts you need (the libraries are needed always).

			Tools <code>out-gui.qt</code> and <code>tr-python</code> require additional libraries and are not built by default.

		</p>

	</text>

</stránka>