--- a/relpipe-data/download.xml Tue Apr 09 22:53:40 2019 +0200
+++ b/relpipe-data/download.xml Tue May 28 21:18:20 2019 +0200
@@ -33,6 +33,7 @@
hg clone https://hg.globalcode.info/relpipe/relpipe-out-recfile.cpp;
hg clone https://hg.globalcode.info/relpipe/relpipe-out-tabular.cpp;
hg clone https://hg.globalcode.info/relpipe/relpipe-out-xml.cpp;
+hg clone https://hg.globalcode.info/relpipe/relpipe-tr-awk.cpp;
hg clone https://hg.globalcode.info/relpipe/relpipe-tr-cut.cpp;
hg clone https://hg.globalcode.info/relpipe/relpipe-tr-grep.cpp;
hg clone https://hg.globalcode.info/relpipe/relpipe-tr-guile.cpp;
@@ -58,6 +59,7 @@
<li>2019-01-18: <m:a href="release-v0.9">v0.9</m:a></li>
<li>2019-02-20: <m:a href="release-v0.10">v0.10</m:a></li>
<li>2019-04-08: <m:a href="release-v0.11">v0.11</m:a></li>
+ <li>2019-05-28: <m:a href="release-v0.12">v0.12</m:a></li>
</ul>
</text>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-awk-aggregations.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,90 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Aggregating data with AWK</nadpis>
+ <perex>counting records and computing sum or appending new records</perex>
+ <m:pořadí-příkladu>02600</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>
+ We have filtered records, modified attribute values, added and removed attributes, dropped a relation…
+ and there is one more operation that we can do with AWK: <code>INSERT</code> resp. appending or preppending additional records to the relation
+ – and we can also completely replace the record set by skipping the original records.
+ </p>
+
+ <h2>Adding records</h2>
+
+ <p>
+ Using options <code>--before-records</code> and <code>--after-records</code> we can pass additional AWK code that will be executed – once for given relation.
+ The <code>record()</code> function will then generate an additional record (can be called multiple times and generate more records):
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[awkCode='
+ scheme = "";
+ device = "/dev/vg1/volume123";
+ mount_point = "/mnt/v123";
+ type = "btrfs";
+ options = "relatime";
+ dump = 0;
+ pass = 2;
+
+ record();
+';
+
+relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '1' \
+ --after-records "$awkCode" \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>Which will <code>INSERT</code> one new record:</p>
+
+ <pre><![CDATA[fstab:
+ ╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬───────────────────────────────────────┬────────────────┬────────────────╮
+ │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (integer) │ pass (integer) │
+ ├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼───────────────────────────────────────┼────────────────┼────────────────┤
+ │ UUID │ 29758270-fd25-4a6c-a7bb-9a18302816af │ / │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 1 │
+ │ │ /dev/sr0 │ /media/cdrom0 │ udf,iso9660 │ user,noauto │ 0 │ 0 │
+ │ │ /dev/sde │ /mnt/data │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 2 │
+ │ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │
+ │ │ /dev/mapper/sdf_crypt │ /mnt/private │ xfs │ relatime │ 0 │ 2 │
+ │ │ /dev/vg1/volume123 │ /mnt/v123 │ btrfs │ relatime │ 0 │ 2 │
+ ╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────┴────────────────┴────────────────╯
+Record count: 6]]></pre>
+
+ <h2>Counting and summarizing values</h2>
+
+ <p>We can also compute some statistics like <code>COUNT()</code> and <code>SUM()</code>:</p>
+
+ <m:pre jazyk="bash"><![CDATA[find -print0 | relpipe-in-filesystem \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --before-records 'count = 0; total_size = 0;' \
+ --for-each '{ count++; total_size += size; }' \
+ --after-records 'record();' \
+ --output-attribute count integer \
+ --output-attribute total_size integer \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>and get result:</p>
+
+ <pre><![CDATA[filesystem:
+ ╭─────────────────┬──────────────────────╮
+ │ count (integer) │ total_size (integer) │
+ ├─────────────────┼──────────────────────┤
+ │ 9 │ 818747 │
+ ╰─────────────────┴──────────────────────╯
+Record count: 1]]></pre>
+
+ <p>Where the <code>total_size</code> is the same as will <code>du</code> compute:</p>
+
+ <pre>find . -type f -print0 | du -b -c --files0-from=-</pre>
+
+ <p>Analogously we can compute minimum, maximum etc. using AWK transformation.</p>
+
+ </text>
+
+</stránka>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-awk-boolean-logic.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,61 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Using boolean logic with AWK</nadpis>
+ <perex>boolean data type and logical AND, OR, XOR operators</perex>
+ <m:pořadí-příkladu>02500</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>Expect that we have a relation with four combinations of logical values:</p>
+
+ <pre><![CDATA[logic:
+╭─────────────┬─────────────╮
+│ a (boolean) │ b (boolean) │
+├─────────────┼─────────────┤
+│ true │ true │
+│ true │ false │
+│ false │ true │
+│ false │ false │
+╰─────────────┴─────────────╯
+Record count: 4]]></pre>
+
+ <p>
+ we can use logical operators and functions:
+ </p>
+
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-cli \
+ generate logic 2 \
+ a boolean \
+ b boolean \
+ true true \
+ true false \
+ false true \
+ false false \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --output-attribute AND boolean \
+ --output-attribute OR boolean \
+ --output-attribute XOR boolean \
+ --input-attributes-prepend \
+ --for-each '{ AND = a && b; OR = a || b; XOR = xor(a,b); record(); }' \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>and append their results to the relation as additional attributes:</p>
+
+ <pre><![CDATA[logic:
+ ╭─────────────┬─────────────┬───────────────┬──────────────┬───────────────╮
+ │ a (boolean) │ b (boolean) │ AND (boolean) │ OR (boolean) │ XOR (boolean) │
+ ├─────────────┼─────────────┼───────────────┼──────────────┼───────────────┤
+ │ true │ true │ true │ true │ false │
+ │ true │ false │ false │ true │ true │
+ │ false │ true │ false │ true │ true │
+ │ false │ false │ false │ false │ false │
+ ╰─────────────┴─────────────┴───────────────┴──────────────┴───────────────╯
+Record count: 4]]></pre>
+
+ </text>
+
+</stránka>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-awk-changing-structure.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,112 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Changing structures with AWK</nadpis>
+ <perex>adding or removing attributes or dropping a relation</perex>
+ <m:pořadí-příkladu>02400</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>
+ The AWK transformations can also change the structure of transformed relation.
+ It means adding or removing attributes or dropping the whole relation.
+ </p>
+
+ <h2>Adding attributes with AWK</h2>
+
+ <p>
+ Using <code>--output-attribute</code> we can specify the output attributes.
+ If we do not want to explicitly specify all of them and just want to add some new ones, we will use <code>--input-attributes-append</code> (or <code>--input-attributes-prepend</code>),
+ which will preserve also the input attributes:
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '{ id = NR; record(); }' \
+ --output-attribute id integer \
+ --input-attributes-append \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>This adds one new attribute with ordinal numbers:</p>
+
+ <pre><![CDATA[fstab:
+ ╭──────────────┬─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬───────────────────────────────────────┬────────────────┬────────────────╮
+ │ id (integer) │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (integer) │ pass (integer) │
+ ├──────────────┼─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼───────────────────────────────────────┼────────────────┼────────────────┤
+ │ 1 │ UUID │ 29758270-fd25-4a6c-a7bb-9a18302816af │ / │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 1 │
+ │ 2 │ │ /dev/sr0 │ /media/cdrom0 │ udf,iso9660 │ user,noauto │ 0 │ 0 │
+ │ 3 │ │ /dev/sde │ /mnt/data │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 2 │
+ │ 4 │ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │
+ │ 5 │ │ /dev/mapper/sdf_crypt │ /mnt/private │ xfs │ relatime │ 0 │ 2 │
+ ╰──────────────┴─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────┴────────────────┴────────────────╯
+Record count: 5]]></pre>
+
+
+ <h2>Remnoving attributes with AWK</h2>
+
+ <p>Or we can omit omit attributes unless explicitly specified ones:</p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '{ type_big = toupper(type); record(); }' \
+ --output-attribute mount_point string \
+ --output-attribute type string \
+ --output-attribute type_big string \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>which effectively removes unlisted attributes:</p>
+
+ <pre><![CDATA[fstab:
+ ╭──────────────────────┬───────────────┬───────────────────╮
+ │ mount_point (string) │ type (string) │ type_big (string) │
+ ├──────────────────────┼───────────────┼───────────────────┤
+ │ / │ ext4 │ EXT4 │
+ │ /media/cdrom0 │ udf,iso9660 │ UDF,ISO9660 │
+ │ /mnt/data │ ext4 │ EXT4 │
+ │ /home │ btrfs │ BTRFS │
+ │ /mnt/private │ xfs │ XFS │
+ ╰──────────────────────┴───────────────┴───────────────────╯
+Record count: 5]]></pre>
+
+
+ <p>AWK is a powerful language so we can use conditions, for cycles etc. and write much more complex transformations.</p>
+
+ <h2>Dropping a relation</h2>
+
+ <p>
+ A relation can be „dropped“ which means that transformation will run but no relational output will be generated for it
+ (even the header will be omitted, so it differs from just eliminating all records by a condition).
+ Using AWK for such a simple operation like <code>DROP</code> seems weird but sometimes it might make sense due to intentional side effects.
+ </p>
+
+ <p>
+ Because the AWK code is executed for each record, we can e.g. write some output to a file or to STDERR:
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '{ printf("%s → %s\n", device, mount_point) > "/dev/stderr" }' \
+ --drop]]></m:pre>
+
+ <p>Which prints text:</p>
+
+ <pre><![CDATA[29758270-fd25-4a6c-a7bb-9a18302816af → /
+/dev/sr0 → /media/cdrom0
+/dev/sde → /mnt/data
+a2b5f230-a795-4f6f-a39b-9b57686c86d5 → /home
+/dev/mapper/sdf_crypt → /mnt/private]]></pre>
+
+ <p>
+ Then <code>relpipe-tr-awk</code> works much like an output filter (converts relational data to another format).
+ However, if there are more relations and some of theme are not matched by <code>--relation</code>, they will be passed through and delivered to the STDOUT in the relational format.
+ STDERR might be occasionally polluted by some warning messages, so using a dedicated file for such output is a safer way.
+ </p>
+
+
+ </text>
+
+</stránka>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-awk-changing-values.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,45 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Changing values with AWK</nadpis>
+ <perex>regular expression text replacement</perex>
+ <m:pořadí-příkladu>02300</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>
+ Besides filtering, we can use an AWK transformation to modify attribute values.
+ This means simply rewriting the value of given variable in AWK and calling <code>record()</code> function at the end.
+ </p>
+
+ <p>For example we can move all volumes mounted under <code>/mnt/</code> to another directory using regular expressions:</p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '{ mount_point = gensub("^/mnt/", "/mnt/old/", "g", mount_point); record(); }' \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>which will result in:</p>
+
+ <pre><![CDATA[fstab:
+ ╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬───────────────────────────────────────┬────────────────┬────────────────╮
+ │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (integer) │ pass (integer) │
+ ├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼───────────────────────────────────────┼────────────────┼────────────────┤
+ │ UUID │ 29758270-fd25-4a6c-a7bb-9a18302816af │ / │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 1 │
+ │ │ /dev/sr0 │ /media/cdrom0 │ udf,iso9660 │ user,noauto │ 0 │ 0 │
+ │ │ /dev/sde │ /mnt/old/data │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 2 │
+ │ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │
+ │ │ /dev/mapper/sdf_crypt │ /mnt/old/private │ xfs │ relatime │ 0 │ 2 │
+ ╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────┴────────────────┴────────────────╯
+Record count: 5]]></pre>
+
+ <p>
+ We can modify multiple attributes in a single transformation
+ and we can also use other AWK functions like <code>toupper()</code>, <code>tolower()</code> etc.
+ </p>
+
+ </text>
+
+</stránka>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-awk-debugging.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,129 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Debugging AWK transformations</nadpis>
+ <perex>discovering variable mappings and transformation internals</perex>
+ <m:pořadí-příkladu>02200</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>In most cases, AWK transformations should be quite straightforward, but sometimes we need to look inside the box.</p>
+
+ <h2>Mapping attributes to variables</h2>
+
+ <p>
+ Relations have named attributes but in a language like AWK we work with named variables.
+ In most cases, the names will match 1:1. But not always.
+ The mapping is needed because not all valid attribute names are also valid variable names in particular language, thus sometimes some escaping or prefixing is necessary.
+ So there is <code>--debug-variable-mapping</code> option for printing the mappings between attributes and variables.
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '1' \
+ --debug-variable-mapping \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>This option prepends additional relation with these metadata to the stream:</p>
+
+ <pre><![CDATA[fstab.variableMapping:
+ ╭────────────────────┬───────────────────╮
+ │ attribute (string) │ variable (string) │
+ ├────────────────────┼───────────────────┤
+ │ device │ device │
+ │ dump │ dump │
+ │ mount_point │ mount_point │
+ │ options │ options │
+ │ pass │ pass │
+ │ scheme │ scheme │
+ │ type │ type │
+ ╰────────────────────┴───────────────────╯
+Record count: 7
+fstab:
+ ╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬───────────────────────────────────────┬────────────────┬────────────────╮
+ │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (integer) │ pass (integer) │
+ ├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼───────────────────────────────────────┼────────────────┼────────────────┤
+ │ UUID │ 29758270-fd25-4a6c-a7bb-9a18302816af │ / │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 1 │
+ │ │ /dev/sr0 │ /media/cdrom0 │ udf,iso9660 │ user,noauto │ 0 │ 0 │
+ │ │ /dev/sde │ /mnt/data │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 2 │
+ │ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │
+ │ │ /dev/mapper/sdf_crypt │ /mnt/private │ xfs │ relatime │ 0 │ 2 │
+ ╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────┴────────────────┴────────────────╯
+Record count: 5]]></pre>
+
+ <p>If we are interested only in the mappings, we should use it in combination with <code>--drop</code> option:</p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '1' \
+ --debug-variable-mapping \
+ --drop \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>which skips the actual data:</p>
+
+ <pre><![CDATA[fstab.variableMapping:
+ ╭────────────────────┬───────────────────╮
+ │ attribute (string) │ variable (string) │
+ ├────────────────────┼───────────────────┤
+ │ device │ device │
+ │ dump │ dump │
+ │ mount_point │ mount_point │
+ │ options │ options │
+ │ pass │ pass │
+ │ scheme │ scheme │
+ │ type │ type │
+ ╰────────────────────┴───────────────────╯
+Record count: 7]]></pre>
+
+ <p>Because there were no collisions, variables have same names as attributes. But in this case:</p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-cli generate t 3 \
+ "if" string \
+ "6pack" string \
+ "spa ces" string \
+ | relpipe-tr-awk \
+ --relation t \
+ --debug-variable-mapping \
+ --drop \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>mapping rules come in to the play:</p>
+
+ <pre><![CDATA[t.variableMapping:
+ ╭────────────────────┬───────────────────╮
+ │ attribute (string) │ variable (string) │
+ ├────────────────────┼───────────────────┤
+ │ 6pack │ _6pack │
+ │ if │ _if │
+ │ spa ces │ spa_ces │
+ ╰────────────────────┴───────────────────╯
+Record count: 3]]></pre>
+
+ <p>in order to make variable names valid in AWK.</p>
+
+
+ <h2>Inspecting the internals of an AWK transformation</h2>
+
+ <p>
+ The <code>relpipe-tr-awk</code> calls AWK as a child process and passes data of given relation to it for actual processing.
+ Because it executes <code>awk</code> program found on <code>$PATH</code>, we can easily switch the AWK implementations.
+ In the source code repository, there is <code>scripts/awk</code> – a wrapper script.
+ We can modify the <code>$PATH</code>, so this wrapper will be called by <code>relpipe-tr-awk</code>.
+ This script captures CLI arguments, STDIN, STDOUT, STDERR and the exit code and saves them to files in the temp directory.
+ Using GNU Screen and the <em>inotifywait</em> we can build a kind of IDE and watch what happens inside during the transformation:
+ </p>
+
+ <m:img src="img/awk-wrapper-debug-1.png"/>
+
+ <p>
+ So we can inspect the generated AWK code and the inputs and outputs of the AWK process.
+ Recommended usage is described in the <code>scripts/awk</code> script.
+ </p>
+
+ </text>
+
+</stránka>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-awk-filtering.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,108 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Complex filtering with AWK</nadpis>
+ <perex>filtering records with AND, OR and functions</perex>
+ <m:pořadí-příkladu>02100</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>
+ If we need more complex filtering than <code>relpipe-tr-grep</code> can offer, we can write an AWK transformation.
+ Then we can use AND and OR operators and functions like regular expression matching or numerical formulas.
+ </p>
+
+ <p>
+ The tool <code>relpipe-tr-awk</code> calls real AWK program (usually GNU AWK) installed on our system and passes data of given relation to it.
+ Thus we can use any AWK feature in our pipeline while processing relational data.
+ Relational attributes are mapped to AWK variables, so we can reference them by their names instead of mere field numbers.
+ </p>
+
+ <p>
+ The <code>--for-each</code> option is used for both filtering (instead of <code>--where</code>)
+ and arbitrary code execution (for data modifications, adding records, computations or intentional side effects).
+ In AWK, filtering conditions are surrounded by <code>(…)</code> and actions by <code>{…}</code>.
+ Both can be combined together and multiple expressions can be separated by <code>;</code> semicolon.
+ The <code>record()</code> function should be called instead of AWK <code>print</code> (which should never be used directly).
+ Calling <code>record()</code> is not necessary, when only filtering is done (and there are no data modifications).
+ </p>
+
+ <h2>Filtering numbers</h2>
+
+ <p>With AWK we can filter records using standard numeric operators like ==, <, >, >= etc.</p>
+
+ <m:pre jazyk="bash"><![CDATA[find -print0 | relpipe-in-filesystem \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '(size > 2000)' \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>and e.g. list files with certain sizes:</p>
+
+ <pre><![CDATA[filesystem:
+ ╭──────────────────────┬───────────────┬────────────────┬────────────────┬────────────────╮
+ │ path (string) │ type (string) │ size (integer) │ owner (string) │ group (string) │
+ ├──────────────────────┼───────────────┼────────────────┼────────────────┼────────────────┤
+ │ ./relpipe-tr-awk.cpp │ f │ 2880 │ hacker │ hacker │
+ │ ./CLIParser.h │ f │ 5264 │ hacker │ hacker │
+ │ ./AwkHandler.h │ f │ 17382 │ hacker │ hacker │
+ ╰──────────────────────┴───────────────┴────────────────┴────────────────┴────────────────╯
+Record count: 3]]></pre>
+
+
+ <h2>Filtering strings</h2>
+
+ <p>String values can be searched for certain regular expression:</p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '(mount_point ~ /cdrom/)' \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>e.g. <code>fstab</code> records having <code>cdrom</code> in the <code>mount_point</code>:</p>
+
+ <pre><![CDATA[fstab:
+ ╭─────────────────┬─────────────────┬──────────────────────┬───────────────┬──────────────────┬────────────────┬────────────────╮
+ │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (integer) │ pass (integer) │
+ ├─────────────────┼─────────────────┼──────────────────────┼───────────────┼──────────────────┼────────────────┼────────────────┤
+ │ │ /dev/sr0 │ /media/cdrom0 │ udf,iso9660 │ user,noauto │ 0 │ 0 │
+ ╰─────────────────┴─────────────────┴──────────────────────┴───────────────┴──────────────────┴────────────────┴────────────────╯
+Record count: 1]]></pre>
+
+ <p>Case-insensitive search can be switched on by adding:</p>
+
+ <pre>--define IGNORECASE integer 1</pre>
+
+ <h2>AND and OR</h2>
+
+ <p>We can combine multiple conditions using <code>||</code> and <code>&&</code> logical operators:</p>
+
+ <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '(type == "btrfs" || pass == 1)' \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>and build arbitrary complex filters</p>
+
+ <pre><![CDATA[fstab:
+ ╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬───────────────────────────────────────┬────────────────┬────────────────╮
+ │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (integer) │ pass (integer) │
+ ├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼───────────────────────────────────────┼────────────────┼────────────────┤
+ │ UUID │ 29758270-fd25-4a6c-a7bb-9a18302816af │ / │ ext4 │ relatime,user_xattr,errors=remount-ro │ 0 │ 1 │
+ │ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │
+ ╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────┴────────────────┴────────────────╯
+Record count: 2]]></pre>
+
+ <p>Nested <code>(…)</code> work as expected.</p>
+
+ <p>
+ And AWK can do much more – it offers plenty of functions and language constructs that we can use in our transformations.
+ Comperhensive documentation can be found here: <a href="https://www.gnu.org/software/gawk/manual/">Gawk: Effective AWK Programming</a>.
+ </p>
+
+ </text>
+
+</stránka>
--- a/relpipe-data/examples-guile-filtering.xml Tue Apr 09 22:53:40 2019 +0200
+++ b/relpipe-data/examples-guile-filtering.xml Tue May 28 21:18:20 2019 +0200
@@ -27,7 +27,7 @@
We are looking for „satanistic“ icons in our filesystem – those that have size = 666 bytes.
</p>
- <m:pre jazyk="bash"><![CDATA[$ find /usr/share/icons/ -type f -print0 \
+ <m:pre jazyk="bash"><![CDATA[find /usr/share/icons/ -type f -print0 \
| relpipe-in-filesystem \
| relpipe-tr-guile --relation 'files.*' --where '(= $size 666)' \
| relpipe-out-tabular]]></m:pre>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples/release-v0.12.sh Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,44 @@
+# Install dependencies as root:
+su -c "apt install g++ make cmake mercurial pkg-config"
+su -c "apt install libxerces-c-dev" # needed only for relpipe-in-xml module
+su -c "apt install guile-2.2-dev" # needed only for relpipe-tr-guile module; guile-2.0-dev also works but requires a patch (see below)
+su -c "apt install gawk" # needed only for relpipe-tr-awk module
+
+# Run rest of installation as a non-root user:
+export RELPIPE_VERSION="v0.12"
+export RELPIPE_SRC=~/src
+export RELPIPE_BUILD=~/build
+export RELPIPE_INSTALL=~/install
+export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
+export PATH="$RELPIPE_INSTALL/bin:$PATH"
+
+rm -rf "$RELPIPE_BUILD"/relpipe-*
+mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
+
+# Helper functions:
+relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
+relpipe_install() { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
+
+# Download all sources:
+relpipe_download lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter in-cli in-fstab in-xml in-csv in-filesystem in-recfile out-gui.qt out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-python tr-sed tr-validator tr-guile tr-awk
+
+# Optional: At this point, we have all dependencies and sources downloaded, so we can disconnect this computer from the internet in order to verify that our build process is sane, deterministic and does not depend on any external resources.
+
+# Build and install libraries:
+relpipe_install lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter
+
+# Build and install tools:
+relpipe_install in-fstab in-cli in-fstab in-xml in-csv in-recfile tr-cut tr-grep tr-sed tr-guile tr-awk out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile
+
+# relpipe_install in-filesystem # requires GCC 8 or patching (see below)
+
+# Clean-up:
+unset -f relpipe_install
+unset -f relpipe_download
+unset -v RELPIPE_VERSION
+unset -v RELPIPE_SRC
+unset -v RELPIPE_BUILD
+unset -v RELPIPE_INSTALL
+
+# Filter your your fstab using AWK and view it like on an 80s green screen terminal!
+relpipe-in-fstab | relpipe-tr-awk --relation 'fstab' --for-each '(pass == 1 || type == "swap")' | relpipe-out-tabular
Binary file relpipe-data/img/awk-wrapper-debug-1.png has changed
--- a/relpipe-data/implementation.xml Tue Apr 09 22:53:40 2019 +0200
+++ b/relpipe-data/implementation.xml Tue May 28 21:18:20 2019 +0200
@@ -33,6 +33,7 @@
relpipe-out-recfile.cpp executable output c++ GNU GPLv3+
relpipe-out-tabular.cpp executable output c++ GNU GPLv3+
relpipe-out-xml.cpp executable output c++ GNU GPLv3+
+ relpipe-tr-awk.cpp executable transformation c++ GNU GPLv3+
relpipe-tr-cut.cpp executable transformation c++ GNU GPLv3+
relpipe-tr-grep.cpp executable transformation c++ GNU GPLv3+
relpipe-tr-guile.cpp executable transformation c++ GNU GPLv3+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/release-v0.12.xml Tue May 28 21:18:20 2019 +0200
@@ -0,0 +1,111 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Release v0.12</nadpis>
+ <perex>fifth public release of Relational pipes</perex>
+ <m:release>v0.12</m:release>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+ <p>
+ We are pleased to introduce you the new development version of <m:name/>.
+ This release brings AWK support and some smaller changes:
+ </p>
+
+ <ul>
+ <li>
+ <strong>AWK transformations</strong>:
+ now it is possible to write transformations using the classic AWK tool and its language.
+ Relational data can be filtered and modified (incuding additions of new records) or any AWK code can be executed for given relation or records.
+ Structural changes are also possible (adding or removing attributes or dropping relations).
+ The command line syntax is mostly the same as for the Guile transformation.
+ The AWK and Guile transformations are now the most powerful ones in the Relpipe world.
+ </li>
+
+ <li>
+ <strong>AWK and Guile transformations</strong>:
+ option <code>--debug-variable-mapping</code> was added, so it is possible to print mappings (in relational format, of course)
+ between relational attributes and AWK or Guile variables.
+ The mapping is needed because not all valid attribute names are also valid variable names in particular language,
+ thus sometimes some escaping or prefixing is necessary.
+ </li>
+
+ </ul>
+
+ <p>
+ See the <m:a href="examples">examples</m:a> and <m:a href="screenshots">screenshots</m:a> pages for details.
+ </p>
+
+ <p>
+ Please note that this is still a development relasease and thus the API (libraries, CLI arguments, formats) might and will change.
+ Any suggestions, ideas and bug reports are welcome in our <m:a href="contact">mailing list</m:a>.
+ </p>
+
+ <h3>Data types</h3>
+ <ul>
+ <li m:since="v0.8">boolean</li>
+ <li m:since="v0.8">variable unsigned integer (prototype)</li>
+ <li m:since="v0.8">string in UTF-8</li>
+ </ul>
+ <h3>Inputs</h3>
+ <ul>
+ <li m:since="v0.11">Recfile</li>
+ <li m:since="v0.9">XML</li>
+ <li m:since="v0.9">CSV</li>
+ <li m:since="v0.9">file system</li>
+ <li m:since="v0.8">CLI</li>
+ <li m:since="v0.8">fstab</li>
+ </ul>
+ <h3>Transformations</h3>
+ <ul>
+ <li m:since="v0.12">awk: filtering and transformations using the classic AWK tool and language</li>
+ <li m:since="v0.10">guile: filtering and transformations defined in the Scheme language using GNU Guile</li>
+ <li m:since="v0.8">grep: regular expression filter, removes unwanted records from the relation</li>
+ <li m:since="v0.8">cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)</li>
+ <li m:since="v0.8">sed: regular expression replacer</li>
+ <li m:since="v0.8">validator: just a pass-through filter that crashes on invalid data</li>
+ <li m:since="v0.8">python: highly experimental</li>
+ </ul>
+ <h3>Outputs</h3>
+ <ul>
+ <li m:since="v0.11">ASN.1 BER</li>
+ <li m:since="v0.11">Recfile</li>
+ <li m:since="v0.9">CSV</li>
+ <li m:since="v0.8">tabular</li>
+ <li m:since="v0.8">XML</li>
+ <li m:since="v0.8">nullbyte</li>
+ <li m:since="v0.8">GUI in Qt</li>
+ <li m:since="v0.8">ODS (LibreOffice)</li>
+ </ul>
+
+ <p>
+ Instalation was tested on Debian GNU/Linux 9.6.
+ The process should be similar on other distributions.
+ </p>
+
+ <m:pre src="examples/release-v0.12.sh" jazyk="bash" odkaz="ano"/>
+
+ <p>
+ <m:name/> are modular thus you can download and install only parts you need (the libraries are needed always).
+ Tools <code>out-gui.qt</code> and <code>tr-python</code> require additional libraries and are not built by default.
+ </p>
+
+ <p>
+ The module <code>relpipe-in-filesystem</code> uses C++ filesystem API which is supported since GCC 8.
+ This module can be compiled and seems usable even with GCC 6, but requires some patching (switch to the experimental API):
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[sed 's@#include <filesystem>@#include <experimental/filesystem>@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
+sed 's@std::filesystem@std::experimental::filesystem@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
+sed 's/.*PROPERTY CXX_STANDARD.*/#\0/g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/CMakeLists.txt]]></m:pre>
+
+ <p>
+ The module <code>relpipe-tr-guile</code> uses GNU Guile 2.2 but can also work with 2.0.
+ In such case, it requires this patch:
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[sed 's/guile-2\.2/guile-2.0/g' -i "$RELPIPE_SRC"/relpipe-tr-guile.cpp/src/CMakeLists.txt]]></m:pre>
+
+ </text>
+
+</stránka>
--- a/relpipe-data/roadmap.xml Tue Apr 09 22:53:40 2019 +0200
+++ b/relpipe-data/roadmap.xml Tue May 28 21:18:20 2019 +0200
@@ -16,7 +16,7 @@
Released versions are described on the <m:a href="download">download</m:a> page.
</p>
- <h2>v0.12, v0.13, v0.14 etc.</h2>
+ <h2>v0.13, v0.14, v0.15 etc.</h2>
<p>
Releases for discussion and verification of the format and API design.
@@ -48,7 +48,6 @@
<h3>Transformations</h3>
<ul>
- <li>AWK</li>
<li>SQL: sqlite</li>
</ul>
<h3>Outputs</h3>
