# HG changeset patch # User František Kučera # Date 1645400591 -3600 # Node ID 5bc2bb8b7946a924f567ad47b4a31b598e21ec1b # Parent cc60c8dd79249b6be3ebf23307e2854d42e31037 Release v0.18 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/big-picture/relpipe-1.tmx --- a/relpipe-data/big-picture/relpipe-1.tmx Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/big-picture/relpipe-1.tmx Mon Feb 21 00:43:11 2022 +0100 @@ -1,134 +1,172 @@ - + - + -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 - + -48,42,60,25,0,26,6,12,30,25,0,0,0,0,0, -48,42,60,28,0,31,6,12,30,28,0,26,36,42,54, -48,42,60,33,34,33,6,12,30,28,0,31,36,42,54, -48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, -48,42,60,28,0,31,6,12,30,33,34,33,36,42,54, -48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, -48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, -48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, -48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, -48,42,60,28,0,27,34,34,34,23,0,27,36,42,54, -48,42,60,28,0,0,0,0,0,0,0,0,0,0,0, -48,42,60,23,0,0,0,0,0,0,0,0,0,0,0 +0,0,0,0,48,42,60,25,0,26,6,12,30,25,0,0,0,0,0, +0,0,0,0,48,42,60,28,0,31,6,12,30,28,0,26,36,42,54, +48,42,60,25,48,42,60,33,34,33,6,12,30,28,0,31,36,42,54, +48,42,60,33,36,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +48,42,60,28,48,42,60,28,0,31,6,12,30,33,34,33,36,42,54, +48,42,60,28,48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +48,42,60,28,48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +48,42,60,23,48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +0,0,0,0,48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +0,0,0,0,48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +0,0,0,0,48,42,60,28,0,31,6,12,30,28,0,31,36,42,54, +0,0,0,0,48,42,60,28,0,31,6,12,30,28,0,27,36,42,54, +0,0,0,0,48,42,60,23,0,27,34,34,34,23,0,0,0,0,0 - + filesystem - + fstab - + XML - + CSV - + CLI - + grep - + cut - + sed - + validator - + Scheme - + SQL - + Python - - XQuery + + XPath - + tabular - + XML - + CSV - + nullbyte - + ODS - + GUI - + → - + → - + recfile - + XMLTable - + AWK - + recfile - + ASN.1 BER - + SQL script - + + INI + + + YAML + + + JSON + + + JACK/MIDI + + + JACK/MIDI + + INI - - YAMLTable + + ASN.1 BER + + + CBOR - - JSONTable + + DSV + + + TOML + + + YAML - - JACK/MIDI + + JSON + + + INI + + + Janet - - JACK/MIDI + + types + + + union + + + YAML diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/classic-example.xml --- a/relpipe-data/classic-example.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/classic-example.xml Mon Feb 21 00:43:11 2022 +0100 @@ -61,7 +61,7 @@

Each program used in the pipeline can be written in different programming language and they will work together. - Tools written in C, C++, Java, Lisp, Perl, Python, Rust or any other language can be combined together. + Tools written in C, C++, D, Java, Lisp, Perl, Python, Rust or any other language can be combined together. Thus optimal language can be used for each task.

diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/contact.xml --- a/relpipe-data/contact.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/contact.xml Mon Feb 21 00:43:11 2022 +0100 @@ -49,6 +49,7 @@
  • BTC: 1APShtP1D4iBNTqX1MhY9uDpeyibarWJjP
    • + diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/download.xml --- a/relpipe-data/download.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/download.xml Mon Feb 21 00:43:11 2022 +0100 @@ -14,15 +14,26 @@ -

    @@ -59,11 +75,13 @@

    Build instructions are in release notes below. + (including the script that downloads all sources and builds them)

    Released versions

      +
    • 2021-12-04: v0.18
    • 2020-10-24: v0.17.1
    • 2020-10-20: v0.17
    • 2020-06-06: v0.16
      • diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-asn1-x509.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-asn1-x509.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,121 @@ + + + Exploring content of X.509 certificates + open and query common SSL/TLS certificates or other ASN.1 data encoded in BER/DER/CER + 05000 + + + +

      + X.509 certificates and keys used for SSL/TLS (HTTPS, POP3S, IMAPS etc.) are usually distributed as files either with .pem or .der extension. + Or bundled together in a PKCS#12 container as a .p12 file. + The „text“ PEM format is often considered more „accessible“ or „friendly“ than the binary DER. + However PEM is just Base64 encoded DER original and is actually less legible to the naked eye than DER, + because in DER we can spot at least some strings like common and domain names or validity/expiration dates or recognize certain data structures in a HEX editor. + Base64 just obfuscates everything. PEM can be easily copied through clipboard, which is probably the only advantage of this format (but it can also more likely leak). +

      + + + +

      + So our first step is to get rid of the annoying Base64 pseudo-plain-text encoding – we use one of these commands: +

      + + certificate.der +cat certificate.pem | openssl x509 -inform PEM -outform DER > certificate.der]]> + +

      + Telco veterans could now start reading the DER file with hd or xxd, jumping over the offsets and traversing the sequences and sets… + However most people would appreciate some software that helps them parsing the ASN.1 BER encoding (the superset of DER and CER). + Such software is e.g. Wireshark or dumpasn1. These programs are good for ad-hoc inspection or quick check. +

      + +

      + In v0.18 we have (early and bit raw) support for ASN.1 BER encoding and thus we can get the structured data in a machine-readable form + – which is good for further processing, conversion to other formats or use in scripts. + Because the ASN.1 data model is not relational – actually it is a tree – this format is supported in the relpipe-in-asn1table + command that is modelled after the well-known XMLTable() database function that allows translating arbitrary tree structures to relations using the XPath expressions. + So in relpipe-in-asn1table we can write XPath expressions to query the ASN.1 tree data structures and extract relations, records and attributes + from X.509 certificates, keys or other cryptographic artifacts, LDAP or SNMP packets or any other ASN.1 BER data. +

      + +

      + But how do we know what XPath expressions should we run? + It is useful to see the XML representation of whole source data. + There is a simple trick to do this – use "/" as the XPath for selecting records (is always selects the single record, single node – the root) + and use "." as the XPath to select a single attribute (it always select the root element) + and add --mode raw-xml, so we get the raw XML source instead of the text content of given elements. + We do not have to write this routine by hand – just create a symlink to the example script: +

      + +

      + This example is generic and works also for other formats supported by the relpipe-in-*table commands. +

      + +

      + Then we can analyze X.509 DER certificates stored on our disk or we can fetch some from live servers. + The openssl command helps us with that: +

      + + + /dev/null \ + | openssl x509 -inform PEM -outform DER; +}]]> + +

      Now put both commands together in a pipeline:

      + + + +

      and get this XML representation of the ASN.1 X.509 tree:

      + + + + +

      Once we know the structure, we can easily hack together a function that extracts parts of the tree as relations:

      + + + +

      Everything put together:

      + + + +

      will print:

      + + + +

      + The function above is just a „hello world“ example. + Please note that the XPath expressions need to be carefully crafted with respect to the given format in order to match exactly what we want. +

      + +

      + Instead of printing a table, we can use the relpipe-out-nullbyte tool + the read_nullbyte function + and shell loop over the records (alternative names) and e.g. ping each domain or fetch given root web page using wget or curl. + We can also write a simple script that checks the validity of our own certificates and notifies us in advance when some of them are going to expire. +

      + +

      + Later versions of relpipe-in-asn1table will probably support OID names, so it will not be necessary to use the numeric object identifiers. +

      + +

      + n.b. there is also the relpipe-in-asn1 – this tool reads data generated by its counterpart, the relpipe-out-asn1 (or other ASN.1 BER capable software) + i.e. it is not as universal as relpipe-in-asn1table, it has simpler interface, needs no configuration and expects certain ASN.1 structures (relations serialized in BER format). +

      + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-barcode.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-barcode.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,86 @@ + + + Reading barcodes and QR + recognize barcodes (including QR) in image files or streams + 05400 + + + +

      + Either ubiquitous 1D barcodes (procudct labels, ISBN in form of EAN-13 etc.) or 2D barcodes (QR containing hyperlinks, vCards etc.) + encode numbers, texts or other data into images that can be printed and scanned or photographed later. + Since v0.18, can interact with this technology. +

      + +

      + The relpipe-in-barcode tool reads image data from the standard input, so we can feed a file containing a barcode into it or we can generate one on-the-fly. + We can also use tee command to store generated image data in a file. +

      + + +

      This pipeline generates this relation:

      + + +

      + There might be multiple barcodes (symbols) in the image file and we obtain also their sizes and positions. + We may read a photo: +

      + + + +

      and get plenty of symbols:

      + + +

      + There is also (a bit experimental) example of barcode streamlet that is called from relpipe-in-filesystem + and works with several files at once (can even run in parallel, like any other streamlet): +

      + + + +

      + When we are generating QR codes and use Unicode characters (e.g. the → arrow): +

      + + +

      we may get unexpected result:

      + + + +

      + QR codes use the ISO 8859-1 encoding as default and readers usually do some heuristics to guess actual encoding. + Sometimes it works, sometimes not. + In this particular case, the reader accidentally thought that the encoding was SJIS. + So we can fix it by adding | iconv -t SJIS at the end of the pipeline. +

      + +

      + But this is just a workaround. + We rather want to add UTF-8 BOM or ECI (Extended Channel Interpretations) to make the reader use the same encoding as we did (UTF-8): +

      + + + +

      and get desired result:

      + + +

      + Because the relpipe-in-barcode tool and the barcode-reader streamlet produce data in machine readable form, + we can use them not only for manual ah-hoc reading but also in scripts or batch processing – + e.g. extract payment information from invoices or contact information from scanned business cards, catalogize books or read package labels. +

      + + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-csv-data-types.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-csv-data-types.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,169 @@ + + + CSV and data types + declare or recognize integers and booleans in a typeless format + 04800 + + + +

      + CSV (RFC 4180) is quite good solution when we want to store or share relational data in a simple text format – + both, human-readable and well supported by many existing applications and libraries. + We have even ready-to-use GUI editors, so called spreadsheets (e.g. LibreOffice Calc). + However, such simple formats have usually some drawbacks. + CSV may contain only a single relation (table, sheet). This is not a big issue – we can use several files. + A more serious problem is the absence of data types – in CSV, everything is just a text string. + Thus it was impossible to have loss-less conversion to CSV and back. +

      + + + +

      Data types are missing in CSV by default:

      + + +

      The size attribute was integer and now it is mere string:

      + + + +

      Declare data types in the CSV header

      + +

      + Since v0.18 we can encode the data types (currently strings, integers and booleans) in the CSV header and then recover them while reading. + Such „CSV with data types“ is valid CSV according to the RFC specification and can be viewed or edited in any CSV-capable software. +

      + +

      + The attribute name and data type are separated by the :: symbol e.g. name::string,age::integer,member::boolean. + Attribute names may contain :: (unlike the data type names). +

      + +

      The data type declarations may be added simply by hand or automatically using relpipe-out-csv.

      + + + +

      The relpipe-out-csv + relpipe-in-csv round-trip now does not degrade the data quality:

      + + + +

      + So we can put e.g. a CSV editor between them while storing and versioning the data in a different format (like XML or Recfile). + Such workflow can be effectively managed by make – + make edit will convert versioned data to CSV and launch the editor, + make commit will convert data back from the CSV and commit them in Mercurial, Git or other version control system (VCS). +

      + +

      + Why put into VCS data in different format than CSV? + Formats like XML or Recfile may have each attribute on a separate line which leads to more readable diffs. + At a glance we can see which attributes have been changed. + While in CSV we see just a changed long line and even with a better tools we need to count the comas to know which attribute it was. +

      + +

      + The relpipe-out-csv tool generates data types only when explicitly asked for: --write-types true. + The relpipe-in-csv tool automatically looks for these type declarations + and if all attributes have valid type declarations, they are used, otherwise they are considered to be a part of the attribute name. + This behavior can be disabled by --read-types false (true will require valid type declarations). +

      + + +

      Recognize data types using relpipe-tr-infertypes

      + +

      + Sometimes we may also want to infer data types from the values automatically without any explicit declaration. + Then we put the relpipe-tr-infertypes tool in our pipeline. + It buffers whole relations and checks all values of each attribute. + If they contain all integers or all booleans they are converted to given type. +

      + + + +

      + This approach is inefficient and contradicts streaming, however it is sometimes useful and convenient for small data coming from external sources. + We can e.g. download some data set from network and pipe it through relpipe-in-csv + relpipe-tr-infertypes and improve the data quality a bit. +

      + +

      + We may apply the type inference only on certain relations: --relation "my_relation" + or chose different mode: --mode data or metadata or auto. + The data mode is described above. + In the metadata mode the relpipe-tr-infertypes works similar to relpipe-in-csv --read-types true. + The auto mode checks for the metadata in attribute names first and if not found, it fallbacks to the data mode. + This tool works with any relational data regardless their original format or source (not only with CSV). +

      + + +

      No header? Specify types as CLI parameters

      + +

      + Some CSV files contain just data – have no header line containing the column names. + Then we specify the attribute names and data types as CLI parameters of relpipe-in-csv: +

      + + + +

      + We may also skip existing header line: tail -n +2 and force our own names and types. + However this will not work if there are multiline values in the header – which is not common – + in such cases we should use some relpipe-tr-* tool to rewrite the names or types + (these tools work with relational data instead of plain text). +

      + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-csv-sql-join.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-csv-sql-join.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,258 @@ + + + Running SQL JOINs on multiple CSV files + query a collection of (not only) CSV files using SQL + 05100 + + + +

      + CSV (RFC 4180) is quite good solution when we want to store or share relational data in a simple text format – + both, human-readable and well supported by many existing applications and libraries. + We have even ready-to-use GUI editors, so called spreadsheets e.g. LibreOffice Calc. + (on the other hand, such simple formats have usually some drawbacks…) +

      +

      + In this example, we will show how to query a set of CSV files like it was a relational database. +

      + +

      Suppose we have a CSV file describing our network interfaces:

      + + + +

      and another CSV file with IP addresses assigned to them:

      + + + +

      Loading a CSV file and running basic queries

      + +

      + Simplest task is to parse the file and print it as a table in our terminal or convert it to another format (XML, Recfile, ODS, YAML, XHTML, ASN.1 etc.) + We can also add relpipe-tr-sql in the middle of our pipeline and run some SQL queries – + transform data on-the-fly and send the query result to the relpipe-out-tabular (or other output filter) in place of the original data. + For now, we will filter just the IPv6 addresses: +

      + +

      and get them printed:

      + + +

      + It is alo possible to run several queries at once + and thanks to the format, the result sets are not mixed together, their boundaries are retained and everything is safely passed to the next stage of the pipeline: +

      + +

      resulting in two nice tables:

      + + +

      Using parametrized queries to avoid SQL injection

      +

      + When "4" and "6" are not fixed values, we should not glue them to the query string like version = $version, + because it is a dangerous practice that may lead to SQL injection. + We have parametrized queries for such tasks: +

      + + + +

      Running SQL JOINs, UNIONs etc. on multiple CSV files

      + +

      + To load multiple CSV files into our in-memory database, we just concatenate the relational streams + using the means of our shell – the semicolons and parenthesis: +

      + + +

      Generic version that loads all *.csv files:

      + + +

      Then we can JOIN data from multiple CSV files or do UNIONs, INTERSECTions etc.

      + + + +

      Leveraging shell functions

      + +

      + Good practice is to wrap common code blocks into functions and thus make them reusable. + In shell, the function still works with input and output streams and we can use them when building our pipelines. + Shell functions can be seen as named reusable parts of a pipeline. +

      + + + +

      + The format_result() function checks whether the STDOUT is a terminal or not. + and when printing to the terminal, it generates a table. + When writing to a regular file or STDIN of another process, it passes through original relational data. + Thus ./our-script.sh will print a nice table in the terminal, while ./our-script.sh > data.rp will create a file containing machine-readable data + and ./our-script.sh | relpipe-out-xhtml > report.xhtml will create an XHTML report and ./our-script.sh | relpipe-out-gui will show a GUI window full of tables and maybe also charts. +

      + + + + + + +

      Makefile version

      + +

      + Shell scripts are not the only way to structure and organize our pipelines or generally our data-processing code. + We can also use Make (the tool intended mainly for building sofware), write a Makefile and organize our code around some temporary files and other targets instead of functions. +

      + + $(@) + +define SQL_IP_NIC + SELECT + ip.address AS ip_address, + nic.name AS interface, + nic.address AS mac_address + FROM ip + JOIN nic ON (nic.name = ip.interface) +endef +export SQL_IP_NIC + +define SQL_COUNT_VERSIONS + SELECT + interface, + count(CASE WHEN version=4 THEN 1 ELSE NULL END) AS ipv4_count, + count(CASE WHEN version=6 THEN 1 ELSE NULL END) AS ipv6_count + FROM ip + GROUP BY interface + ORDER BY interface +endef +export SQL_COUNT_VERSIONS + +# Longer SQL queries are better kept in separate .sql files, +# because we can enjoy syntax highlighting and other support in our editors. +# Then we use it like this: --relation "ip_nic" "$$(cat ip_nic.sql)" + +summary.rp: nic.rp ip.rp + cat $(^) \ + | relpipe-tr-sql \ + --relation "ip_nic" "$$SQL_IP_NIC" \ + --relation "counts" "$$SQL_COUNT_VERSIONS" \ + > $(@) + +print_summary: summary.rp + cat $(<) | relpipe-out-tabular +]]> + +

      + We can even combine advantages of Make and Bash together (without calling or including Bash scripts from Make) + and have reusable shell functions available in the Makefile: +

      + + + +

      usage example:

      + + + +

      + Both approaches – the shell script and the Makefile – have pros and cons. + With Makefile, we usually create some temporary files containing intermediate results. + That avoids streaming. But on the other hand, we process (parse, transform, filter, format etc.) only data that have been changed. +

      + + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-ini.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-ini.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,333 @@ + + + Reading and writing INI and unix configs + work with INI, classic key=value configurations, Java .properties or Manifests + 05300 + + + +

      + INI is very common simple text format used for configuration files. + It extends classic key=value config files and adds one more level – sections – for structuring. +

      + +

      + Unfortunatelly, there is no single INI standard and we have to deal with various INI dialects. + Some organizations specify their own INI dialect, but often INI files lacking any formal specification are used. + Our tools support various specifics and dialects through the --parser-option options. + Like any other CLI options, they are supported in our Bash completion scripts. +

      + + + +

      + Classic (unix) key=value config files can be considered a dialect or a subset of the INI format + and thus be processed by INI input and output filters. + It is like INI without any sections or INI having only global properties (key=value at the beginning of the file). + Java .properties and MANIFEST.MF files are similar case. They also can be processed by our tools. +

      + +

      Simple INI example

      + +

      This is a simple INI file:

      + + + +

      We read such INI file using this command:

      + + + +

      and get:

      + + + +

      + Of course, we can do more that listing the entries as a table. + We can convert them to other formats or e.g. run some command on each entry. + Or modify certain values or add/remove entries and save as new INI file. +

      + +

      More complex INI example

      + +

      + The INI format is just seemingly simple. There is much more than section/key/value. + Our parser is by default configured to eat as much INI as possible. + It can be tuned by --parser-option CLI options. + Sometimes, such tuning is necessary, because some INI features are conflicting – some INI dialects are mutually exclusive. +

      + + +

      (please note that syntax highlighting does not support advanced INI features and is sometimes broken)

      + + + +

      We read such INI file using this command:

      + + + +

      and get:

      + + + +

      + If we omit the --enable-sub-keys true option, the sub_key attribute disappears + and we get a[x] and a[y] in the key attribute + i.e. brackets are not interpreted in any special way – they are considered just part of the key. +

      + +

      See Bash completion for complete list of options and dialects.

      + + +

      Keeping comments and whitespace while modifying the INI files

      + +

      + Comments and empty lines are ignored by default and does not generate any output. + This is desired behavior in most cases. + But sometimes we want to pass them through. + We may e.g. generate some XHTML report or other user-friendly output containing the comments, + or we may want to modify existing INI without losing original comments and formatting. + The tool offers the --enable-comments and --enable-whitespace options. + They allows lossless or almost lossless round-trip conversion from INI to relations and back. +

      + +

      + Besides that, we have also the --enable-event-numbers and --enable-line-numbers options. + But they are useful mostly for debugging purposes. +

      + +

      If we enable all that options, we get this output:

      + + + +

      + So we can take the simple INI file (see above) and pass it through a filter + that changes (on-the-fly) the x value in the first-section: +

      + + + +

      and get an INI file that contains original comments and modified values:

      + + + +

      + Both the comment at the beginning of the file + and the x value in second-section + were kept intact. +

      + + + +

      + This AWK filter will generate the same output. + And we can simply add comments just by adding e.g. comment = "the script changed this value"; after the value = "256"; + which results in: +

      + + + +

      This way we can automatically modify our configuration files.

      + + +

      Generating INI files

      + +

      + The relpipe-out-ini tool serves not only for recreating already existing INI files but also for creating new ones. + Data may come from various sources – CSV editor, script, SQL query etc. – and may come in diferent shapes. + While on the INI side we use the term „dialect“, on the relational side we use the term „style“. +

      + + +

      So in relpipe-out-ini we can chose from several styles:

      + +
        +
      • standard: expects same structure as generated by relpipe-in-ini; can generate also comments and whitespace
        • +
      • literal: relation name → section name; attribute name → key; attribute value → value; multiple records → duplicate section names
        • +
      • section-first: like literal, just section names are taken from the first attribute
        • +
      • dropped: this relation is ignored/skipped
        • +
      • auto: use standard style if the relation has key and value attributes; otherwise use literal style
        • +
      + +

      Each relation might be processed in different way – use the --relation and regular expression describing the relation(s) name.

      + +

      + We may for example run: +

      + + + +

      and get INI like this:

      + + + + +

      + This way, we can generate INI files to be loaded as configuration files by other programs. + We can also use relpipe-out-ini for displaying data rather vertically than horizontally (like in common relpipe-out-tabular). + This is useful for data with long values or many attributes. + The relpipe-out-recfile tool can also server this purpose. +

      + +

      Generating XHTML report from several INI files

      + +

      + INI format is used by many programs like Mercurial, Git, KDE, Gnome… and also by ODBC. + Its configuration is spread across several INI files (system-wide driver configuration and connections and user-specific connections). + We can collect all these INI files, use their names as relation names and generate a nice XHTML report: +

      + + odbc-report.xhtml]]> + +

      + n.b. It might contain also sensitive information like passwords. + We may use some relpipe-tr-* filter to remove such records or replace such values by ********. +

      + + +

      Java .properties and MANIFEST.MF

      + +

      + Java has built-in support for simple key=value format – so called .properties files. + They are often used for storing flat data where more advanced format (like XML) is not necessary. + This format supports comments and some escaping sequences. + Escaping of non-ASCII characters was mandatory but since Java 9 the UTF-8 encoding is used by default (no need to call native2ascii). + In we consider .properties files to be a dialect of the INI format so it is supported in relpipe-in-ini and relpipe-out-ini. + The same applies to the MANIFEST.MF which is another key=value format used in the Java world. + Thus we can read and write e.g. Java resource bundles (localization), Java EE or Spring application configs or MANIFEST.MF containing OSGi metadata and other information. +

      + + example.properties +… | relpipe-out-ini --writer-option dialect java-manifest-mf > MANIFEST.MF]]> + +

      + We can pass such data through various transformation (SQL, AWK, Scheme, XPath etc.) + and do conversions to/from various formats (CSV, INI, XML, XHTML, YAML, Recfile, ASN.1 etc.) + or convert .properties to MANIFEST.MF and vice versa. +

      + + +

      Classic unix key=value config files

      + +

      + Many unix or GNU/Linux programs use simple key=value configuration files with # comments + and some basic escaping (like \\ → \, \n → new line etc.) + It is very similar to Java .properties mentioned above. +

      + +

      + The configuration is often spread across many small files heavily stuffed with comments. + If we want to collect just the valid configuration entries from all the files, + we may somehow misuse the INI input filter and the fact that given files contain no sections – turn the filenames into section names + and feed the result to the relpipe-in-ini: +

      + + + (.*) <==/[\1]/g' \ + | relpipe-in-ini --relation 'sysctl' \ + | relpipe-out-tabular]]> + +

      This gives us nice overview of our system configuration:

      + + + +

      We can also do this:

      + + +

      Or create a reusable function:

      + (.*) <==/[\1]/g' \ + | relpipe-in-ini --relation "configs" \ + | relpipe-out-tabular --write-types false; +}]]> + +

      and call it this way:

      + + +

      It is a bit dirty hack (what if the file name, key or value contains <== or […]?) but it will work quite well.

      + +

      And like with Java .properties and MANIFEST-MF, we can use the relpipe-out-ini to generate the configuration files.

      + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-reading-querying-uniform-way.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-reading-querying-uniform-way.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,170 @@ + + + Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way + run XPath queries and turn data from various sources to relations + 04600 + + + +

      + Data come in different shapes and formats. + We can distinguish several main logical models: + relational, + tree + and graph + (a tree is an undirected graph with no cycles) + Arbitrary trees or even graphs are more flexible, but they are also harder to comprehend and work with. + Relational model is somehow limited and easier to grasp, however still flexible enough to describe almost anything. + (actually it can describe anything, it is just a question of how nice and native it should look) + Unsurprisingly, are build around the relational model. + However, sometimes we have to interact with the tree/graph world and deal with data that have other than relational shape. + So we need to bridge the gap between trees/graphs and relations. +

      + +

      + While we have just few logical models, there is abundance of serialization formats i.e. mappings of given logical model to a sequence of octets (bytes). + Relations might be serialized as CSV, ODS, tables in a database, Recfiles etc. + Trees might be serialized as XML, YAML, ASN.1, CBOR, JSON etc. +

      + +

      + Why reinvent the wheel and repeat the same work for each format? +

      + +

      + We already have reusable code for relational data – this is given by the design of , because it separates: inputs, transformations and outputs. + Once the data (e.g. CSV) passes through the input filter, it becomes relational data and can be processed in a uniform way by any transformation(s) or output filter. +

      + +

      + But what about the tree data? We have created a set of tools (input filters) that support various serialization formats, in v0.18: +

      + +
        +
      • XML: relpipe-in-xmltable
        • +
      • ASN.1: relpipe-in-asn1table
        • +
      • CBOR: relpipe-in-cbortable
        • +
      • HTML: relpipe-in-htmltable
        • +
      • INI: relpipe-in-initable
        • +
      • MIME: relpipe-in-mimetable
        • +
      • YAML: relpipe-in-yamltable
        • +
      + +

      + These tools follow the same design principle and offer the same user interface. + So once the user learns one tool, he can use this knowledge also while working with other formats. + The principle is: +

      + +
        +
      • We are converting the tree structure to one or more relations.
        • +
      • For each relation, define the expression that selects record nodes from the tree.
        • +
      • For each attribute, define the expression (relative to the record node) that selects the attribute value.
        • +
      • If anything can not (or is not desired to) be mapped to relations, keep is as a tree, so we can process it later – these (sub)trees might be embedded in normal records or reside in a separate relation.
        • +
      • We may do a full (lossless) conversion, but we may also extract just a single value from the whole tree (generate a single relation with single record and single attribute). Or anything in between. Anyway, the tool and the logic used is still the same.
        • +
      + +

      + This is nothing new – and experienced SQL users should already know where the inspiration comes from: + the XMLTable() SQL function that converts XML tree to a result set (relation). + We just implemented the same functionality as a separate CLI tool, without dependency on any SQL engine and with support for not only XML but also for alternative serialization formats. + And for all of them, we use the same query language: XPath. +

      + +

      + Despite this sounds so XML-ish, we do not translate the alternative formats to the XML markup. There is no text full of angle brackets and ampersands in the middle of the process. + In our case, we should see XML not as a markup text (meta)format, but rather as an in-memory model – a generic tree of node objects stored in the RAM that allows us doing various tree operations (queries, modifications). +

      + + +

      Converting a YAML tree to a set of relations

      + +

      + Flat key-value lists become sooner or later insufficient for software configuration and it is necessary to somehow manage trees of configuration items (or relations, of course). + YAML is quite good tree-serialization format. + It is used e.g. for configuring Java Spring applications or for Netplan network configuration in the Ubuntu GNU/Linux distribution: +

      + + + +

      We can use following command to convert the tree to a set of relations:

      + + + +

      + So we can do a full relational conversion of the original tree structure or extract just few desired values (e.g. the gateway IP address). + We can also pipe a relation to a shell loop and execute some command for each record (e.g. DNS server or IP address). +

      + + + +

      + n.b. YAML is considered to be a superset of JSON, thus tools that can read YAML, can also read JSON. + In current version (v0.18) of the relpipe-in-json and relpipe-in-jsontable are just symbolic links to their YAML counterparts. +

      + +

      + There is also similar example: Reading Libvirt XML files using XMLTable + where we build relations from a XML tree. + The principles are the same for all input formats. +

      + +

      Dealing with the HTML tagsoup

      + +

      + With relpipe-in-htmltable we can extract structured information from poor HTML pages. + And unlike relpipe-in-xmltable, this tool does not require valid XML/XHTML, so it is good for the dirty work. + Processing such invalid data is always bit unreliable, but still better than nothing. +

      + + + +

      Although Mr. Ryszczyks is unable to create a valid document, this script will print:

      + + + +

      + And thanks to the terminal autodetection in the format_result() function, + we can even pipe the result of this script to any relpipe-tr-* or relpipe-out-* + and get machine-readable data instead of the ANSI-colored tables – + so we can do some further processing or conversion to a different format (XHTML, GUI, ODS, Recfile etc.). +

      + +

      The 2xml helper script: yaml2xml, json2xml, asn12xml, mime2xml etc.

      + +

      + Mapping from the original syntax to the tree structure is usually quite intuitive and straightforward. + However, sometimes it is useful to see the XML serialization of this in-memory model. + In the relpipe-in-xmltable.cpp repository we have a helper script called + 2xml + + – this script is not intended to be called directly – instead the user should create a symlink e.g. ini2xml, yaml2xml, asn12xml etc. + The 2xml script choses the right input filter according to the symlink name and uses it for conversion from the source tree-serialization format to the XML tree-serialization format. +

      + +

      + If we want to do the same thing without the helper script, it is quite simple. + We use appropriate relpipe-in-*table tool and extract a single relation with single attribute and single record. + The --records expression is '/' i.e. the root node. + The --attribute expression is '.' i.e. still the root node. + And then we just add the --mode raw-xml to this attribute, so we get the XML serialization of given node (root) instead of the text content. +

      + +

      + In addition to this, the 2xml script does also formatting/indentation and syntax highlighting, + if given tools (xmllint and pygmentize) are available and the STDOUT is a terminal. +

      + +

      + This script is useful when writing the expressions for relpipe-in-*table, + but also as a pipeline filter that allows us to use the whole XML ecosystem also for other formats. + We can read YAML, JSON, INI, MIME or even some binary formats etc. and apply a XSLT transformation on such data and generate e.g. some XHTML report or a DocBook document, + or validate such structures using XSD or Relax NG schema or we can process such data using XQuery functional language. +

      + + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-rename-groups-backreferences.xml --- a/relpipe-data/examples-rename-groups-backreferences.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/examples-rename-groups-backreferences.xml Mon Feb 21 00:43:11 2022 +0100 @@ -12,8 +12,15 @@ This tool also support regex groups and backreferences. Thus we can use parts of the matched string in our replacement string:

      -

      Which would convert this:

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-rename-vg-fstab.xml --- a/relpipe-data/examples-rename-vg-fstab.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/examples-rename-vg-fstab.xml Mon Feb 21 00:43:11 2022 +0100 @@ -21,7 +21,11 @@

      @@ -35,7 +39,13 @@ For example we can put zeroes in both dump and pass attributes:

      - +

      n.b. the data types must be respected, we can not e.g. put abc in the pass attribute because it is declared as integer. diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-x11-basics.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-x11-basics.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,358 @@ + + + Exploring X11 windows and devices and emulating mouse movements and keystrokes + connect to X11 server and have some fun + 04900 + + + +

      + The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland). + This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice. +

      + +

      + Since + v0.18 we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools. + In this example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements). +

      + +

      Listing windows

      + +

      + In X11 almost everything is a window. Especially classic applications like XCalc are composed from many windows. + We list all windows of the current display this way: +

      + + + +

      and realize that there are so many windows (hundreds on a common desktop) with so much metadata (long names, titles etc.).

      + + + + +

      + Windows are identified by their id and organized hierarchically – have parent attribute that contains ID of their ancestor in the tree. + There is also the process attribute that contains the PID (process ID) of the running program that created this window. + However we should be aware that this information is voluntarily provided by the X11 clients, not managed by the X11 server, and thus might be missing (quite often) or even misleading (rarely). +

      + +

      + In order to not get lost in so many windows, we can run a separate X11 server and put only some programs/windows on its display: +

      + + + + +

      This classic calculator consist of just 51 windows:

      + + + + +

      Filtering windows with SQL and recursive CTE

      + +

      + Another way to deal with the abundance of windows is SQL, especially the recursive CTE (common table expressions). + This kind of SQL expressions allows us to work with tree structures and filter all descendants of given parent window. +

      + + + + +

      + When we run this script on a desktop with many windows and running XCalc, we will get a listing of the only 51 windows – similar to the listing above. + The root window (screen) is not included in this case, but we can also add it using another UNION ALL. +

      + + +

      + Thanks to modular design we can use also other transformation tools and languages – e.g. Scheme (relpipe-tr-scheme), AWK (relpipe-tr-awk) or XPath (relpipe-tr-xpath). +

      + +

      Embedding and composing windows

      + +

      + Once we know the ID of a particular window, we can have some fun and run another application inside it: +

      + + + +

      + So we can e.g. watch a film or enjoy a screen saver while calculating: +

      + + + +

      And we can setup everything automatically whithout typing the IDs by hand:

      + + = 5' \ + --output-attribute 'wid' integer 'id' \ + | relpipe-out-nullbyte | while read_nullbyte wid; do + sleep 1; + /usr/lib/xscreensaver/euphoria --regular -window-id "$wid" & + done]]> + +

      and experience pure euphoria:

      + + + +

      + This X11 feature can be used also in much more useful way for composing GUI from several separate programs that might be even written in different programming languages. +

      + + +

      Listing input devices

      + +

      In order to list our keyboards, mice and similar devices we will invoke:

      + + + +

      or we can explicitly say --list-input-devices true; in both cases we will get a listing similar to this one:

      + + + +

      Device IDs can be used to identify the source of input events and their filtering.

      + +

      Capturing input events

      + +

      + Once we connect to an X11 server (through the relpipe-in-x11 X11 client in our case) + we can listen to the input events and monitor the key codes and mouse movements. +

      + + relpipe-in-x11 --list-input-events true | relpipe-out-csv + +

      + To see the events immediatelly, it is important to use an output filter that does no buffering (e.g. relpipe-out-csv or relpipe-out-gui). +

      + + + +

      + In order to capture events from a different X11 server, we set the DISPLAY environment variable. + This way, we can also capture events remotely over SSH. +

      + + ssh example.com DISPLAY=:0 relpipe-in-x11 --list-input-events true | relpipe-out-csv + +

      + We can log the events to a file or forward them to another X11 server through relpipe-out-x11. + Or we can multiplex the events and forward them to several X11 servers e.g. to run simultaneous tests of different versions or configurations or certain GUI application. +

      + + +

      Simulating input events

      + +

      + With the relpipe-out-x11 client we can connect to an X11 server and emit input events like they come from a keyboard or mouse + (actually they would come from a virtual/simulated one). + This command reads relational data with same structure as produced by its counterpart relpipe-in-x11. + We can use data captured earlier or create some new one using any tool like + relpipe-in-cli or other relpipe-in-* optionally transformed through a relpipe-tr-* filter. + We may also translate other signals like MIDI events to X11 ones (relpipe-in-jack). +

      + +

      + For example, the following command moves the cursor to the coordinates 100,200 and then to 640,480. + Such events might be produced also in a shell loop and passed through the pipe to a single relpipe-out-x11 process + (i.e. without connecting and disconnecting the X11 server on each event). +

      + + + +

      + The relpipe-out-x11 tool has two options: + --dry-run true that suppresses the events and just tests validity of the input; and + --debug true that prints the events in a text form (XML) on STDERR (so we can see that something comes through). +

      + + relpipe-in-x11 --list-input-events true | relpipe-out-x11 --dry-run true --debug true + +

      + Using SSH, DDC, relpipe-in-x11 and relpipe-out-x11, + we can build a software replacement of a KVM switch and control several computers from one seat (keyboard, video, mouse). +

      + + + + diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-xpath-filtering-transforming.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-xpath-filtering-transforming.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,202 @@ + + + Filtering and transforming relational data with XPath + do simple restrictions and projections using a well-established query language + 04700 + + + +

      + In v0.18 we got a new powerful language for filtering and transformations: XPath. + It is now part of the toolset consisting of SQL, AWK, Scheme and others. + However XPath is originally a language designed for XML, in we can use it for relational data coming from various sources, not only XML, + and also for data that violates the rules of normal forms. + We can process quite complex tree structures entangled in records but we can also write simple and intuitive expressions like x = "a" or y = 123. +

      + + +

      Basic filtering

      + +

      Let us have some CSV data:

      + + +

      That look like this formatted as a table:

      + + + +

      Attributes of particular relations are available in XPath under their names, so we can directly reference them in our queries:

      + = 1980 and year < 1990' \ + | relpipe-out-tabular]]> + +

      filtered result:

      + + +

      + n.b. If there were any characters that are not valid XML name, they would be escaped in the same way as relpipe-in-*table commands do it + i.e. by adding underscores and unicode codepoints of given characters – e.g. the weird:field attribute will be available as weird_3a_field in XPath. +

      + + +

      Filtering records with tree structures

      + +

      + The CSV above is not a best example of data modeling. + Actually, it is quite terrible. + But in the real world, we often have to deal with such data – either work with them directly or give them some better shape before we start doing our job. +

      + +

      + Usually the best way is to normalize the model – follow the rules of Normal forms. + In this case, we would break this denormalized CSV table into several relations: + film, director, screenwriter… + or rather film, role, person, film_person_role… +

      + +

      + But for now, we will keep the data denormalized and just give them a better and machine-readable structure instead of limited and ambiguous notation of screenwriter = name1 + name2 + (that makes trouble when the value contains certain characters and requires writing a parser for never-specified syntax). + So, we will keep some data in classic relational attributes and some in nested XML structure. + This approach allows us to combine rigid attributes with free-form rich tree structures. +

      + + + +

      + The relpipe-tr-xpath seamlessly integrates the schema-backed (year) and schema-free (metadata/film) parts of our data model. + We use the same language syntax and principles for both kinds of attributes: +

      + + + + +

      Filtered result:

      + + + +

      + n.b. In current version, we have to mark the attributes containing XML: --xml-attribute 'metadata'. + In later versions, there will be a dedicated data type for XML, so these hints will not be necessary. +

      + +

      + This way, we can work with free-form attributes containing multiple values or run various functions on them. + We can e.g. list films that have more than one screenwriter: +

      + + --where 'count(metadata/film/screenwriter) > 1' + +

      Well, well… here we are:

      + + + +

      + We can also run XPath from SQL queries (relpipe-tr-sql) e.g. in PostgreSQL. +

      + + + + +

      Adding new attributes and transforming data

      + +

      + The relpipe-tr-xpath does not only restriction but also projection. + It can add, remove or modify the attributes while converting the input to the result set. +

      + + + +

      We removed some attributes and created new ones:

      + + + +

      Or we may concatenate the values:

      + + + + +

      and build some sentences:

      + + +

      Exctracting values from multiple XML files

      + +

      + Input data may come not only from some kind of database or some carefully designed data set, + they may be e.g. scattered on our filesystem in some already defined file format never intended for use as a database… + despite this fact, we can still collect and query such data in a relational way. +

      + +

      + For example, Maven (a build system for Java) describe its modules in XML format in pom.xml files. + Using the find and relpipe-in-filesystem we collect them and create a relation containing names and contents of such files: +

      + + + + +

      Then we extract desired values using relpipe-tr-xpath and get:

      + + +

      + This way we can harvest useful values from XML files – and not only XML files, also from various alternative formats, after we convert them (on-the-fly) to XML. + Such conversions are already available for formats like INI, ASN.1, MIME, HTML JSON, YAML etc. +

      + + +

      Post scriptum

      + +

      + The abovementioned combination of classic relational attributes with free-form XML structures is definitely not a design of first choice. + But sometimes it makes sense and sometimes we have to work with data not designed by us and need some tools to deal with them. + When we are designing the data model ourselves, we should always pursue the normalized form …and break the rules only if we have really good reason to do so. +

      + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-yaml.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-yaml.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,79 @@ + + + Reading and writing YAML + convert relations to YAML or YAML/JSON to relations + 05200 + + + +

      + YAML is a text format for serializing tree structures into text (like XML). + It can be sometimes tricky, but basic scenarios are intuitive and straightforward. + We can use YAML to describe relational data and load them using the relpipe-in-yaml input filter: +

      + + +

      This YAML notation is almost as simple as Recfiles and like Recfiles it can be easily written by hand.

      + +

      + We can also use the JSON notation (it can be read by the same tool – relpipe-in-json is currently just a symlink to relpipe-in-yaml). + JSON is inferior to YAML and XML, but the JSON input filter might be useful when interacting with software that can produce only JSON. + The film relation would look like this in JSON format (shortened): +

      + + +

      + When we want to declare data types + – which is usually a good idea – + we add the attribute-metadata header and move the data under the record node: +

      + +

      + This form (with data types and quotes) is also produced by the relpipe-out-yaml tool. +

      + + +

      When we load such YAML, we get this relational data:

      + +

      Of course, there might be multiple relations, not only one.

      + + +

      + Once we convert the YAML to relational format, we can run any transformations and conversions as usual. + For example we can use SQL to sort the recordsy by year: +

      + + relpipe-data/examples/film-1.rec]]> + +

      and Recfile output filter to get:

      + + +

      Or we may use XPath filter:

      + = 1980 and year < 1990']]> +

      to get the 80s films:

      + + +

      YAML comes into play when we are looking for a friendly format for user-supplied data.

      + +       + +       diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/barcode-qr-IMG_5758.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/barcode-qr-IMG_5758.txt Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,17 @@ +symbol: + ╭──────────────┬───────────────┬─────────────────────────────────────────────────────┬─────────────┬─────────────┬─────────────────┬──────────────────╮ + │ id (integer) │ type (string) │ value (string) │ x (integer) │ y (integer) │ width (integer) │ height (integer) │ + ├──────────────┼───────────────┼─────────────────────────────────────────────────────┼─────────────┼─────────────┼─────────────────┼──────────────────┤ + │ 0 │ QR-Code │ Every day, once a day, give yourself a present. │ 2100 │ 791 │ 411 │ 410 │ + │ 1 │ QR-Code │ Does Barry Manilow know that you raid his wardrobe? │ 2844 │ 859 │ 375 │ 369 │ + │ 2 │ QR-Code │ I'm in love. – That could be a problem. │ 2384 │ 2354 │ 320 │ 313 │ + │ 3 │ QR-Code │ Grow up Heather, bulimia is so '87. │ 1625 │ 1003 │ 366 │ 365 │ + │ 4 │ QR-Code │ Trust no 1. │ 4560 │ 917 │ 401 │ 384 │ + │ 5 │ QR-Code │ Damn good coffee. │ 3738 │ 1775 │ 358 │ 348 │ + │ 6 │ QR-Code │ Did you say a cherry or Coke slushie? │ 2946 │ 1784 │ 309 │ 310 │ + │ 7 │ QR-Code │ The truth is out there. │ 4271 │ 285 │ 314 │ 301 │ + │ 8 │ EAN-13 │ 9781593272203 │ 4254 │ 711 │ 138 │ 390 │ + │ 9 │ EAN-13 │ 9788025712979 │ 2110 │ 1691 │ 481 │ 194 │ + │ 10 │ EAN-13 │ 9788074329579 │ 2134 │ 1285 │ 314 │ 231 │ + ╰──────────────┴───────────────┴─────────────────────────────────────────────────────┴─────────────┴─────────────┴─────────────────┴──────────────────╯ +Record count: 11 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1-no-types.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1-no-types.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +film: + ╭──────────────────────┬───────────────┬─────────────────────╮ + │ title (string) │ year (string) │ director (string) │ + ├──────────────────────┼───────────────┼─────────────────────┤ + │ Beetlejuice │ 1988 │ Tim Burton │ + │ Vacation │ 1983 │ Harold Ramis │ + │ Christmas Vacation │ 1989 │ Jeremiah S. Chechik │ + │ Celebrity │ 1998 │ Woody Allen │ + │ Great Balls of Fire! │ 1989 │ Jim McBride │ + │ Flatliners │ 1990 │ Joel Schumacher │ + │ Heathers │ 1988 │ Michael Lehmann │ + │ Blue Velvet │ 1986 │ David Lynch │ + ╰──────────────────────┴───────────────┴─────────────────────╯ +Record count: 8 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1-no-types.yaml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1-no-types.yaml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,25 @@ +film: + - title: Beetlejuice + year: 1988 + director: Tim Burton + - title: Vacation + year: 1983 + director: Harold Ramis + - title: Christmas Vacation + year: 1989 + director: Jeremiah S. Chechik + - title: Celebrity + year: 1998 + director: Woody Allen + - title: Great Balls of Fire! + year: 1989 + director: Jim McBride + - title: Flatliners + year: 1990 + director: Joel Schumacher + - title: Heathers + year: 1988 + director: Michael Lehmann + - title: Blue Velvet + year: 1986 + director: David Lynch diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1.csv --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1.csv Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,9 @@ +"title::string","year::integer","director::string","screenwriter::string" +Beetlejuice,1988,Tim Burton,Warren Skaaren + Michael McDowell +Vacation,1983,Harold Ramis,John Hughes +Christmas Vacation,1989,Jeremiah S. Chechik,John Hughes +Celebrity,1998,Woody Allen,Woody Allen +Great Balls of Fire!,1989,Jim McBride,Jack Baran + Jim McBride +Flatliners,1990,Joel Schumacher,Peter Filardi +Heathers,1988,Michael Lehmann,Daniel Waters +Blue Velvet,1986,David Lynch,David Lynch diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1.filtered-1.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1.filtered-1.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,12 @@ +csv: + ╭──────────────────────┬────────────────┬─────────────────────┬───────────────────────────────────╮ + │ title (string) │ year (integer) │ director (string) │ screenwriter (string) │ + ├──────────────────────┼────────────────┼─────────────────────┼───────────────────────────────────┤ + │ Beetlejuice │ 1988 │ Tim Burton │ Warren Skaaren + Michael McDowell │ + │ Vacation │ 1983 │ Harold Ramis │ John Hughes │ + │ Christmas Vacation │ 1989 │ Jeremiah S. Chechik │ John Hughes │ + │ Great Balls of Fire! │ 1989 │ Jim McBride │ Jack Baran + Jim McBride │ + │ Heathers │ 1988 │ Michael Lehmann │ Daniel Waters │ + │ Blue Velvet │ 1986 │ David Lynch │ David Lynch │ + ╰──────────────────────┴────────────────┴─────────────────────┴───────────────────────────────────╯ +Record count: 6 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1.filtered-2.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1.filtered-2.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,12 @@ +film: + ╭──────────────────────┬────────────────┬─────────────────────╮ + │ title (string) │ year (integer) │ director (string) │ + ├──────────────────────┼────────────────┼─────────────────────┤ + │ Beetlejuice │ 1988 │ Tim Burton │ + │ Vacation │ 1983 │ Harold Ramis │ + │ Christmas Vacation │ 1989 │ Jeremiah S. Chechik │ + │ Great Balls of Fire! │ 1989 │ Jim McBride │ + │ Heathers │ 1988 │ Michael Lehmann │ + │ Blue Velvet │ 1986 │ David Lynch │ + ╰──────────────────────┴────────────────┴─────────────────────╯ +Record count: 6 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1.rec --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1.rec Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,35 @@ +%rec: film +%type: year int + +title: Vacation +year: 1983 +director: Harold Ramis + +title: Blue Velvet +year: 1986 +director: David Lynch + +title: Beetlejuice +year: 1988 +director: Tim Burton + +title: Heathers +year: 1988 +director: Michael Lehmann + +title: Christmas Vacation +year: 1989 +director: Jeremiah S. Chechik + +title: Great Balls of Fire! +year: 1989 +director: Jim McBride + +title: Flatliners +year: 1990 +director: Joel Schumacher + +title: Celebrity +year: 1998 +director: Woody Allen + diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +film: + ╭──────────────────────┬────────────────┬─────────────────────╮ + │ title (string) │ year (integer) │ director (string) │ + ├──────────────────────┼────────────────┼─────────────────────┤ + │ Beetlejuice │ 1988 │ Tim Burton │ + │ Vacation │ 1983 │ Harold Ramis │ + │ Christmas Vacation │ 1989 │ Jeremiah S. Chechik │ + │ Celebrity │ 1998 │ Woody Allen │ + │ Great Balls of Fire! │ 1989 │ Jim McBride │ + │ Flatliners │ 1990 │ Joel Schumacher │ + │ Heathers │ 1988 │ Michael Lehmann │ + │ Blue Velvet │ 1986 │ David Lynch │ + ╰──────────────────────┴────────────────┴─────────────────────╯ +Record count: 8 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-1.yaml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-1.yaml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,33 @@ +"film": + "attribute-metadata": + - "name": "title" + "type": "string" + - "name": "year" + "type": "integer" + - "name": "director" + "type": "string" + "record": + - "title": "Beetlejuice" + "year": 1988 + "director": "Tim Burton" + - "title": "Vacation" + "year": 1983 + "director": "Harold Ramis" + - "title": "Christmas Vacation" + "year": 1989 + "director": "Jeremiah S. Chechik" + - "title": "Celebrity" + "year": 1998 + "director": "Woody Allen" + - "title": "Great Balls of Fire!" + "year": 1989 + "director": "Jim McBride" + - "title": "Flatliners" + "year": 1990 + "director": "Joel Schumacher" + - "title": "Heathers" + "year": 1988 + "director": "Michael Lehmann" + - "title": "Blue Velvet" + "year": 1986 + "director": "David Lynch" diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-2.csv --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-2.csv Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,9 @@ +"title::string","year::integer","metadata::string" +Beetlejuice,1988,Tim BurtonWarren SkaarenMichael McDowell +Vacation,1983,Harold RamisJohn Hughes +Christmas Vacation,1989,Jeremiah S. ChechikJohn Hughes +Celebrity,1998,Woody AllenWoody Allen +Great Balls of Fire!,1989,Jim McBrideJack BaranJim McBride +Flatliners,1990,Joel SchumacherPeter Filardi +Heathers,1988,Michael LehmannDaniel Waters +Blue Velvet,1986,David LynchDavid Lynch diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-2.filtered-1.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-2.filtered-1.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,9 @@ +film: + ╭────────────────────┬────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────╮ + │ title (string) │ year (integer) │ metadata (string) │ + ├────────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ + │ Vacation │ 1983 │ Harold RamisJohn Hughes │ + │ Christmas Vacation │ 1989 │ Jeremiah S. ChechikJohn Hughes │ + │ Blue Velvet │ 1986 │ David LynchDavid Lynch │ + ╰────────────────────┴────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────╯ +Record count: 3 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-2.filtered-2.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-2.filtered-2.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,8 @@ +film: + ╭──────────────────────┬────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ + │ title (string) │ year (integer) │ metadata (string) │ + ├──────────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ + │ Beetlejuice │ 1988 │ Tim BurtonWarren SkaarenMichael McDowell │ + │ Great Balls of Fire! │ 1989 │ Jim McBrideJack BaranJim McBride │ + ╰──────────────────────┴────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +Record count: 2 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-2.filtered-3.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-2.filtered-3.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +film: + ╭──────────────────────┬─────────────────────┬──────────────────────────────╮ + │ title (string) │ director (string) │ screenwriter_count (integer) │ + ├──────────────────────┼─────────────────────┼──────────────────────────────┤ + │ Beetlejuice │ Tim Burton │ 2 │ + │ Vacation │ Harold Ramis │ 1 │ + │ Christmas Vacation │ Jeremiah S. Chechik │ 1 │ + │ Celebrity │ Woody Allen │ 1 │ + │ Great Balls of Fire! │ Jim McBride │ 2 │ + │ Flatliners │ Joel Schumacher │ 1 │ + │ Heathers │ Michael Lehmann │ 1 │ + │ Blue Velvet │ David Lynch │ 1 │ + ╰──────────────────────┴─────────────────────┴──────────────────────────────╯ +Record count: 8 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-2.filtered-4.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-2.filtered-4.txt Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,8 @@ +The film Beetlejuice was directed by Tim Burton in year 1988. +The film Vacation was directed by Harold Ramis in year 1983. +The film Christmas Vacation was directed by Jeremiah S. Chechik in year 1989. +The film Celebrity was directed by Woody Allen in year 1998. +The film Great Balls of Fire! was directed by Jim McBride in year 1989. +The film Flatliners was directed by Joel Schumacher in year 1990. +The film Heathers was directed by Michael Lehmann in year 1988. +The film Blue Velvet was directed by David Lynch in year 1986. diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/film-2.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/film-2.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +film: + ╭──────────────────────┬────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ + │ title (string) │ year (integer) │ metadata (string) │ + ├──────────────────────┼────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ + │ Beetlejuice │ 1988 │ Tim BurtonWarren SkaarenMichael McDowell │ + │ Vacation │ 1983 │ Harold RamisJohn Hughes │ + │ Christmas Vacation │ 1989 │ Jeremiah S. ChechikJohn Hughes │ + │ Celebrity │ 1998 │ Woody AllenWoody Allen │ + │ Great Balls of Fire! │ 1989 │ Jim McBrideJack BaranJim McBride │ + │ Flatliners │ 1990 │ Joel SchumacherPeter Filardi │ + │ Heathers │ 1988 │ Michael LehmannDaniel Waters │ + │ Blue Velvet │ 1986 │ David LynchDavid Lynch │ + ╰──────────────────────┴────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +Record count: 8 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/html-tagsoup-1.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/html-tagsoup-1.sh Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,63 @@ +#!/bin/bash + +HTML=' + +

      Our company is focused on: +

        +
      • video game arcades +
      • laundry
        • +
      • cigarette machines and trucking +
      • personal loans and politics +
      + + + +

      Visit our front page and check the news!!! + + + +

      +download free MP3 now +

      + + + +'; + +fetch_html() { + # there might be a wget/curl call to download a fresh version of the web page + echo "$HTML"; +} + + +extract_relations() { + relpipe-in-htmltable \ + --relation 'field_of_business' \ + --records '//li' \ + --attribute 'priority' integer 'count(preceding::li)+1' \ + --attribute 'name' string '.' \ + --attribute 'normalized' string 'normalize-space(.)' \ + --relation 'hyperlink' \ + --records '//a' \ + --attribute 'url' string '@href' \ + --attribute 'name' string '.' \ + --attribute 'xpath' string '.' --mode xpath \ + --relation 'download_token' \ + --records '//a[@class="mp3"]' \ + --attribute 'value' string 'substring(@href, 15)' \ + --relation 'hidden_footer' \ + --records '//comment()[count(following::*)=0]' \ + --attribute 'text' string 'normalize-space(.)' +} + +format_result() { [[ -t 1 ]] && relpipe-out-tabular || cat; } + + +# fetch_html | html2xml 1>&2 + +fetch_html | extract_relations | format_result diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/html-tagsoup-1.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/html-tagsoup-1.txt Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,36 @@ +field_of_business: + ╭────────────────────┬──────────────────────────────────┬─────────────────────────────────╮ + │ priority (integer) │ name (string) │ normalized (string) │ + ├────────────────────┼──────────────────────────────────┼─────────────────────────────────┤ + │ 1 │ video game arcades↲ │ video game arcades │ + │ 2 │ laundry │ laundry │ + │ 3 │ cigarette machines and trucking↲ │ cigarette machines and trucking │ + │ 4 │ personal loans and politics↲ │ personal loans and politics │ + ╰────────────────────┴──────────────────────────────────┴─────────────────────────────────╯ +Record count: 4 + +hyperlink: + ╭────────────────────────────────────────────────────┬───────────────┬──────────────────────╮ + │ url (string) │ name (string) │ xpath (string) │ + ├────────────────────────────────────────────────────┼───────────────┼──────────────────────┤ + │ index.htm │ front page │ /html/body/p[2]/a[1] │ + │ news.php │ news │ /html/body/p[2]/a[2] │ + │ mp3.cgi?token=61686f6a-e1ab-4b92-c357-474e552f4c69 │ free MP3 │ /html/body/p[3]/a │ + ╰────────────────────────────────────────────────────┴───────────────┴──────────────────────╯ +Record count: 3 + +download_token: + ╭──────────────────────────────────────╮ + │ value (string) │ + ├──────────────────────────────────────┤ + │ 61686f6a-e1ab-4b92-c357-474e552f4c69 │ + ╰──────────────────────────────────────╯ +Record count: 1 + +hidden_footer: + ╭─────────────────────────────────────────────────────────────────────────────────────────────╮ + │ text (string) │ + ├─────────────────────────────────────────────────────────────────────────────────────────────┤ + │ Best viewed in Netscape Navigator and resolution 800×600 (c) Fvkgrra Pnaqyrf ltd. 1984-2022 │ + ╰─────────────────────────────────────────────────────────────────────────────────────────────╯ +Record count: 1 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/ini-complex.full.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/ini-complex.full.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,55 @@ +ini: + ╭────────────────┬─────────────────┬──────────────────┬─────────────────────┬──────────────────┬──────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────┬─────────────────────╮ + │ line (integer) │ event (integer) │ section (string) │ key (string) │ sub_key (string) │ value (string) │ comment (string) │ whitespace (string) │ + ├────────────────┼─────────────────┼──────────────────┼─────────────────────┼──────────────────┼──────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┼─────────────────────┤ + │ 1 │ 1 │ │ global │ │ value │ │ │ + │ 2 │ 2 │ │ │ │ │ │ ↲ │ + │ 4 │ 4 │ section │ │ │ │ section tags are currently supported by the parser (do not generate error) │ │ + │ 5 │ 5 │ section │ │ │ │ but do not generate any output │ │ + │ 6 │ 6 │ section │ section-tag-support │ │ partial │ │ │ + │ 7 │ 7 │ section │ │ │ │ │ ↲ │ + │ 9 │ 9 │ quoting │ │ │ │ │ ↲ │ + │ 10 │ 10 │ quoting │ quoted │ │ some text in quotes │ │ │ + │ 11 │ 11 │ quoting │ apostrophed │ │ some text in apostrophes │ │ │ + │ 12 │ 12 │ quoting │ │ │ │ │ ↲ │ + │ 13 │ 13 │ quoting │ unquoted │ │ values are trimmed │ │ │ + │ 14 │ 14 │ quoting │ we-need │ │ quotes or apostrophes │ │ │ + │ 15 │ 15 │ quoting │ to-keep │ │ the leading/trailing whitespace │ │ │ + │ 16 │ 16 │ quoting │ │ │ │ │ ↲ │ + │ 18 │ 18 │ multiline │ │ │ │ │ ↲ │ + │ 19 │ 19 │ multiline │ quoted │ │ first line↲second line │ │ │ + │ 21 │ 20 │ multiline │ │ │ │ │ ↲ │ + │ 22 │ 21 │ multiline │ apostrophed │ │ first line↲second line │ │ │ + │ 24 │ 22 │ multiline │ │ │ │ │ ↲ │ + │ 25 │ 23 │ multiline │ │ │ │ unlike quoted/apostrophed strings, there is no line end inside the value: │ │ + │ 26 │ 24 │ multiline │ backslashed │ │ first line continues here │ │ │ + │ 28 │ 25 │ multiline │ │ │ │ │ ↲ │ + │ 29 │ 26 │ comments │ │ │ │ we can comment also secions │ │ + │ 30 │ 27 │ comments │ │ │ │ │ ↲ │ + │ 31 │ 28 │ comments │ ini │ │ true │ and even values │ │ + │ 32 │ 29 │ comments │ unix │ │ true │ │ │ + │ 33 │ 30 │ comments │ │ │ │ │ ↲ │ + │ 34 │ 31 │ comments │ │ │ │ classic INI comment │ │ + │ 35 │ 32 │ comments │ │ │ │ unix-style comments are also supported │ │ + │ 36 │ 33 │ comments │ │ │ │ │ ↲ │ + │ 37 │ 34 │ comments │ but │ │ be aware that ; this is part of the value, not a comment │ │ │ + │ 38 │ 35 │ comments │ │ │ │ we need '' or "" to put comment on the same line as the value │ │ + │ 39 │ 36 │ comments │ │ │ │ │ ↲ │ + │ 41 │ 38 │ subkeys │ │ │ │ │ ↲ │ + │ 42 │ 39 │ subkeys │ │ │ │ we may specify subkeys in [] brackets: │ │ + │ 43 │ 40 │ subkeys │ a │ x │ AX │ │ │ + │ 44 │ 41 │ subkeys │ a │ y │ AY │ │ │ + │ 45 │ 42 │ subkeys │ │ │ │ │ ↲ │ + │ 46 │ 43 │ subkeys │ │ │ │ because this feature is quite uncommon, it is disabled by default │ │ + │ 47 │ 44 │ subkeys │ │ │ │ and whole key+subkey becomes a key │ │ + │ 48 │ 45 │ subkeys │ │ │ │ but we can turn on the subkey support by the CLI option: --enable-sub-keys true │ │ + │ 49 │ 46 │ subkeys │ │ │ │ and get separete key and sub_key attributes on the output │ │ + │ 50 │ 47 │ subkeys │ │ │ │ │ ↲ │ + │ 52 │ 49 │ escaping │ │ │ │ │ ↲ │ + │ 53 │ 50 │ escaping │ escaped-multiline │ │ first line↲second line │ │ │ + │ 54 │ 51 │ escaping │ backslash │ │ back\slash │ │ │ + │ 55 │ 52 │ escaping │ quotes │ │ quoted text with "quotes" inside │ │ │ + │ 56 │ 53 │ escaping │ tab │ │ how↹very │ │ │ + │ 57 │ 54 │ escaping │ │ │ │ │ ↲↲ │ + ╰────────────────┴─────────────────┴──────────────────┴─────────────────────┴──────────────────┴──────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┴─────────────────────╯ +Record count: 49 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/ini-complex.ini --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/ini-complex.ini Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,58 @@ +global=value + +[section][section-tag] +; section tags are currently supported by the parser (do not generate error) +; but do not generate any output +section-tag-support=partial + +[quoting] + +quoted = "some text in quotes" +apostrophed = 'some text in apostrophes' + +unquoted = values are trimmed +we-need = " quotes or apostrophes" +to-keep = 'the leading/trailing whitespace ' + +[multiline] + +quoted = "first line +second line" + +apostrophed = 'first line +second line' + +; unlike quoted/apostrophed strings, there is no line end inside the value: +backslashed = first line \ +continues here + +[comments] ; we can comment also secions + +ini = "true" ; and even values +unix = true + +; classic INI comment +# unix-style comments are also supported + +but = be aware that ; this is part of the value, not a comment +; we need '' or "" to put comment on the same line as the value + +[subkeys] + +; we may specify subkeys in [] brackets: +a[x] = AX +a[y] = AY + +; because this feature is quite uncommon, it is disabled by default +; and whole key+subkey becomes a key +; but we can turn on the subkey support by the CLI option: --enable-sub-keys true +; and get separete key and sub_key attributes on the output + +[escaping] + +escaped-multiline = first line\nsecond line +backslash = back\\slash +quotes = "quoted text with \"quotes\" inside" +tab = how\tvery + + diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/ini-complex.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/ini-complex.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,25 @@ +ini: + ╭──────────────────┬─────────────────────┬──────────────────┬──────────────────────────────────────────────────────────╮ + │ section (string) │ key (string) │ sub_key (string) │ value (string) │ + ├──────────────────┼─────────────────────┼──────────────────┼──────────────────────────────────────────────────────────┤ + │ │ global │ │ value │ + │ section │ section-tag-support │ │ partial │ + │ quoting │ quoted │ │ some text in quotes │ + │ quoting │ apostrophed │ │ some text in apostrophes │ + │ quoting │ unquoted │ │ values are trimmed │ + │ quoting │ we-need │ │ quotes or apostrophes │ + │ quoting │ to-keep │ │ the leading/trailing whitespace │ + │ multiline │ quoted │ │ first line↲second line │ + │ multiline │ apostrophed │ │ first line↲second line │ + │ multiline │ backslashed │ │ first line continues here │ + │ comments │ ini │ │ true │ + │ comments │ unix │ │ true │ + │ comments │ but │ │ be aware that ; this is part of the value, not a comment │ + │ subkeys │ a │ x │ AX │ + │ subkeys │ a │ y │ AY │ + │ escaping │ escaped-multiline │ │ first line↲second line │ + │ escaping │ backslash │ │ back\slash │ + │ escaping │ quotes │ │ quoted text with "quotes" inside │ + │ escaping │ tab │ │ how↹very │ + ╰──────────────────┴─────────────────────┴──────────────────┴──────────────────────────────────────────────────────────╯ +Record count: 19 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/ini-simple.ini --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/ini-simple.ini Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +; this entry is a „global property“ – does not belong to any section: +color=blue + +[first-section] +x = 10 +y = 20 +z = 30 +some = value +other = VALUE + +[second-section] +x = 100 +y = 200 +z = 300 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/ini-simple.modified.ini --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/ini-simple.modified.ini Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +; this entry is a „global property“ – does not belong to any section: +color = blue + +[first-section] +x = 256 +y = 20 +z = 30 +some = value +other = VALUE + +[second-section] +x = 100 +y = 200 +z = 300 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/ini-simple.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/ini-simple.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,15 @@ +ini: + ╭──────────────────┬──────────────┬────────────────╮ + │ section (string) │ key (string) │ value (string) │ + ├──────────────────┼──────────────┼────────────────┤ + │ │ color │ blue │ + │ first-section │ x │ 10 │ + │ first-section │ y │ 20 │ + │ first-section │ z │ 30 │ + │ first-section │ some │ value │ + │ first-section │ other │ VALUE │ + │ second-section │ x │ 100 │ + │ second-section │ y │ 200 │ + │ second-section │ z │ 300 │ + ╰──────────────────┴──────────────┴────────────────╯ +Record count: 9 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/maven-filesystem-xpath.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/maven-filesystem-xpath.sh Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,14 @@ +#!/bin/bash + +# replaces multiple: --option 'env:xmlns_m' 'http://maven.apache.org/POM/4.0.0' +export xmlns_m='http://maven.apache.org/POM/4.0.0' + +find -name pom.xml -print0 \ + | relpipe-in-filesystem \ + --relation 'maven' \ + --file path --as 'module' \ + --script xpath --option 'env:xpath' '/m:project/m:groupId' --as 'groupId' \ + --script xpath --option 'env:xpath' '/m:project/m:artifactId' --as 'artifactId' \ + --script xpath --option 'env:xpath' '/m:project/m:version' --as 'version' \ + --hash sha256 \ + | relpipe-out-tabular diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/netplan-1.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/netplan-1.sh Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,28 @@ +#!/bin/bash + +cat netplan.yaml \ + | relpipe-in-yamltable \ + --relation 'nic' \ + --records '/yaml/network/ethernets/*|/yaml/network/bridges/*' \ + --attribute 'name' string 'name()' \ + --attribute 'type' string 'name(..)' \ + --attribute 'dhcp4' boolean 'dhcp4' \ + --attribute 'dhcp6' boolean 'dhcp6' \ + --attribute 'accept-ra' boolean 'accept-ra' \ + --relation 'bridge' \ + --records '/yaml/network/bridges/*' \ + --attribute 'name' string 'name()' \ + --attribute 'gateway4' string 'gateway4' \ + --attribute 'mtu' string 'mtu' \ + --relation 'bridge_interface' \ + --records '/yaml/network/bridges/*/interfaces' \ + --attribute 'bridge' string 'name(..)' \ + --attribute 'interface' string '.' \ + --relation 'ip' \ + --records '/yaml/network/bridges/*/addresses' \ + --attribute 'address' string 'substring-before(., "/")' \ + --attribute 'mask' string 'substring-after(., "/")' \ + --attribute 'interface' string 'name(..)' \ + | relpipe-out-tabular + +# an early draft of the relational mapping diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/netplan-1.yaml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/netplan-1.yaml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,26 @@ +network: + version: 2 + + ethernets: + enp84s0: + dhcp4: false + dhcp6: false + accept-ra: false + + bridges: + br0: + interfaces: [enp84s0, eth0] + addresses: + - 192.168.1.101/24 + gateway4: 192.168.1.1 + nameservers: + addresses: + - 192.168.1.10 + - 192.168.1.11 + mtu: 1500 + parameters: + stp: true + forward-delay: 4 + dhcp4: false + dhcp6: false + accept-ra: false diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/release-v0.18.sh --- a/relpipe-data/examples/release-v0.18.sh Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/examples/release-v0.18.sh Mon Feb 21 00:43:11 2022 +0100 @@ -13,14 +13,14 @@ RELPIPE_DEPENDENCIES+=(libzbar-dev libmagick++-dev) # needed only for relpipe-in-barcode and the barcode-reader streamlet RELPIPE_DEPENDENCIES+=(libgumbo-dev) # needed only for relpipe-in-htmltable RELPIPE_DEPENDENCIES+=(libyaml-dev) # needed only for relpipe-in-yaml, relpipe-in-yamltable, relpipe-in-json and relpipe-in-jsontable + RELPIPE_DEPENDENCIES+=(libcbor-dev) # needed only for relpipe-in-cbortable RELPIPE_DEPENDENCIES+=(libvmime-dev) # needed only for relpipe-in-mimetable apt install "${RELPIPE_DEPENDENCIES[@]}" exit; fi # Run rest of installation as a non-root user: -export RELPIPE_VERSION="v0.18" -export RELPIPE_VERSION="tip" # FIXME: v0.18 +export RELPIPE_VERSION=${RELPIPE_VERSION:-"v0.18"} export RELPIPE_SRC=~/src export RELPIPE_BUILD=~/build export RELPIPE_INSTALL=~/install @@ -36,7 +36,7 @@ RELPIPE_LIBS=(lib-common lib-reader lib-writer lib-cli lib-xmlwriter) RELPIPE_TOOLS=(in-cli in-fstab in-xml in-xmltable in-csv in-filesystem in-recfile out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-sed tr-validator tr-scheme tr-awk tr-sql in-jack out-jack) -RELPIPE_TOOLS+=(in-asn1table in-barcode in-htmltable in-ini in-initable in-mimetable in-x11 in-yaml in-yamltable out-ini out-x11 out-yaml tr-xpath) # since v0.18 +RELPIPE_TOOLS+=(in-asn1 in-asn1table in-barcode in-cbortable in-htmltable in-ini in-initable in-mimetable in-x11 in-yaml in-yamltable out-ini out-x11 out-yaml tr-infertypes tr-xpath) # since v0.18 # RELPIPE_TOOLS+=(out-gui.qt tr-python) # not enabled by default @@ -65,5 +65,9 @@ unset -v RELPIPE_BUILD unset -v RELPIPE_INSTALL -# List available JACK ports (the JACK daemon should be running): -relpipe-in-jack --list-ports true --list-midi-messages false | relpipe-out-tabular +# List mice, keyboards and some fstab entries: +(relpipe-in-fstab ; relpipe-in-x11 2>/dev/null) \ + | relpipe-tr-xpath \ + --relation 'fstab' --where 'pass > 1' \ + --relation 'x11_input_device' --where 'type = "keyboard" or type = "mouse"' \ + | relpipe-out-tabular diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/runnable-jars.sh --- a/relpipe-data/examples/runnable-jars.sh Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/examples/runnable-jars.sh Mon Feb 21 00:43:11 2022 +0100 @@ -16,7 +16,7 @@ } shortenPath() { - relpipe-tr-sed '.*' 'path' "^$HOME/?" "~/"; + relpipe-tr-sed --relation '.*' --attribute 'path' --value "^$HOME/?" --replacement "~/"; } findFiles | fetchAttributes | filterRunable | shortenPath | relpipe-out-tabular diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/x509-gnu.org.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/x509-gnu.org.txt Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,82 @@ +validity: + ╭───────────────────────────┬───────────────────────────╮ + │ from (string) │ to (string) │ + ├───────────────────────────┼───────────────────────────┤ + │ 2021-12-10T13:52:28+00:00 │ 2022-03-10T13:52:27+00:00 │ + ╰───────────────────────────┴───────────────────────────╯ +Record count: 1 + +alternative_name: + ╭─────────────────────────────╮ + │ name (string) │ + ├─────────────────────────────┤ + │ archive.gnewsense.org │ + │ beta.gnewsense.org │ + │ bloodnok.gnewsense.org │ + │ bofh.gnewsense.org │ + │ bugs.gnewsense.org │ + │ bzr.gnewsense.org │ + │ cdimage.gnewsense.org │ + │ classpath.org │ + │ config.gnewsense.org │ + │ digitalspeech.org │ + │ donate.digitalspeech.org │ + │ dotgnu.org │ + │ eccles.gnewsense.org │ + │ emacs.org │ + │ glibc.gnu.org │ + │ gnewsense.org │ + │ gnu.org │ + │ gnukids.org │ + │ gplfaq.org │ + │ hurd.gnu.org │ + │ ipv6.nongnu.org │ + │ kindleswindle.org │ + │ nongnu.org │ + │ patch-tracker.gnewsense.org │ + │ playfreedom.org │ + │ playogg.com │ + │ playogg.net │ + │ playogg.org │ + │ rsync.gnewsense.org │ + │ seagoon.gnewsense.org │ + │ security.gnewsense.org │ + │ smalltalk.gnu.org │ + │ torrent.gnewsense.org │ + │ upgradefromwindows.com │ + │ upgradefromwindows.org │ + │ upgradefromwindows8.com │ + │ upgradefromwindows8.org │ + │ us.archive.gnewsense.org │ + │ vcdimager.org │ + │ wiki.gnewsense.org │ + │ wildebeest.ipv6.gnu.org │ + │ wildebeest1p.gnu.org │ + │ www.classpath.org │ + │ www.digitalspeech.org │ + │ www.dotgnu.org │ + │ www.emacs.org │ + │ www.gnewsense.org │ + │ www.gnu.org │ + │ www.gnukids.org │ + │ www.gplfaq.org │ + │ www.hurd.gnu.org │ + │ www.ipv6.gnu.org │ + │ www.ipv6.nongnu.org │ + │ www.kindleswindle.org │ + │ www.nongnu.org │ + │ www.playfreedom.org │ + │ www.playogg.com │ + │ www.playogg.net │ + │ www.playogg.org │ + │ www.upgradefromwindows.com │ + │ www.upgradefromwindows.org │ + │ www.upgradefromwindows8.com │ + │ www.upgradefromwindows8.org │ + │ www.vcdimager.org │ + │ www6.gnu.org │ + │ www6.nongnu.org │ + │ x86-32.gnewsense.org │ + │ x86-64.gnewsense.org │ + ╰─────────────────────────────╯ +Record count: 68 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/x509-gnu.org.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/x509-gnu.org.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,215 @@ + + + + + + 2 + + + + 1.2.840.113549.1.1.11 + + + + + + 2.5.4.6 + US + + + + + 2.5.4.10 + Let's Encrypt + + + + + 2.5.4.3 + R3 + + + + + 2021-12-10T13:52:28+00:00 + 2022-03-10T13:52:27+00:00 + + + + + 2.5.4.3 + wildebeest1p.gnu.org + + + + + + 1.2.840.113549.1.1.1 + + + 100000000000000010000000110000000100000010110010100110101010101011000010001101001001001101000111111011101100011100011110000011010000010000011010011110000101011001010011110101011111011110111011001001110001010101011111100111110101001001111001100001000101111100111000110111111010000111101010101110001111000001011000101100101111010010011111101101101100110000011110001101111100011011010010001001000100001100010011110101110011001110101010101111110011010011111110000010001101000100100011011111011010000010110001101101011000010111011101001110000011000001110001100100100001111100010101111000110011000100010000101011010010011101001101101100001000010011011011001010000011010110100001010111101011010110101001111110101011011011101110010110101111100111001101111010101111001101000100001001101100001101010111111110100111010011010001101101110111111111001110110010011110101011001110100111000100101101100101011100110001100001011000100000000001001100010001110100110001000000010011111101111111110000000010111100010101000010100111001110000011111001000110011001000111000010101001000000110101011100001110110001111000111010110101101011010110010110110010001100001010111001000111000001000001110011110010111110010101000011010101110110011001001000111101000111101111101100110010101000100111100110000010010111001110001010111101110000010101010011011001100011010011000111001010000000010110001001100011000010010010010110000001110011001011010100101101101111011100110100011111001010110111110101011111101101010111110110110000100001110101100101011011111100101101000100111111101110010100011011101010000100000100001010111111010110110010000100011011110101001101101110001001110000011101001000101001001101111101111100100001100011111101010001111001110111100100101100001110100010001001000101111100010001011010110010000111000000011011101101100010101100001110001110011100011100010110101101000101101100001100010101101110001010111000110000100110010011011101100000011000011101001111111111101100111011001011111001011010101010001101100000101111000001101110110110111101101011010000001101111001110011101110000010000110100001101100101101010101000000001000000010000000010000010100000001010000100000000100000100001100 + + + + + 2.5.29.15 + true + + 101 + + + + 2.5.29.37 + + + 1.3.6.1.5.5.7.3.1 + 1.3.6.1.5.5.7.3.2 + + + + + 2.5.29.19 + true + + + + + + 2.5.29.14 + + 9e05272100500e8f5139ae2e3ad87ea30ea08448 + + + + 2.5.29.35 + + + .....XV..P.@........ + + + + + 1.3.6.1.5.5.7.1.1 + + + + 1.3.6.1.5.5.7.48.1 + http://r3.o.lencr.org + + + 1.3.6.1.5.5.7.48.2 + http://r3.i.lencr.org/ + + + + + + 2.5.29.17 + + + archive.gnewsense.org + beta.gnewsense.org + bloodnok.gnewsense.org + bofh.gnewsense.org + bugs.gnewsense.org + bzr.gnewsense.org + cdimage.gnewsense.org + classpath.org + config.gnewsense.org + digitalspeech.org + donate.digitalspeech.org + dotgnu.org + eccles.gnewsense.org + emacs.org + glibc.gnu.org + gnewsense.org + gnu.org + gnukids.org + gplfaq.org + hurd.gnu.org + ipv6.nongnu.org + kindleswindle.org + nongnu.org + patch-tracker.gnewsense.org + playfreedom.org + playogg.com + playogg.net + playogg.org + rsync.gnewsense.org + seagoon.gnewsense.org + security.gnewsense.org + smalltalk.gnu.org + torrent.gnewsense.org + upgradefromwindows.com + upgradefromwindows.org + upgradefromwindows8.com + upgradefromwindows8.org + us.archive.gnewsense.org + vcdimager.org + wiki.gnewsense.org + wildebeest.ipv6.gnu.org + wildebeest1p.gnu.org + www.classpath.org + www.digitalspeech.org + www.dotgnu.org + www.emacs.org + www.gnewsense.org + www.gnu.org + www.gnukids.org + www.gplfaq.org + www.hurd.gnu.org + www.ipv6.gnu.org + www.ipv6.nongnu.org + www.kindleswindle.org + www.nongnu.org + www.playfreedom.org + www.playogg.com + www.playogg.net + www.playogg.org + www.upgradefromwindows.com + www.upgradefromwindows.org + www.upgradefromwindows8.com + www.upgradefromwindows8.org + www.vcdimager.org + www6.gnu.org + www6.nongnu.org + x86-32.gnewsense.org + x86-64.gnewsense.org + + + + + 2.5.29.32 + + + + 2.23.140.1.2.1 + + + 1.3.6.1.4.1.44947.1.1.1 + + + 1.3.6.1.5.5.7.2.1 + http://cps.letsencrypt.org + + + + + + + + 1.3.6.1.4.1.11129.2.4.2 + + 00f1007700dfa55eab68824f1f6cadeeb85f4e3e5aeacda212a46a5e8e3b12c020445c2a730000017da4d4c8e00000040300483046022100e459e6392e11771d2425d9ace43e194a9f1776ed5ec8f56db3db661b3a9ac77e022100d6fd9e62b74b86201d0aedc85e31a42fb3b6a3cbd46ef769de3bef49f0cf2b290076002979bef09e393921f056739f63a577e5be577d9c600af8f94d5d265c255dc7840000017da4d4c9540000040300473045022100de46ef3a2991ba373fef55fc6b03e0b813986e1393d5699f7f22ae84350d01d002202a205df263e45819c7d9afa7b0e2298b911414e5f940de94ec43c8cfba3e7659 + + + + + + + 1.2.840.113549.1.1.11 + + + 11111111011111011010000011000111111100100110100101111000100011011011101111101100100011110111000100000000010010101011010111010001100001110000010010011010110101000101000000011111101110000111111000010110001000011101101101110110101001001001100001111011111100111100110110100010111000100001101111011100100011100101100101011110111010010110000101011111100101100100011011011011000111110000011111000001001110111111111111110111110110111011001000110000111001010100111010010000001100111001011000001001110100110000111010000011100111011100010110011011101111100101101111111001010000000100110011110010000101101011011011001111010100000110000110100001111011001001000000001011110110100111000111100010111111110111111111001010010001101000111100001111101111001010010011101100010111001100010001111101101001101100110000001010000100101001110001000010101010111000000001110011001100110100010101100111111001010001111110000101011000001100110110101100000000100101000000101011010000101011000111110000111011000011001001000001000110111001100011111100110011110101111101000010101100010000110011101111111100100011111010001111110110110110100011011101111111110001000010011101000000001100101010010000010011101101101110100010101000011010011111100111111011011111011001111100010101110000100011100110000111111111110011011011110001101001001001110100001110001000001001111010100000001010011111011111001110101100101001011011110100111111010001010011110000111001110010101110101011011101101101001101010000100001111111110100011011000100011101000010001011001010110110101110011010010110001101110110111100111110111101000111011111010000110010110100011110001000101011100010110100011110010001101001010010101111100011100110011110110010100101100010010110000101100110110010100110100000110011010011101100001111111110110011010011100001011001010010110000110101011001000011001010111010100111011110011010011000000010010111111001011001101000011110100100011000100001010111010010010101011111101000011110000001011001110110010110010011100000001000000111110100110100011100011010100010110111110101100000110001000110101010 + + diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples/xpath-maven-1.tabular --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/xpath-maven-1.tabular Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,10 @@ +module: + ╭─────────────────────────────────────┬────────────────────────┬──────────────────────┬──────────────────╮ + │ path (string) │ group_id (string) │ artifact_id (string) │ version (string) │ + ├─────────────────────────────────────┼────────────────────────┼──────────────────────┼──────────────────┤ + │ ./java/jdbc-dk-driver/pom.xml │ info.globalcode.sql.dk │ jdbc-dk-driver │ 0.11-SNAPSHOT │ + │ ./java/jdbc-loopback-driver/pom.xml │ info.globalcode.sql.dk │ jdbc-loopback-driver │ 0.11-SNAPSHOT │ + │ ./java/sql-dk/pom.xml │ info.globalcode.sql.dk │ sql-dk │ 0.11-SNAPSHOT │ + │ ./java/pom.xml │ info.globalcode.sql.dk │ sql-dk-parent │ 0.11-SNAPSHOT │ + ╰─────────────────────────────────────┴────────────────────────┴──────────────────────┴──────────────────╯ +Record count: 4 diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/img/barcode-qr-IMG_5758.jpeg Binary file relpipe-data/img/barcode-qr-IMG_5758.jpeg has changed diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/img/csv-sql-gui-ip-address-counts.png Binary file relpipe-data/img/csv-sql-gui-ip-address-counts.png has changed diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/img/wmaker-yaml-xml-tabular-1.png Binary file relpipe-data/img/wmaker-yaml-xml-tabular-1.png has changed diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/img/xcalc-x11-embedding-1.png Binary file relpipe-data/img/xcalc-x11-embedding-1.png has changed diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/img/xcalc-x11-embedding-2.png Binary file relpipe-data/img/xcalc-x11-embedding-2.png has changed diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/implementation.xml --- a/relpipe-data/implementation.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/implementation.xml Mon Feb 21 00:43:11 2022 +0100 @@ -14,14 +14,25 @@ name type subtype language license + relpipe-in-asn1.cpp executable input c++ GNU GPLv3 + relpipe-in-asn1table.cpp executable input c++ GNU GPLv3 + relpipe-in-barcode.cpp executable input c++ GNU GPLv3 + relpipe-in-cbortable.cpp executable input c++ GNU GPLv3 relpipe-in-cli.cpp executable input c++ GNU GPLv3 relpipe-in-csv.cpp executable input c++ GNU GPLv3 relpipe-in-filesystem.cpp executable input c++ GNU GPLv3 relpipe-in-fstab.cpp executable input c++ GNU GPLv3 + relpipe-in-htmltable.cpp executable input c++ GNU GPLv3 + relpipe-in-ini.cpp executable input c++ GNU GPLv3 + relpipe-in-initable.cpp executable input c++ GNU GPLv3 relpipe-in-jack.cpp executable input c++ GNU GPLv3 + relpipe-in-mimetable.cpp executable input c++ GNU GPLv3 relpipe-in-recfile.cpp executable input c++ GNU GPLv3 relpipe-in-xml.cpp executable input c++ GNU GPLv3 relpipe-in-xmltable.cpp executable input c++ GNU GPLv3 + relpipe-in-x11.cpp executable input c++ GNU GPLv3 + relpipe-in-yaml.cpp executable input c++ GNU GPLv3 + relpipe-in-yamltable.cpp executable input c++ GNU GPLv3 relpipe-lib-cli.cpp library header-only c++ GNU GPLv3 relpipe-lib-common.cpp library shared c++ GNU LGPLv3 or GPLv2 relpipe-lib-reader.cpp library shared c++ GNU LGPLv3 or GPLv2 @@ -30,19 +41,24 @@ relpipe-out-asn1.cpp executable output c++ GNU GPLv3 relpipe-out-csv.cpp executable output c++ GNU GPLv3 relpipe-out-gui.qt.cpp executable output c++ GNU GPLv3 + relpipe-out-ini.qt.cpp executable output c++ GNU GPLv3 relpipe-out-nullbyte.cpp executable output c++ GNU GPLv3 relpipe-out-ods.cpp executable output c++ GNU GPLv3 relpipe-out-recfile.cpp executable output c++ GNU GPLv3 relpipe-out-tabular.cpp executable output c++ GNU GPLv3 relpipe-out-xml.cpp executable output c++ GNU GPLv3 + relpipe-out-x11.cpp executable output c++ GNU GPLv3 + relpipe-out-yaml.cpp executable output c++ GNU GPLv3 relpipe-tr-awk.cpp executable transformation c++ GNU GPLv3 relpipe-tr-cut.cpp executable transformation c++ GNU GPLv3 relpipe-tr-grep.cpp executable transformation c++ GNU GPLv3 - relpipe-tr-scheme.cpp executable transformation c++ GNU GPLv3 + relpipe-tr-infertypes.cpp executable transformation c++ GNU GPLv3 relpipe-tr-python.cpp executable transformation c++ GNU GPLv3 relpipe-tr-sed.cpp executable transformation c++ GNU GPLv3 + relpipe-tr-scheme.cpp executable transformation c++ GNU GPLv3 relpipe-tr-sql.cpp executable transformation c++ GNU GPLv3 relpipe-tr-validator.cpp executable transformation c++ GNU GPLv3 + relpipe-tr-xpath.cpp executable transformation c++ GNU GPLv3

      -

      Optional complexity

      +

      Optional complexity

      We are not scared by things like XML, SQL, RDF, Java or even C++ and we do not hate them. diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.11.xml --- a/relpipe-data/release-v0.11.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.11.xml Mon Feb 21 00:43:11 2022 +0100 @@ -40,7 +40,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      Data types

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.12.xml --- a/relpipe-data/release-v0.12.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.12.xml Mon Feb 21 00:43:11 2022 +0100 @@ -38,7 +38,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      Data types

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.13.xml --- a/relpipe-data/release-v0.13.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.13.xml Mon Feb 21 00:43:11 2022 +0100 @@ -53,7 +53,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      Data types

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.14.xml --- a/relpipe-data/release-v0.14.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.14.xml Mon Feb 21 00:43:11 2022 +0100 @@ -55,7 +55,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      Data types

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.15.xml --- a/relpipe-data/release-v0.15.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.15.xml Mon Feb 21 00:43:11 2022 +0100 @@ -33,7 +33,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      Streamlets

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.16.xml --- a/relpipe-data/release-v0.16.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.16.xml Mon Feb 21 00:43:11 2022 +0100 @@ -29,7 +29,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      ODBC in the SQL transformation module

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.17.1.xml --- a/relpipe-data/release-v0.17.1.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.17.1.xml Mon Feb 21 00:43:11 2022 +0100 @@ -43,7 +43,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      New CLI interfaces

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.17.xml --- a/relpipe-data/release-v0.17.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/release-v0.17.xml Mon Feb 21 00:43:11 2022 +0100 @@ -43,7 +43,7 @@

      Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. - Any suggestions, ideas and bug reports are welcome in our mailing list. + Any suggestions, ideas and bug reports are welcome in our mail box.

      New CLI interfaces

      diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.18.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/release-v0.18.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,301 @@ + + + Release v0.18 + new public release of Relational pipes + v0.18 + + +

      + We are pleased to introduce you the new development version of . + This release brings several new tools of each type (inputs, outputs, transformations and streamlet) and some smaller improvements. +

      + +
        +
      • + New input tools for reading various formats: + read ASN.1, YAML and JSON; see details below +
        • +
      • + New XMLTable-like input tools for converting arbitrary tree structures to relations: + read ASN.1, INI, MIME, YAML, JSON, CBOR and HTML; see details below +
        • +
      • + New input tool and streamlet for reading barcodes and QR codes: see details below +
        • +
      • + New input and output tool for the INI format: see details below +
        • +
      • + New input and output tool for interacting with X11: see details below +
        • +
      • + New XPath transformation: see details below +
        • +
      • + Data types support on CSV input and output: strings, integers and booleans; see details below +
        • +
      • + The grep, cut and sed transformations have new CLI interface: see details below +
        • +
      + +

      + See the (new) examples and (new) screenshots pages for details. +

      + +

      + Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. + Any suggestions, ideas and bug reports are welcome in our mail box. +

      + +

      New standard input modules

      + +

      + These tools allows serialization of arbitrary relational data + and then reading resulting file or stream back to the relational data. + An of course, we can read data created or generated somewhere else. + Since this release, we can newly serialize our relations to YAML and ASN.1, store, edit or transfer these files or streams and then deserialize them and pipe them through relational transformations or outputs. + Formats like XML or Recfiles were already supported in previous releases. + Full lossless round-trip is now possible with: XML, YAML, ASN.1 and Recfile formats. + Partial lossless round-trip is possible with CSV – these input and output tools are limited to a single relation. + Lossy round-trip is possible with relpipe-out-nullbyte and relpipe-in-cli – + the nullbyte stream may contain only a single relation and its data types need to be passed as CLI arguments to the input tool. + Other formats (e.g. JACK, INI or X11) may also have corresponding inputs and outputs, but they produce and consume some specific structures – they are not universal and usable with arbitrary relational data. +

      + + +

      YAML and JSON

      + +

      + YAML is a text format for serializing tree structures into text (like XML). + It can be sometimes tricky, but basic scenarios are intuitive and straightforward. + We can use YAML to describe relational data and load them using the relpipe-in-yaml input filter + and we can produce YAML files using the relpipe-out-yaml output filter. +

      + +

      More details in the example: Reading and writing YAML

      + +

      ASN.1

      +

      + There are actually more ASN.1 applications around us than most people realize. + ASN.1 is almost everywhere – in cryptography (X.509 / TLS / HTTP, S/MIME etc.), LDAP, payment cards, telecommunications (including GSM, GPRS, EDGE, UMTS, LTE, 5G), Kerberos authentication etc. + So it is very useful to be able to access these data. + Because ASN.1 is abstract, there are several ways how such data can be serialized into a stream of octets (bytes). + Most common way is BER (Basic Encoding Rules), so we support BER (and thus its variants DER and CER) in the first version of our tools. + Later other encoding might be supported and we could also parse the ASN.1 syntax (the schema language). +

      +

      + ASN.1 BER output was added several releases ago. + Now we have also ASN.1 BER input (relpipe-in-asn1) so we can read data in this format generated earlier. + Of course, BER data can be generated or read also by other ASN.1 capable software. + However, maybe more interesting than relpipe-in-asn1 is relpipe-in-asn1table (see below) that reads arbitrary ASN.1 BER data (not only serialized relational data). +

      + +

      Reading and writing INI

      +

      + INI is very common simple text format used for configuration files. + It extends classic key=value config files and adds one more level – sections – for structuring. + In this release we got input and output filter for INI and similar formats (Java .properties, MANIFEST.MF, key=value configs). + Because comments and whitespace are also supported, we can do (almost) loss-less conversions and transformations (e.g. change value of an entry). +

      + +

      More details in the example: Reading and writing INI and unix configs

      + +

      Reading barcodes and QR codes

      +

      + Either ubiquitous 1D barcodes (procudct labels, ISBN in form of EAN-13 etc.) or 2D barcodes (QR containing hyperlinks, vCards etc.) + encode numbers, texts or other data into images that can be printed and scanned or photographed later. + Now we can interact with this technology – read barcodes using the + relpipe-in-barcode tool or barcode-reader streamlet. +

      +

      More details in the example: Reading barcodes and QR

      + +

      New XMLTable-like input modules: ASN.1, INI, MIME, YAML, JSON, CBOR, HTML

      +

      + More details in the examples: +

      +
        +
      • Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way
        • +
      • Exploring content of X.509 certificates
        • +
      + +

      X11 input and output modules

      +

      + The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland). + This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice. + Since this release of we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools. + In the following example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements). +

      +

      More details in the example: Exploring X11 windows and devices and emulating mouse movements and keystrokes

      + +

      XPath transformation

      +

      + We got a new powerful language for filtering and transformations: XPath. + It is now part of the toolset consisting of SQL, AWK, Scheme and others. + However XPath is originally a language designed for XML, in we can use it for relational data coming from various sources, not only XML, + and also for data that violates the rules of normal forms. + We can process quite complex tree structures entangled in records but we can also write simple and intuitive expressions like x = "a" or y = 123. +

      +

      More details in the example: Filtering and transforming relational data with XPath

      + +

      Data types in CSV + generic relpipe-tr-infertypes tool

      +

      + CSV input and output filters now support data types i.e. CSV can now carry not only text strings, but also booleans and integers (or more types in future releases). + We also have a new tool relpipe-tr-infertypes that can automatically recognize the types in data that came from CSV or any other source. +

      +

      More details in the example: CSV and data types

      + +

      Feature overview

      + +

      Data types

      +
        +
      • boolean
        • +
      • variable-length signed integer (SLEB128)
        • +
      • string in UTF-8
        • +
      +

      Inputs

      +
        +
      • Recfile
        • +
      • XML
        • +
      • XMLTable
        • +
      • CSV
        • +
      • file system
        • +
      • CLI
        • +
      • fstab
        • +
      • SQL script
        • +
      • JACK
        • +
      • barcode
        • +
      • X11
        • +
      • ASN.1 BER
        • +
      • ASN.1 BER Table
        • +
      • INI
        • +
      • INITable
        • +
      • MIMETable
        • +
      • YAML
        • +
      • YAMLTable
        • +
      • JSON
        • +
      • JSONTable
        • +
      • CBORTable
        • +
      • HTMLTable
        • +
      +

      Transformations

      +
        +
      • sql: filtering and transformations using the SQL language
        • +
      • xpath: filtering and transformations using the XPath language
        • +
      • awk: filtering and transformations using the classic AWK tool and language
        • +
      • scheme: filtering and transformations defined in the Scheme language using GNU Guile
        • +
      • grep: regular expression filter, removes unwanted records from the relation
        • +
      • cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)
        • +
      • sed: regular expression replacer
        • +
      • validator: just a pass-through filter that crashes on invalid data
        • +
      • infertypes: derive data types from attribute values
        • +
      • python: highly experimental
        • +
      +

      Streamlets

      +
        +
      • xpath (example, unstable)
        • +
      • hash (example, unstable)
        • +
      • jar_info (example, unstable)
        • +
      • mime_type (example, unstable)
        • +
      • exiftool (example, unstable)
        • +
      • pid (example, unstable)
        • +
      • cloc (example, unstable)
        • +
      • exiv2 (example, unstable)
        • +
      • inode (example, unstable)
        • +
      • lines_count (example, unstable)
        • +
      • pdftotext (example, unstable)
        • +
      • pdfinfo (example, unstable)
        • +
      • tesseract (example, unstable)
        • +
      • barcode (example, unstable)
        • +
      +

      Outputs

      +
        +
      • ASN.1 BER
        • +
      • Recfile
        • +
      • CSV
        • +
      • tabular
        • +
      • XML
        • +
      • nullbyte
        • +
      • GUI in Qt
        • +
      • ODS (LibreOffice)
        • +
      • JACK
        • +
      • X11
        • +
      • INI
        • +
      • YAML
        • +
      + +

      New examples

      +
        +
      • Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way
        • +
      • Filtering and transforming relational data with XPath
        • +
      • CSV and data types
        • +
      • Exploring X11 windows and devices and emulating mouse movements and keystrokes
        • +
      • Exploring content of X.509 certificates
        • +
      • Running SQL JOINs on multiple CSV files
        • +
      • Reading and writing YAML
        • +
      • Reading and writing INI and unix configs
        • +
      • Reading barcodes and QR
        • + +
      + +

      New screenshots

      + + + +

      Backward incompatible changes

      + +

      + In relpipe-tr-sql and relpipe-in-sql the --list-data-sources option has now boolean parameter + i.e. it is not the presence of the option but the true or false value what matters. + We can also list the data sources while we simultaneously run some SQL. +

      + +

      + The relpipe-tr-grep, relpipe-tr-cut and relpipe-tr-sed + were the last tools with the obsolete positional CLI interface. + In this release, they were updated and now share the same style of CLI arguments as other tools. + Besides consistent and predictable user interface, these tools are now more powerful because they can transforma more relations at once. +

      + +

      Instead of e.g.

      + 
      relpipe-tr-grep 'a' 'b' 'c'
      +

      we now write:

      + 
      relpipe-tr-grep --relation 'a' --attribute 'b' --value 'c'
      +

      Bash completion helps us while writing such commands and result is much more readable than original cryptic version.

      + +

      See updated examples:

      + +
        +
      • Renaming VG in /etc/fstab using relpipe-tr-sed
        • +
      • Using relpipe-tr-sed with groups and backreferences
        • +
      • Filtering /etc/fstab using relpipe-tr-grep
        • +
      • Doing projection and restriction using cut and grep
        • +
      • Reading apt (Debian package system) results
        • +
      • Generating statistics and charts using XMLTable
        • +
      • Streamlets preview
        • +
      • Running SQL JOINs on multiple CSV files
        • +
      • Finding runnable JARs
        • +
      + + +

      Installation

      + +

      + Instalation was tested on Debian GNU/Linux 10.2. + The process should be similar on other distributions. +

      + + + +

      + are modular thus you can download and install only parts you need (the libraries are needed always). + Tools out-gui.qt and tr-python require additional libraries and are not built by default. +

      + +       + +       \ No newline at end of file diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/roadmap.xml --- a/relpipe-data/roadmap.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/roadmap.xml Mon Feb 21 00:43:11 2022 +0100 @@ -16,12 +16,19 @@ Released versions are described on the download page.

      -

      v0.17, v0.18, v0.19 etc.

      +

      v0.19, v0.20, v0.21 etc.

      Releases for discussion and verification of the format and API design.

      +

      + This phase (before v1.0.0) might seem quite long. + But it is important to verify the ideas and design in various scenarios, on various use cases. + The general idea and the big picture are quite clear and stable. + However there are many technical details that need to be carefully tuned. +

      +

      Data types

      • fractions
        • @@ -59,6 +66,7 @@
      • publish documentation of the stable API and file format
      • publish automated complex tests (specification vs. implementation compliance)
      • verify the format from the performance point of view
        • +
      • relpipe-lib-writer: several modes of output buffering (auto, relation, record, value) or manual control
      • improve parsing (corrupted input may currently lead to huge memory allocations), more fuzzing
      • code clean-up and refactoring, move some reusable parts to common libraries
      • test the build with another compiler and tune the code
        • diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/screenshots.xml --- a/relpipe-data/screenshots.xml Fri Nov 26 22:14:18 2021 +0100 +++ b/relpipe-data/screenshots.xml Mon Feb 21 00:43:11 2022 +0100 @@ -32,6 +32,9 @@ +

        GNU/Linux and Window Maker

        + +

        Haiku OS