diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/release-v0.18.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/release-v0.18.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,301 @@ + + + Release v0.18 + new public release of Relational pipes + v0.18 + + +

+ We are pleased to introduce you the new development version of . + This release brings several new tools of each type (inputs, outputs, transformations and streamlet) and some smaller improvements. +

+ + + +

+ See the (new) examples and (new) screenshots pages for details. +

+ +

+ Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. + Any suggestions, ideas and bug reports are welcome in our mail box. +

+ +

New standard input modules

+ +

+ These tools allows serialization of arbitrary relational data + and then reading resulting file or stream back to the relational data. + An of course, we can read data created or generated somewhere else. + Since this release, we can newly serialize our relations to YAML and ASN.1, store, edit or transfer these files or streams and then deserialize them and pipe them through relational transformations or outputs. + Formats like XML or Recfiles were already supported in previous releases. + Full lossless round-trip is now possible with: XML, YAML, ASN.1 and Recfile formats. + Partial lossless round-trip is possible with CSV – these input and output tools are limited to a single relation. + Lossy round-trip is possible with relpipe-out-nullbyte and relpipe-in-cli – + the nullbyte stream may contain only a single relation and its data types need to be passed as CLI arguments to the input tool. + Other formats (e.g. JACK, INI or X11) may also have corresponding inputs and outputs, but they produce and consume some specific structures – they are not universal and usable with arbitrary relational data. +

+ + +

YAML and JSON

+ +

+ YAML is a text format for serializing tree structures into text (like XML). + It can be sometimes tricky, but basic scenarios are intuitive and straightforward. + We can use YAML to describe relational data and load them using the relpipe-in-yaml input filter + and we can produce YAML files using the relpipe-out-yaml output filter. +

+ +

More details in the example: Reading and writing YAML

+ +

ASN.1

+

+ There are actually more ASN.1 applications around us than most people realize. + ASN.1 is almost everywhere – in cryptography (X.509 / TLS / HTTP, S/MIME etc.), LDAP, payment cards, telecommunications (including GSM, GPRS, EDGE, UMTS, LTE, 5G), Kerberos authentication etc. + So it is very useful to be able to access these data. + Because ASN.1 is abstract, there are several ways how such data can be serialized into a stream of octets (bytes). + Most common way is BER (Basic Encoding Rules), so we support BER (and thus its variants DER and CER) in the first version of our tools. + Later other encoding might be supported and we could also parse the ASN.1 syntax (the schema language). +

+

+ ASN.1 BER output was added several releases ago. + Now we have also ASN.1 BER input (relpipe-in-asn1) so we can read data in this format generated earlier. + Of course, BER data can be generated or read also by other ASN.1 capable software. + However, maybe more interesting than relpipe-in-asn1 is relpipe-in-asn1table (see below) that reads arbitrary ASN.1 BER data (not only serialized relational data). +

+ +

Reading and writing INI

+

+ INI is very common simple text format used for configuration files. + It extends classic key=value config files and adds one more level – sections – for structuring. + In this release we got input and output filter for INI and similar formats (Java .properties, MANIFEST.MF, key=value configs). + Because comments and whitespace are also supported, we can do (almost) loss-less conversions and transformations (e.g. change value of an entry). +

+ +

More details in the example: Reading and writing INI and unix configs

+ +

Reading barcodes and QR codes

+

+ Either ubiquitous 1D barcodes (procudct labels, ISBN in form of EAN-13 etc.) or 2D barcodes (QR containing hyperlinks, vCards etc.) + encode numbers, texts or other data into images that can be printed and scanned or photographed later. + Now we can interact with this technology – read barcodes using the + relpipe-in-barcode tool or barcode-reader streamlet. +

+

More details in the example: Reading barcodes and QR

+ +

New XMLTable-like input modules: ASN.1, INI, MIME, YAML, JSON, CBOR, HTML

+

+ More details in the examples: +

+ + +

X11 input and output modules

+

+ The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland). + This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice. + Since this release of we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools. + In the following example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements). +

+

More details in the example: Exploring X11 windows and devices and emulating mouse movements and keystrokes

+ +

XPath transformation

+

+ We got a new powerful language for filtering and transformations: XPath. + It is now part of the toolset consisting of SQL, AWK, Scheme and others. + However XPath is originally a language designed for XML, in we can use it for relational data coming from various sources, not only XML, + and also for data that violates the rules of normal forms. + We can process quite complex tree structures entangled in records but we can also write simple and intuitive expressions like x = "a" or y = 123. +

+

More details in the example: Filtering and transforming relational data with XPath

+ +

Data types in CSV + generic relpipe-tr-infertypes tool

+

+ CSV input and output filters now support data types i.e. CSV can now carry not only text strings, but also booleans and integers (or more types in future releases). + We also have a new tool relpipe-tr-infertypes that can automatically recognize the types in data that came from CSV or any other source. +

+

More details in the example: CSV and data types

+ +

Feature overview

+ +

Data types

+ +

Inputs

+ +

Transformations

+ +

Streamlets

+ +

Outputs

+ + +

New examples

+ + +

New screenshots

+ + + +

Backward incompatible changes

+ +

+ In relpipe-tr-sql and relpipe-in-sql the --list-data-sources option has now boolean parameter + i.e. it is not the presence of the option but the true or false value what matters. + We can also list the data sources while we simultaneously run some SQL. +

+ +

+ The relpipe-tr-grep, relpipe-tr-cut and relpipe-tr-sed + were the last tools with the obsolete positional CLI interface. + In this release, they were updated and now share the same style of CLI arguments as other tools. + Besides consistent and predictable user interface, these tools are now more powerful because they can transforma more relations at once. +

+ +

Instead of e.g.

+ 
relpipe-tr-grep 'a' 'b' 'c'
+

we now write:

+ 
relpipe-tr-grep --relation 'a' --attribute 'b' --value 'c'
+

Bash completion helps us while writing such commands and result is much more readable than original cryptic version.

+ +

See updated examples:

+ +
    +
  • Renaming VG in /etc/fstab using relpipe-tr-sed
    • +
  • Using relpipe-tr-sed with groups and backreferences
    • +
  • Filtering /etc/fstab using relpipe-tr-grep
    • +
  • Doing projection and restriction using cut and grep
    • +
  • Reading apt (Debian package system) results
    • +
  • Generating statistics and charts using XMLTable
    • +
  • Streamlets preview
    • +
  • Running SQL JOINs on multiple CSV files
    • +
  • Finding runnable JARs
    • +
+ + +

Installation

+ +

+ Instalation was tested on Debian GNU/Linux 10.2. + The process should be similar on other distributions. +

+ + + +

+ are modular thus you can download and install only parts you need (the libraries are needed always). + Tools out-gui.qt and tr-python require additional libraries and are not built by default. +

+ + + + \ No newline at end of file