# HG changeset patch # User František Kučera # Date 1591401444 -7200 # Node ID 192b0059a6c45c9e6288d998a3c7ae6657202fb6 # Parent 418e11eb6fea78abdd18fbd08f42df9a2cd3b5d7 Release v0.16 – ODBC in SQL + JACK MIDI diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/download.xml --- a/relpipe-data/download.xml Thu May 21 01:23:23 2020 +0200 +++ b/relpipe-data/download.xml Sat Jun 06 01:57:24 2020 +0200 @@ -17,7 +17,9 @@
Released versions
 			
 		
+			2020-06-06: v0.16
 			2020-01-31: v0.15
 			2019-10-30: v0.14
 			2019-07-30: v0.13
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/examples-in-filesystem-tr-sql-indexing.xml
--- a/relpipe-data/examples-in-filesystem-tr-sql-indexing.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/examples-in-filesystem-tr-sql-indexing.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -26,7 +26,7 @@
 		
 		
+	| relpipe-tr-sql --data-source-string 'Driver=SQLite3;Database=file:bin.sqlite']]>
 	
 		
 			This index allows us to do fast searches and various analysis.
@@ -34,7 +34,7 @@
 		
 		
 		
@@ -75,7 +75,7 @@
 		
 
 		
+	| relpipe-tr-sql --data-source-string 'Driver=SQLite3;Database=file:bin.sqlite']]>
 	
 		And then we can make a „popularity contest“ and find 20 most often used libraries:
 		
 		--xattr option).
 		
 		
+		
+			n.b. if we use a database frequently it is convenient to configure it as a data source in the ~/.odbc.ini file
+			– and then connect to it using the --data-source-name option and its name.
+		
+		
 	
 
 
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/examples-in-sql-selecting-existing-database.xml
--- a/relpipe-data/examples-in-sql-selecting-existing-database.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/examples-in-sql-selecting-existing-database.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -10,16 +10,16 @@
 		
 		
 			Both the relpipe-tr-sql and relpipe-in-sql
-			support the option --file and can store data in a file instead of just in-memory.
-			Primary purpose of this option is to offload to HDD and be able to process more data whithout consuming too much RAM.
-			So such file is usually a temporary one and is deleted immediately after the transformation.
-			But we can also direct it to an existing file and append new relations to it.
-			Or we can create a new file and do not delete it – using the --file-keep true option.
+			support options --data-source-name and --data-source-string
+			that can be used for connecting to a different database than the in-memory SQLite one.
+			This is useful if we want to offload to HDD and be able to process more data whithout consuming too much RAM.
+			Or if we want to access an existing database instead of just transforming records from the STDIN to STDOUT.
+			These options allow connecting to any DBMS.
 		
 		
 		
 			Thus we can use the relpipe-in-sql as a database client to access existing SQLite files and SELECT from them (and then convert the results to any supported format).
-			An we can use the relpipe-tr-sql as an output filter which converts relational data to a SQLite file that can be queried later.
+			And we can use the relpipe-tr-sql as an output filter which converts relational data to a SQLite file that can be queried later.
 		
 		
 		
@@ -35,11 +35,11 @@
 		
 		
 		

-			We can then query it using  tools:
+			We can then query the database file sqlite.fossil using  tools:
 		
 		
 		
 		
-		relpipe-in-jack | relpipe-out-gui
+		relpipe-in-jack | relpipe-out-csv
 		
 		
 			We are connected to the JACK daemon now, but no data are routed to us.
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/examples-tr-sql-odbc.xml
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-tr-sql-odbc.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -0,0 +1,286 @@
+
+	
+	Accessing SQLite, PostgreSQL and MySQL through ODBC
+	use various DBMS for SQL transformations or data access
+	04200
+
+	
+		
+		

+			Since v0.16 the relpipe-tr-sql module
+			uses the ODBC abstraction layer and thus we can access data in any DBMS (database management system).
+			Our program depends only on the generic API and the driver for particular DBMS is loaded dynamically depending on the configuration.
+		
+		
+		
+			
+				ODBC (Open Database Connectivity) is an industry standard and provides API for accessing a DBMS.
+				In late 80s several vendors (mostly from the Unix and database communities) established the SQL Access Group (SAG)
+				and then specified the Call Level Interface (CLI). ODBC, which is based on CLI, was published in early 90s.
+				ODBC is available on many operating systems and there are at least two free software implementations:
+				unixODBC and iODBC.
+			
+		
+		
+		For more information see the v0.16 release notes.
+		
+		General concepts and configuration
+		
+		
+			ODBC:
+			the API consisting of C functions; see the files sql.h and sqlext.h e.g. in unixODBC.
+		
+		
+			Database driver:
+			a shared library (an .so file) 
+			that implements the API and connects to particular DBMS (SQLite, PostgreSQL, MySQL, MariaDB, Firebird etc.);
+			is usually provided by the authors of given DBMS, sometimes writen by a third-party
+		
+		
+			Client:
+			a program that calls the API in order to access a database; our relpipe-tr-sql is a client
+		
+		
+			Data Source Name (DSN):
+			the name of a preconfigured data source – when connecting, we need to know only the DSN – all parameters
+			(like server name, user name, password etc.) can be then looked-up in the configuration
+		
+		
+			Connection string:
+			a text string consisting of serialized parameters needed for connecting
+			– we can specify all parameters ad-hoc in the connection string without creating any permanent configuration;
+			a connection string can also refer to a DSN and add or override some parameters
+		
+
+		
+			There is some global configuration in the /etc directory.
+			In /etc/odbcinst.ini we can a find list of ODBC drivers.
+			Thanks to it, we can refer to a driver by its name (e.g. SQLite3)
+			instead of the path to the shared library (e.g. /usr/lib/x86_64-linux-gnu/odbc/libsqlite3odbc.so).
+			In /etc/odbc.ini we can find a list of global (for given computer) data sources.
+			It is uncommon to put complete configurations in this file, because anyone would be able to read the passwords,
+			but we can provide here just a template with public parameters like server name, port etc.
+			and user will supply his own user name and password in the connection string or in his personal configuration file.
+		
+		
+		
+			The ~/.odbc.ini contains personal configuration of given user.
+			There are usually data sources including the passwords.
+			Thus this file must be readable only by given user (chmod 600 ~/.odbc.ini).
+			Providing passwords in connection strings passed as CLI arguments is not a good practice due to security reasons:
+			by default it is stored in the shell history and it is also visible to other users of the same machine in the list of running processes.
+		
+		
+		
+			The section name – in the [] brackets – is the DSN.
+			Then there are parameters in form of key=value on each line.
+		
+		
+		
+		CLI options
+		
+		
+			The relpipe-tr-sql and relpipe-in-sql support these relevant CLI options:
+		
+		
+		
+			
+				--list-data-sources:
+				lists available (configured) data sources in relational format (so we pipe the output to some output filter e.g. to relpipe-out-tabular)
+			
+			
+				--data-source-name:
+				specifies the DSN of a configured data source
+			
+			
+				--data-source-string:
+				specifies the connections string for ad-hoc connection without need of any configuration
+			
+		
+		
+		
+
+		
+			Because output of this command is relational, we can further process it in our relational pipelines.
+			This output is also used for the Bash-completion for suggesting the DSN.
+		
+		
+		
+			If neither --data-source-name nor --data-source-string option is provided,
+			a temporary in-memory SQLite database is used as default.
+		
+		
+		SQLite
+		
+		In Debian GNU/Linux and similar distributions we can install SQLite ODBC driver by this command:
+		
+		apt install libsqliteodbc
+		
+		Which also installs the SQLite library that is all we need (because SQLite is a serverless and self-contained database).
+		
+		
+			Then we can use the default in-memory temporary database or specify the connection string ad-hoc, 
+			access existing SQLite databases
+			or create new ones	– e.g. this command:
+		
+		
+		… | relpipe-tr-sql --data-source-string 'Driver=SQLite3;Database=file:MyDatabase.sqlite'
+		
+		will create the MyDatabase.sqlite file and fill it with relations that came from STDIN.
+		
+		For frequently used databases it is convenient to configure a data source in ~/.odbc.ini:
+		
+		
+
+		
+			and then connect to it simply using --data-source-name MyDatabase
+			(both the option and the name will be suggested by Bash-completion).
+		
+		
+		
+			The SQLite ODBC driver supports several parameters that are described in its documentation.
+			One of them is LoadExt that loads SQLite extensions:
+		
+		
+		
+		
+		
+			So we can write our own SQLite extension with custom functions or other features 
+			(example)
+			or chose some existing one and load it into the SQLite connected through ODBC.
+		
+
+		
+		PostgreSQL
+		
+		In Debian GNU/Linux and similar distributions we can install PostgreSQL ODBC driver by this command:
+		
+		apt install odbc-postgresql
+		
+		
+			PostgreSQL is very powerful DBMS (probably most advanced free software relational database system)
+			and utilizes the client-server architecture.
+			This means that we also needs a server (can be also installed through apt like the driver).
+		
+		
+		
+			Once we have a server – remote or local – we need to create a user (role).
+			For SQL transformations we configure a dedicated role that has no persistent schema and uses the temporary one as default,
+			which means that all relations we create are lost at the end of the session (when the relpipe-tr-sql command finishes),
+			thus it behaves very similar to the SQLite in-memory database.
+		
+		
+		
+
+		
+			And then we configure the ODBC data source:
+		
+
+		
+
+		
+			Now we can use advanced PostgreSQL features for transforming data in our pipelines.
+			We can also configure a DSN for another database that contains some useful data and other database objects, 
+			call existing business functions installed in such database, load data to or from this DB etc.
+		
+		
+		
+		MySQL
+		
+		
+			If the libmyodbc package is missing in our distribution,
+			the	ODBC driver for MySQL can be downloaded from their website.
+			We can get a binary package (.deb, .rpm etc.) or source code.
+			If we are compiling from sources, we do something like this:
+		
+		
+		
+
+		
+			We should use the driver in the same or similar version as the MySQL client library installed on our system.
+			For example 8.x driver will not work with 5.x library.
+			Successful compilation results in libmyodbc*.so files.
+		
+		
+		
+			Like PostgreSQL, also MySQL is a client-server,
+			so we need a server where we create a database and some user account.
+			As root through the mysql mysql command we execute:
+		
+		
+		
+
+		As a normal user we add new data source to our ~/.odbc.ini file:
+
+		
+
+		
+			See that we have compiled the ODBC driver in our home directory
+			and even without installing it anywhere and registering it in the /etc/odbcinst.ini file,
+			we can simply refer to the .so file from our ~/.odbc.ini.
+		
+
+		
+			If we set Server=localhost, the client-server communication does not go through TCP/IP
+			but rather through the unix domain socket specified in the Socket field.
+			If we set Server=127.0.0.1 or some remote IP address or domain name, the communication goes through TCP/IP on given port.
+		
+		
+		
+			The SET SQL_MODE=ANSI_QUOTES; init statement is important,
+			because it tells MySQL server that it should support standard SQL "quoted" identifiers
+			instead of that `weird` MySQL style.
+			We use the standard SQL while creating the tables.
+		
+		
+		
+			There are many other parameters, quite well 
+			documented.
+		
+		
+		
+			Now we can use MySQL as the SQL engine for transformations in our pipelines
+			and we can also access existing MySQL databases,
+			load data to and from them
+			or call functions and procedures installed on the server. 
+		
+		
+		
+	
+
+
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/examples-tr-sqlite-custom-version.xml
--- a/relpipe-data/examples-tr-sqlite-custom-version.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/examples-tr-sqlite-custom-version.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -14,6 +14,17 @@
 		
 		
 		
+			
+				n.b. This method is obsolete since v0.16 that does not use SQLite library directly 
+				and uses arbitrary database driver (including SQLite one) through an abstraction layer (ODBC).
+				This article is still valid as an example of the LD_PRELOAD hack and can be used with older versions of .
+				Since v0.16 we can easily replace whole ODBC driver (and thus use also different version of the SQLite),
+				there is no need for LD_PRELOAD hacking
+				– we can just configure desired driver (the .so file) in the INI file or ad-hoc in the connection string.
+			
+		
+		
+		
 			By default, relpipe-tr-sql links to the SQLite library available in our distribution (e.g. 3.22).
 			As we can check:
 		
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/examples/release-v0.16.sh
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples/release-v0.16.sh	Sat Jun 06 01:57:24 2020 +0200
@@ -0,0 +1,53 @@
+# Install dependencies as root:
+apt install g++ make cmake mercurial pkg-config
+apt install libxerces-c-dev       # needed only for relpipe-in-xml module
+apt install guile-2.2-dev         # needed only for relpipe-tr-guile module
+apt install gawk                  # needed only for relpipe-tr-awk module
+apt install libxml++2.6-dev       # needed only for relpipe-in-xmltable module
+apt install unixodbc-dev          # needed only for relpipe-tr-sql module
+apt install libsqliteodbc         # needed only for relpipe-tr-sql module if used with SQLite
+apt install odbc-postgresql       # needed only for relpipe-tr-sql module if used with PostgreSQL
+apt install libjack-jackd2-dev    # needed only for relpipe-in-jack module
+
+# Run rest of installation as a non-root user:
+export RELPIPE_VERSION="v0.16"
+export RELPIPE_SRC=~/src
+export RELPIPE_BUILD=~/build
+export RELPIPE_INSTALL=~/install
+export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
+export PATH="$RELPIPE_INSTALL/bin:$PATH"
+
+rm -rf "$RELPIPE_BUILD"/relpipe-*
+mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
+
+# Helper functions:
+relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
+relpipe_install()  { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
+
+# Download all sources:
+relpipe_download lib-common lib-reader lib-writer lib-cli lib-xmlwriter in-cli in-fstab in-xml in-xmltable in-csv in-filesystem in-recfile out-gui.qt out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-python tr-sed tr-validator tr-guile tr-awk tr-sql in-jack
+
+# Optional: At this point, we have all dependencies and sources downloaded, so we can disconnect this computer from the internet in order to verify that our build process is sane, deterministic and does not depend on any external resources.
+
+# Build and install libraries:
+relpipe_install lib-common lib-reader lib-writer lib-cli lib-xmlwriter
+
+# Build and install tools:
+relpipe_install in-fstab in-cli in-fstab in-xml in-xmltable in-csv in-recfile tr-cut tr-grep tr-sed tr-guile tr-awk tr-sql out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile in-filesystem in-jack
+
+# Load Bash completion scripts:
+for c in "$RELPIPE_SRC"/relpipe-*/bash-completion.sh ; do . "$c"; done
+
+# Enable streamlet examples:
+export RELPIPE_IN_FILESYSTEM_STREAMLET_PATH="$RELPIPE_SRC"/relpipe-in-filesystem.cpp/streamlet-examples/
+
+# Clean-up:
+unset -f relpipe_install
+unset -f relpipe_download
+unset -v RELPIPE_VERSION
+unset -v RELPIPE_SRC
+unset -v RELPIPE_BUILD
+unset -v RELPIPE_INSTALL
+
+# List configured ODBC data sources:
+relpipe-in-sql --list-data-sources | relpipe-out-tabular
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/implementation.xml
--- a/relpipe-data/implementation.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/implementation.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -18,6 +18,7 @@
 			relpipe-in-csv.cpp	executable	input	c++	GNU GPLv3
 			relpipe-in-filesystem.cpp	executable	input	c++	GNU GPLv3
 			relpipe-in-fstab.cpp	executable	input	c++	GNU GPLv3
+			relpipe-in-jack.cpp	executable	input	c++	GNU GPLv3
 			relpipe-in-recfile.cpp	executable	input	c++	GNU GPLv3
 			relpipe-in-xml.cpp	executable	input	c++	GNU GPLv3
 			relpipe-in-xmltable.cpp	executable	input	c++	GNU GPLv3
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/release-v0.13.xml
--- a/relpipe-data/release-v0.13.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/release-v0.13.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -43,6 +43,11 @@
 		
 		
 		
+			n.b. the --file and --file-keep have been dropped in v0.16
+			(replacement is described in the v0.16 release notes).
+		
+		
+		
 			See the examples and screenshots pages for details.
 		
 		
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/release-v0.14.xml
--- a/relpipe-data/release-v0.14.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/release-v0.14.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -45,6 +45,11 @@
 		
 		
 		
+			n.b. the --file and --file-keep have been dropped in v0.16
+			(replacement is described in the v0.16 release notes).
+		
+		
+		
 			See the examples and screenshots pages for details.
 		
 		
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/release-v0.16.xml
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/release-v0.16.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -0,0 +1,264 @@
+
+	
+	Release v0.16
+	new public release of Relational pipes
+	v0.16
+
+	
+		
+			We are pleased to introduce you the new development version of .
+			This release brings an abstraction layer (ODBC) in the SQL transformation + several smaller improvements.
+		
+		
+		
+			
+				ODBC in the relpipe-tr-sql module: see details below
+			
+				new input relpipe-in-jack module: see details below
+			
+				keyboard shortcuts in the relpipe-out-gui module: use Ctrl+PgUp and Ctrl+PgDown to switch panels (relations) and Ctrl+Q to quit
+			
+				record count in the relpipe-out-xhtml command: number of records is printed under the table (this command part of relpipe-out-xml, not a standalone module)
+		
+		
+		
+			See the examples and screenshots pages for details.
+		
+		
+		
+			Please note that this is still a development relasease and thus the API (libraries, CLI arguments, formats) might and will change.
+			Any suggestions, ideas and bug reports are welcome in our mailing list.
+		
+		
+		ODBC in the SQL transformation module
+		
+		
+			Former versions of relpipe-tr-sql were tied to SQLite
+			and user had no option to change the SQL engine.
+			However great SQLite is (and we are very thankful for it), having some particular DBMS (database management system) hard-coded in our program is too constraining.
+			So we added an abstraction layer (ODBC) and get rid of the direct dependency on SQLite.
+			Now any DBMS can be used with .
+		
+		
+		
+			ODBC (Open Database Connectivity) is an industry standard and provides API for accessing a DBMS.
+			In late 80s several vendors (mostly from the Unix and database communities) established the SQL Access Group (SAG)
+			and then specified the Call Level Interface (CLI). ODBC, which is based on CLI, was published in early 90s.
+			ODBC is available on many operating systems and there are at least two free software implementations:
+			unixODBC and iODBC.
+			We use unixODBC for development and testing.
+			Future releases of  should be tested also with other implementations and various database drivers.
+		
+		
+		
+			SQLite remains the default option
+			(in the C++ implementation, while Java or other implementations may have different default and may use different abstraction layer like JDBC).
+			We count on SQLite for future releases. It is the simplest way to get full SQL power in your relational pipeline.
+			However, relpipe-tr-sql do not depend on SQLite and can be installed without it (and then used e.g. with PostgreSQL driver).
+			Using different DBMS makes sense for two main reasons:
+		
+		
+		
+			
+				We need specific features provided by the DBMS.
+				It might be e.g. some functions for XML processing or some advanced SQL language constructs.
+				Or maybe we have some business logic already implemented as SQL functions in e.g. PostgreSQL
+				– now we can access this logic from our pipelines or seamlessly integrate it in our shell:
+				 some-fancy-report.xhtml
+	# or just: | relpipe-out-xhtml > some-generic-report.xhtml]]>
+			
+			
+				We need access to data in an existing database.
+				The relpipe-tr-sql and relpipe-in-sql can be used as a generic database clients
+				and are able to load relational data to and from any DBMS.
+				We can also write a pipeline to transfer data between two different DBMS, do some ETL (extract, transform, load) tasks
+				or just cache some result sets from a remote database in our local SQLite file.
+				We can cache e.g. some codelist tables or other data for offline use:
+				
+				In previous versions, we needed SQL-DK for this scenario,
+				now it is possible solely in  without any other tools.
+				But SQL-DK is still useful – especially if we have a JDBC driver but do not have an ODBC one
+				(JDBC drivers and Java are also much more portable).
+			
+		
+		
+		
+			n.b. However it still looks like executing a local command, we should be aware that while using a remote data source,
+			our data travel to given remote server – this impacts performance and our privacy.
+			Never use untrustworthy remote server for processing sensitive data (even if using just a temporary schema or tables).
+			If SQLite is „too small“ then PostgreSQL installed on localhost is usually a good option.
+		
+		
+		
+			There are ODBC drivers for any conceivable database system.
+			We can also write a custom driver for any other resource and just plug it in 
+			without recompiling (a driver is a shared library – simply an .so file).
+		
+		
+		
+			This release also comes with better diagnostics. This feature is not specific to ODBC, but was implemented during the rewrite of the database layer.
+			So if we make a mistake in our query or try to create a table with the same name as already exists in the DB, we will get a useful message with detailed description of the problem
+			(instead of a pointless failure notice in the previous version).
+		
+		
+		
+			The new implementation of relpipe-tr-sql is still a bit raw and will be tuned in the upcoming versions,
+			but it seems working quite well (with SQLite, PostgreSQL and MySQL on GNU/Linux).
+			As always, testers are welcomed.
+		
+		
+		
+			More details in the example: Accessing SQLite, PostgreSQL and MySQL through ODBC.
+		
+		
+		JACK (MIDI) input module
+		
+		
+			A powerful audio system called JACK allows us to
+			build pipelines consisting of audio interfaces, players, recorders, filters and effects…
+			and route sound streams (both PCM and MIDI) through them.
+			MIDI messages can come from keyboards or other hardware MIDI controllers or from MIDI players and other software.
+			Sometimes it is useful to check what is happening under the hood and examine particular MIDI messages
+			instead of just playing them on a sound module or synthesizer.
+			Now we can bridge two seemingly unrelated worlds: real-time audio and relational pipes.
+		
+		
+		
+		
+		
+			We can join the JACK graph with relpipe-in-jack command.
+			It does not consume STDIN, it gets events from JACK instead, so no other input data are needed.
+		
+		
+		
+			More details in the example: Monitoring MIDI messages using JACK.
+		
+		
+		Feature overview
+		
+		Data types
+		
+			boolean
+			variable-length signed integer (SLEB128)
+			string in UTF-8
+		
+		Inputs
+		
+			Recfile
+			XML
+			XMLTable
+			CSV
+			file system
+			CLI
+			fstab
+			SQL script
+			JACK
+		
+		Transformations
+		
+			sql: filtering and transformations using the SQL language
+			awk: filtering and transformations using the classic AWK tool and language
+			guile: filtering and transformations defined in the Scheme language using GNU Guile
+			grep: regular expression filter, removes unwanted records from the relation
+			cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)
+			sed: regular expression replacer
+			validator: just a pass-through filter that crashes on invalid data
+			python: highly experimental
+		
+		Streamlets
+		
+			xpath (example, unstable)
+			hash (example, unstable)
+			jar_info (example, unstable)
+			mime_type (example, unstable)
+			exiftool (example, unstable)
+			pid (example, unstable)
+			cloc (example, unstable)
+			exiv2 (example, unstable)
+			inode (example, unstable)
+			lines_count (example, unstable)
+			pdftotext (example, unstable)
+			pdfinfo (example, unstable)
+			tesseract (example, unstable)
+		
+		Outputs
+		
+			ASN.1 BER
+			Recfile
+			CSV
+			tabular
+			XML
+			nullbyte
+			GUI in Qt
+			ODS (LibreOffice)
+		
+		
+		New examples
+		
+			Accessing SQLite, PostgreSQL and MySQL through ODBC
+			Monitoring MIDI messages using JACK
+		
+		
+		Backward incompatible changes
+		
+		
+			The options --file and --file-keep in relpipe-tr-sql (and relpipe-in-sql, which is an alias for the same binary)
+			have been dropped.
+			These options were specific to SQLite and make no sense now, when we do not depend on particular DBMS and can use any engine for SQL processing
+			(even a remote one somewhere on the network that could not reach our local files).
+			However SQLite is still the default option and the:
+		
+		
+		relpipe-tr-sql --file 'myDatabase.sqlite'
+		
+		can be simply replaced by:
+		
+		relpipe-tr-sql --data-source-string 'Driver=SQLite3;Database=file:myDatabase.sqlite'
+		
+		
+			Bash-completion works and will suggest even the Driver=SQLite3;Database=file: part, so it is not necessary to memorize the connection string.
+			Frequently used databases can be configured in the ~/.odbc.ini file and then referenced just by their names using --data-source-name
+			(the data source names – DSN – are also suggested by Bash-completion).
+		
+		
+		
+			There is no built-in replacement for the --file-keep option.
+			But if the user wants to create a temporary file and delete it at the end of the transformation,
+			he can simply add rm -f myDatabase.sqlite to his script.
+		
+		
+		Installation
+		
+		
+			Instalation was tested on Debian GNU/Linux 10.2.
+			The process should be similar on other distributions.
+		
+		
+		
+		
+		
+			 are modular thus you can download and install only parts you need (the libraries are needed always).
+			Tools out-gui.qt and tr-python require additional libraries and are not built by default.
+		
+		
+	
+
+
\ No newline at end of file
diff -r 418e11eb6fea -r 192b0059a6c4 relpipe-data/roadmap.xml
--- a/relpipe-data/roadmap.xml	Thu May 21 01:23:23 2020 +0200
+++ b/relpipe-data/roadmap.xml	Sat Jun 06 01:57:24 2020 +0200
@@ -16,7 +16,7 @@
 			Released versions are described on the download page.
 		
 		
-		v0.16, v0.17, v0.18 etc.
+		v0.17, v0.18, v0.19 etc.
 		
 		
 			Releases for discussion and verification of the format and API design.
@@ -195,6 +195,7 @@
 		
Other libraries and tools:
 		
 			ORM API: mapping between classes/objects and relations/records
+			JDBC and ODBC drivers
 			schemas (XSD-like)
 			generators, compilers, validators, comparators
 			transformer helpers