|
1 <stránka |
|
2 xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" |
|
3 xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> |
|
4 |
|
5 <nadpis>Release v0.16</nadpis> |
|
6 <perex>new public release of Relational pipes</perex> |
|
7 <m:release>v0.16</m:release> |
|
8 |
|
9 <text xmlns="http://www.w3.org/1999/xhtml"> |
|
10 <p> |
|
11 We are pleased to introduce you the new development version of <m:name/>. |
|
12 This release brings an abstraction layer (ODBC) in the SQL transformation + several smaller improvements. |
|
13 </p> |
|
14 |
|
15 <ul> |
|
16 <li> |
|
17 <strong>ODBC in the <code>relpipe-tr-sql</code> module</strong>: see details below</li> |
|
18 <li> |
|
19 <strong>new input <code>relpipe-in-jack</code> module</strong>: see details below</li> |
|
20 <li> |
|
21 <strong>keyboard shortcuts in the <code>relpipe-out-gui</code> module</strong>: use Ctrl+PgUp and Ctrl+PgDown to switch panels (relations) and Ctrl+Q to quit</li> |
|
22 <li> |
|
23 <strong>record count in the <code>relpipe-out-xhtml</code> command</strong>: number of records is printed under the table (this command part of <code>relpipe-out-xml</code>, not a standalone module)</li> |
|
24 </ul> |
|
25 |
|
26 <p> |
|
27 See the <m:a href="examples">examples</m:a> and <m:a href="screenshots">screenshots</m:a> pages for details. |
|
28 </p> |
|
29 |
|
30 <p> |
|
31 Please note that this is still a development relasease and thus the API (libraries, CLI arguments, formats) might and will change. |
|
32 Any suggestions, ideas and bug reports are welcome in our <m:a href="contact">mailing list</m:a>. |
|
33 </p> |
|
34 |
|
35 <h2>ODBC in the SQL transformation module</h2> |
|
36 |
|
37 <p> |
|
38 Former versions of <code>relpipe-tr-sql</code> were tied to <a href="https://sqlite.org/">SQLite</a> |
|
39 and user had no option to change the <i>SQL engine</i>. |
|
40 However great SQLite is (and we are very thankful for it), having some particular DBMS (database management system) hard-coded in our program is too constraining. |
|
41 So we added an abstraction layer (ODBC) and get rid of the direct dependency on SQLite. |
|
42 Now any DBMS can be used with <m:name/>. |
|
43 </p> |
|
44 |
|
45 <p> |
|
46 ODBC (Open Database Connectivity) is an industry standard and provides API for accessing a DBMS. |
|
47 In late 80s several vendors (mostly from the Unix and database communities) established the SQL Access Group (SAG) |
|
48 and then specified the Call Level Interface (CLI). ODBC, which is based on CLI, was published in early 90s. |
|
49 ODBC is available on many operating systems and there are at least two free software implementations: |
|
50 <a href="http://www.unixodbc.org/">unixODBC</a> and <a href="http://www.iodbc.org/">iODBC</a>. |
|
51 We use unixODBC for development and testing. |
|
52 Future releases of <m:name/> should be tested also with other implementations and various database drivers. |
|
53 </p> |
|
54 |
|
55 <p> |
|
56 SQLite remains the default option |
|
57 (in the C++ implementation, while Java or other implementations may have different default and may use different abstraction layer like JDBC). |
|
58 We count on SQLite for future releases. It is the simplest way to get full SQL power in your relational pipeline. |
|
59 However, <code>relpipe-tr-sql</code> do not depend on SQLite and can be installed without it (and then used e.g. with PostgreSQL driver). |
|
60 Using different DBMS makes sense for two main reasons: |
|
61 </p> |
|
62 |
|
63 <ul> |
|
64 <li> |
|
65 We need specific features provided by the DBMS. |
|
66 It might be e.g. some functions for XML processing or some advanced SQL language constructs. |
|
67 Or maybe we have some business logic already implemented as SQL functions in e.g. PostgreSQL |
|
68 – now we can access this logic from our pipelines or seamlessly integrate it in our shell: |
|
69 <m:pre jazyk="bash"><![CDATA[cat source-data.csv | relpipe-in-csv \ |
|
70 | relpipe-tr-sql \ |
|
71 --data-source-name "MyDatabaseServer" \ |
|
72 --relation "transformed_data" " |
|
73 SELECT |
|
74 some_csv_field AS id, |
|
75 our_special_function(some_other, third_one) AS result |
|
76 FROM csv" \ |
|
77 | relpipe-out-xml \ |
|
78 | xsltproc template.xsl - > some-fancy-report.xhtml |
|
79 # or just: | relpipe-out-xhtml > some-generic-report.xhtml]]></m:pre> |
|
80 </li> |
|
81 <li> |
|
82 We need access to data in an existing database. |
|
83 The <code>relpipe-tr-sql</code> and <code>relpipe-in-sql</code> can be used as a generic database clients |
|
84 and are able to load relational data to and from any DBMS. |
|
85 We can also write a pipeline to transfer data between two different DBMS, do some ETL (extract, transform, load) tasks |
|
86 or just cache some result sets from a remote database in our local SQLite file. |
|
87 We can cache e.g. some codelist tables or other data for offline use: |
|
88 <m:pre jazyk="bash"><![CDATA[relpipe-in-sql \ |
|
89 --data-source-name "MyCompanyDatabase" \ |
|
90 --relation "country" "SELECT * FROM country" \ |
|
91 --relation "currency" "SELECT * FROM currency" \ |
|
92 --relation "exchange_rate" "SELECT * FROM exchange_rate WHERE …" \ |
|
93 --relation "phonebook" "SELECT * FROM phonebook" \ |
|
94 | relpipe-tr-sql \ |
|
95 --data-source-string 'Driver=SQLite3;Database=file:MyCachedCompanyData.sqlite']]></m:pre> |
|
96 In previous versions, we needed <a href="https://sql-dk.globalcode.info/">SQL-DK</a> for this scenario, |
|
97 now it is possible solely in <m:name/> without any other tools. |
|
98 But SQL-DK is still useful – especially if we have a JDBC driver but do not have an ODBC one |
|
99 (JDBC drivers and Java are also much more portable). |
|
100 </li> |
|
101 </ul> |
|
102 |
|
103 <p> |
|
104 n.b. However it still looks like executing a local command, we should be aware that while using a remote data source, |
|
105 our data travel to given remote server – this impacts performance and our privacy. |
|
106 Never use untrustworthy remote server for processing sensitive data (even if using just a temporary schema or tables). |
|
107 If SQLite is „too small“ then PostgreSQL installed on <i>localhost</i> is usually a good option. |
|
108 </p> |
|
109 |
|
110 <p> |
|
111 There are ODBC drivers for any conceivable database system. |
|
112 We can also write a custom driver for any other resource and just plug it in <m:name/> |
|
113 without recompiling (a driver is a shared library – simply an <code>.so</code> file). |
|
114 </p> |
|
115 |
|
116 <p> |
|
117 This release also comes with better diagnostics. This feature is not specific to ODBC, but was implemented during the rewrite of the database layer. |
|
118 So if we make a mistake in our query or try to create a table with the same name as already exists in the DB, we will get a useful message with detailed description of the problem |
|
119 (instead of a pointless failure notice in the previous version). |
|
120 </p> |
|
121 |
|
122 <p> |
|
123 The new implementation of <code>relpipe-tr-sql</code> is still a bit <i>raw</i> and will be tuned in the upcoming versions, |
|
124 but it seems working quite well (with SQLite, PostgreSQL and MySQL on GNU/Linux). |
|
125 As always, testers are welcomed. |
|
126 </p> |
|
127 |
|
128 <p> |
|
129 More details in the example: <m:a href="examples-tr-sql-odbc">Accessing SQLite, PostgreSQL and MySQL through ODBC</m:a>. |
|
130 </p> |
|
131 |
|
132 <h2>JACK (MIDI) input module</h2> |
|
133 |
|
134 <p> |
|
135 A powerful audio system called <a href="https://jackaudio.org/">JACK</a> allows us to |
|
136 build pipelines consisting of audio interfaces, players, recorders, filters and effects… |
|
137 and route sound streams (both PCM and MIDI) through them. |
|
138 MIDI messages can come from keyboards or other hardware MIDI controllers or from MIDI players and other software. |
|
139 Sometimes it is useful to check what is happening under the hood and examine particular MIDI messages |
|
140 instead of just playing them on a sound module or synthesizer. |
|
141 Now we can bridge two seemingly unrelated worlds: real-time audio and relational pipes. |
|
142 </p> |
|
143 |
|
144 <m:img src="img/jack-connections-1.png"/> |
|
145 |
|
146 <p> |
|
147 We can join the JACK graph with <code>relpipe-in-jack</code> command. |
|
148 It does not consume STDIN, it gets events from JACK instead, so no other input data are needed. |
|
149 </p> |
|
150 |
|
151 <p> |
|
152 More details in the example: <m:a href="examples-jack-midi-monitoring">Monitoring MIDI messages using JACK</m:a>. |
|
153 </p> |
|
154 |
|
155 <h2>Feature overview</h2> |
|
156 |
|
157 <h3>Data types</h3> |
|
158 <ul> |
|
159 <li m:since="v0.8">boolean</li> |
|
160 <li m:since="v0.15">variable-length signed integer (SLEB128)</li> |
|
161 <li m:since="v0.8">string in UTF-8</li> |
|
162 </ul> |
|
163 <h3>Inputs</h3> |
|
164 <ul> |
|
165 <li m:since="v0.11">Recfile</li> |
|
166 <li m:since="v0.9">XML</li> |
|
167 <li m:since="v0.13">XMLTable</li> |
|
168 <li m:since="v0.9">CSV</li> |
|
169 <li m:since="v0.9">file system</li> |
|
170 <li m:since="v0.8">CLI</li> |
|
171 <li m:since="v0.8">fstab</li> |
|
172 <li m:since="v0.14">SQL script</li> |
|
173 <li m:since="v0.16">JACK</li> |
|
174 </ul> |
|
175 <h3>Transformations</h3> |
|
176 <ul> |
|
177 <li m:since="v0.13">sql: filtering and transformations using the SQL language</li> |
|
178 <li m:since="v0.12">awk: filtering and transformations using the classic AWK tool and language</li> |
|
179 <li m:since="v0.10">guile: filtering and transformations defined in the Scheme language using GNU Guile</li> |
|
180 <li m:since="v0.8">grep: regular expression filter, removes unwanted records from the relation</li> |
|
181 <li m:since="v0.8">cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)</li> |
|
182 <li m:since="v0.8">sed: regular expression replacer</li> |
|
183 <li m:since="v0.8">validator: just a pass-through filter that crashes on invalid data</li> |
|
184 <li m:since="v0.8">python: highly experimental</li> |
|
185 </ul> |
|
186 <h3>Streamlets</h3> |
|
187 <ul> |
|
188 <li m:since="v0.15">xpath (example, unstable)</li> |
|
189 <li m:since="v0.15">hash (example, unstable)</li> |
|
190 <li m:since="v0.15">jar_info (example, unstable)</li> |
|
191 <li m:since="v0.15">mime_type (example, unstable)</li> |
|
192 <li m:since="v0.15">exiftool (example, unstable)</li> |
|
193 <li m:since="v0.15">pid (example, unstable)</li> |
|
194 <li m:since="v0.15">cloc (example, unstable)</li> |
|
195 <li m:since="v0.15">exiv2 (example, unstable)</li> |
|
196 <li m:since="v0.15">inode (example, unstable)</li> |
|
197 <li m:since="v0.15">lines_count (example, unstable)</li> |
|
198 <li m:since="v0.15">pdftotext (example, unstable)</li> |
|
199 <li m:since="v0.15">pdfinfo (example, unstable)</li> |
|
200 <li m:since="v0.15">tesseract (example, unstable)</li> |
|
201 </ul> |
|
202 <h3>Outputs</h3> |
|
203 <ul> |
|
204 <li m:since="v0.11">ASN.1 BER</li> |
|
205 <li m:since="v0.11">Recfile</li> |
|
206 <li m:since="v0.9">CSV</li> |
|
207 <li m:since="v0.8">tabular</li> |
|
208 <li m:since="v0.8">XML</li> |
|
209 <li m:since="v0.8">nullbyte</li> |
|
210 <li m:since="v0.8">GUI in Qt</li> |
|
211 <li m:since="v0.8">ODS (LibreOffice)</li> |
|
212 </ul> |
|
213 |
|
214 <h2>New examples</h2> |
|
215 <ul> |
|
216 <li><m:a href="examples-tr-sql-odbc">Accessing SQLite, PostgreSQL and MySQL through ODBC</m:a></li> |
|
217 <li><m:a href="examples-jack-midi-monitoring">Monitoring MIDI messages using JACK</m:a></li> |
|
218 </ul> |
|
219 |
|
220 <h2>Backward incompatible changes</h2> |
|
221 |
|
222 <p> |
|
223 The options <code>--file</code> and <code>--file-keep</code> in <code>relpipe-tr-sql</code> (and <code>relpipe-in-sql</code>, which is an alias for the same binary) |
|
224 have been dropped. |
|
225 These options were specific to SQLite and make no sense now, when we do not depend on particular DBMS and can use any <i>engine</i> for SQL processing |
|
226 (even a remote one somewhere on the network that could not reach our local files). |
|
227 However SQLite is still the default option and the: |
|
228 </p> |
|
229 |
|
230 <m:pre jazyk="bash">relpipe-tr-sql --file 'myDatabase.sqlite'</m:pre> |
|
231 |
|
232 <p>can be simply replaced by:</p> |
|
233 |
|
234 <m:pre jazyk="bash">relpipe-tr-sql --data-source-string 'Driver=SQLite3;Database=file:myDatabase.sqlite'</m:pre> |
|
235 |
|
236 <p> |
|
237 Bash-completion works and will suggest even the <code>Driver=SQLite3;Database=file:</code> part, so it is not necessary to memorize the connection string. |
|
238 Frequently used databases can be configured in the <code>~/.odbc.ini</code> file and then referenced just by their names using <code>--data-source-name</code> |
|
239 (the data source names – DSN – are also suggested by Bash-completion). |
|
240 </p> |
|
241 |
|
242 <p> |
|
243 There is no built-in replacement for the <code>--file-keep</code> option. |
|
244 But if the user wants to create a temporary file and delete it at the end of the transformation, |
|
245 he can simply add <code>rm -f myDatabase.sqlite</code> to his script. |
|
246 </p> |
|
247 |
|
248 <h2>Installation</h2> |
|
249 |
|
250 <p> |
|
251 Instalation was tested on Debian GNU/Linux 10.2. |
|
252 The process should be similar on other distributions. |
|
253 </p> |
|
254 |
|
255 <m:pre src="examples/release-v0.16.sh" jazyk="bash" odkaz="ano"/> |
|
256 |
|
257 <p> |
|
258 <m:name/> are modular thus you can download and install only parts you need (the libraries are needed always). |
|
259 Tools <code>out-gui.qt</code> and <code>tr-python</code> require additional libraries and are not built by default. |
|
260 </p> |
|
261 |
|
262 </text> |
|
263 |
|
264 </stránka> |