<stránka

	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"

	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">

	<nadpis>Running SQL JOINs on multiple CSV files</nadpis>

	<perex>query a collection of (not only) CSV files using SQL</perex>

	<m:pořadí-příkladu>05100</m:pořadí-příkladu>

	<text xmlns="http://www.w3.org/1999/xhtml">

		<p>

			CSV (<m:a href="4180" typ="rfc">RFC 4180</m:a>) is quite good solution when we want to store or share relational data in a simple text format –

			both, human-readable and well supported by many existing applications and libraries.

			We have even ready-to-use GUI editors, so called spreadsheets e.g. LibreOffice Calc.

			(on the other hand, such simple formats have usually some drawbacks…)

		</p>

		<p>

			In this example, we will show how to query a set of CSV files like it was a relational database.

		</p>

		<p>Suppose we have a CSV file describing our network interfaces:</p>

		<m:pre jazyk="text"><![CDATA[address,name

00:00:00:00:00:00,lo

00:D0:D8:00:26:00,eth0

00:01:02:01:33:70,eth1]]></m:pre>

		<p>and another CSV file with IP addresses assigned to them:</p>

		<m:pre jazyk="text"><![CDATA[address,mask,version,interface

127.0.0.1,8,4,lo

::1,128,6,lo

192.168.1.2,24,4,eth0

192.168.1.8,24,4,eth0

10.21.12.24,24,4,eth0

75.748.86.91,95,4,eth1

23.75.345.200,95,4,eth1

2a01:430:2e::cafe:babe,64,6,eth1]]></m:pre>

		<h2>Loading a CSV file and running basic queries</h2>

		<p>

			Simplest task is to parse the file and print it as a table in our terminal or convert it to another format (XML, Recfile, ODS, YAML, XHTML, ASN.1 etc.)

			We can also add <code>relpipe-tr-sql</code> in the middle of our pipeline and run some SQL queries –

			transform data on-the-fly and send the query result to the <code>relpipe-out-tabular</code> (or other output filter) in place of the original data.

			For now, we will filter just the IPv6 addresses:

		</p>

		<m:pre jazyk="bash"><![CDATA[cat ip.csv \

	| relpipe-in-csv --relation 'ip' \

	| relpipe-tr-sql \

		--relation 'ipv6' "SELECT * FROM ip WHERE version = 6" \

	| relpipe-out-tabular]]></m:pre>

		<p>and get them printed:</p>

		<m:pre jazyk="text"><![CDATA[ipv6:

 ╭────────────────────────┬───────────────┬──────────────────┬────────────────────╮

 │ address       (string) │ mask (string) │ version (string) │ interface (string) │

 ├────────────────────────┼───────────────┼──────────────────┼────────────────────┤

 │ ::1                    │ 128           │ 6                │ lo                 │

 │ 2a01:430:2e::cafe:babe │ 64            │ 6                │ eth1               │

 ╰────────────────────────┴───────────────┴──────────────────┴────────────────────╯

Record count: 2]]></m:pre>

		<p>

			It is alo possible to run several queries at once

			and thanks to the <m:name/> format, the result sets are not mixed together, their boundaries are retained and everything is safely passed to the next stage of the pipeline:

		</p>

		<m:pre jazyk="bash"><![CDATA[cat ip.csv \

	| relpipe-in-csv --relation 'ip' \

	| relpipe-tr-sql \

		--relation 'ipv4' "SELECT * FROM ip WHERE version = 4" \

		--relation 'ipv6' "SELECT * FROM ip WHERE version = 6" \

	| relpipe-out-tabular]]></m:pre>

		<p>resulting in two nice tables:</p>

		<m:pre jazyk="text"><![CDATA[ipv4:

 ╭──────────────────┬───────────────┬──────────────────┬────────────────────╮

 │ address (string) │ mask (string) │ version (string) │ interface (string) │

 ├──────────────────┼───────────────┼──────────────────┼────────────────────┤

 │ 127.0.0.1        │ 8             │ 4                │ lo                 │

 │ 192.168.1.2      │ 24            │ 4                │ eth0               │

 │ 192.168.1.8      │ 24            │ 4                │ eth0               │

 │ 10.21.12.24      │ 24            │ 4                │ eth0               │

 │ 75.748.86.91     │ 95            │ 4                │ eth1               │

 │ 23.75.345.200    │ 95            │ 4                │ eth1               │

 ╰──────────────────┴───────────────┴──────────────────┴────────────────────╯

Record count: 6

ipv6:

 ╭────────────────────────┬───────────────┬──────────────────┬────────────────────╮

 │ address       (string) │ mask (string) │ version (string) │ interface (string) │

 ├────────────────────────┼───────────────┼──────────────────┼────────────────────┤

 │ ::1                    │ 128           │ 6                │ lo                 │

 │ 2a01:430:2e::cafe:babe │ 64            │ 6                │ eth1               │

 ╰────────────────────────┴───────────────┴──────────────────┴────────────────────╯

Record count: 2]]></m:pre>

		<h2>Using parametrized queries to avoid SQL injection</h2>

		<p>

			When <code>"4"</code> and <code>"6"</code> are not fixed values, we should not glue them to the query string like <code>version = $version</code>,

			because it is a dangerous practice that may lead to SQL injection.

			We have parametrized queries for such tasks:

		</p>

		<m:pre jazyk="bash"><![CDATA[--relation 'ipv6' "SELECT * FROM ip WHERE version = ?" --parameter "6"]]></m:pre>

		<h2>Running SQL JOINs, UNIONs etc. on multiple CSV files</h2>

		<p>

			To load multiple CSV files into our <i>in-memory database</i>, we just concatenate the relational streams

			using the means of our shell – the semicolons and parenthesis:

		</p>

		<m:pre jazyk="bash"><![CDATA[(relpipe-in-csv --relation 'ip' < ip.csv; relpipe-in-csv --relation 'nic' < nic.csv) \

	| relpipe-tr-sql \

		--relation 'ip_nic' "SELECT * FROM ip JOIN nic ON nic.name = ip.interface" \

	| relpipe-out-tabular]]></m:pre>

		<p>Generic version that loads all <code>*.csv</code> files:</p>

		<m:pre jazyk="bash"><![CDATA[for csv in *.csv; do relpipe-in-csv --relation "$(basename "$csv" .csv)" < "$csv"; done \

	| relpipe-tr-sql \

		--relation 'ip_nic' "SELECT * FROM ip JOIN nic ON nic.name = ip.interface" \

	| relpipe-out-tabular]]></m:pre>

		<p>Then we can JOIN data from multiple CSV files or do UNIONs, INTERSECTions etc.</p>

		<m:pre jazyk="text"><![CDATA[ip_nic:

 ╭────────────────────────┬───────────────┬──────────────────┬────────────────────┬───────────────────┬───────────────╮

 │ address       (string) │ mask (string) │ version (string) │ interface (string) │ address  (string) │ name (string) │

 ├────────────────────────┼───────────────┼──────────────────┼────────────────────┼───────────────────┼───────────────┤

 │ 127.0.0.1              │ 8             │ 4                │ lo                 │ 00:00:00:00:00:00 │ lo            │

 │ ::1                    │ 128           │ 6                │ lo                 │ 00:00:00:00:00:00 │ lo            │

 │ 192.168.1.2            │ 24            │ 4                │ eth0               │ 00:D0:D8:00:26:00 │ eth0          │

 │ 192.168.1.8            │ 24            │ 4                │ eth0               │ 00:D0:D8:00:26:00 │ eth0          │

 │ 10.21.12.24            │ 24            │ 4                │ eth0               │ 00:D0:D8:00:26:00 │ eth0          │

 │ 75.748.86.91           │ 95            │ 4                │ eth1               │ 00:01:02:01:33:70 │ eth1          │

 │ 23.75.345.200          │ 95            │ 4                │ eth1               │ 00:01:02:01:33:70 │ eth1          │

 │ 2a01:430:2e::cafe:babe │ 64            │ 6                │ eth1               │ 00:01:02:01:33:70 │ eth1          │

 ╰────────────────────────┴───────────────┴──────────────────┴────────────────────┴───────────────────┴───────────────╯

Record count: 8]]></m:pre>

		<h2>Leveraging shell functions</h2>

		<p>

			Good practice is to wrap common code blocks into functions and thus make them reusable.

			In shell, the function still works with input and output streams and we can use them when building our pipelines.

			Shell functions can be seen as named reusable parts of a pipeline.

		</p>

		<m:pre jazyk="bash"><![CDATA[csv2relation()  { for file; do relpipe-in-csv --relation "$(basename "$file" .csv)" < "$file"; done }

do_query()      { relpipe-tr-sql --relation 'ip_nic' "SELECT * FROM ip JOIN nic ON nic.name = ip.interface"; }

format_result() { [[ -t 1 ]] && relpipe-out-tabular || cat; }

csv2relation *.csv | do_query | format_result]]></m:pre>

		<p>

			The <code>format_result()</code> function checks whether the STDOUT is a terminal or not.

			and when printing to the terminal, it generates a table.

			When writing to a regular file or STDIN of another process, it passes through original relational data.

			Thus <code>./our-script.sh</code> will print a nice table in the terminal, while <code>./our-script.sh > data.rp</code> will create a file containing machine-readable data

			and <code>./our-script.sh | relpipe-out-xhtml > report.xhtml</code> will create an XHTML report and <code>./our-script.sh | relpipe-out-gui</code> will show a GUI window full of tables and maybe also charts.

		</p>

		<m:img src="img/csv-sql-gui-ip-address-counts.png"/>

		<m:pre jazyk="sql"><![CDATA[SELECT

	nic.name || ' IPv' || ip.version AS label,

	nic.name AS interface,

	ip.version AS ip_version,

	count(*) AS address_count

FROM nic 

	LEFT JOIN ip ON (ip.interface = nic.name)

GROUP BY nic.name, ip.version

ORDER BY count(*) DESC]]></m:pre>

		<h2>Makefile version</h2>

		<p>

			Shell scripts are not the only way to structure and organize our pipelines or generally our data-processing code.

			We can also use Make (the tool intended mainly for building sofware), write a <i>Makefile</i> and organize our code around some temporary files and other targets instead of functions.

		</p>

		<m:pre jazyk="Makefile"><![CDATA[all: print_summary

.PHONY: clean print_summary run_commands

clean:

	rm -rf *.rp

%.rp: %.csv

	relpipe-in-csv --relation "$(basename $(<))" < $(<) > $(@)

define SQL_IP_NIC

	SELECT

		ip.address AS ip_address,

		nic.name AS interface,

		nic.address AS mac_address

	FROM ip

		JOIN nic ON (nic.name = ip.interface)

endef

export SQL_IP_NIC

define SQL_COUNT_VERSIONS

	SELECT

		interface,

		count(CASE WHEN version=4 THEN 1 ELSE NULL END) AS ipv4_count,

		count(CASE WHEN version=6 THEN 1 ELSE NULL END) AS ipv6_count

	FROM ip

	GROUP BY interface

	ORDER BY interface

endef

export SQL_COUNT_VERSIONS

# Longer SQL queries are better kept in separate .sql files,

# because we can enjoy syntax highlighting and other support in our editors.

# Then we use it like this: --relation "ip_nic" "$$(cat ip_nic.sql)"

summary.rp: nic.rp ip.rp

	cat $(^) \

		| relpipe-tr-sql \

			--relation "ip_nic" "$$SQL_IP_NIC" \

			--relation "counts" "$$SQL_COUNT_VERSIONS" \

		> $(@)

print_summary: summary.rp

	cat $(<) | relpipe-out-tabular

]]></m:pre>

	<p>

		We can even combine advantages of Make and Bash together (without calling or including Bash scripts from Make)

		and have reusable shell functions available in the Makefile:

	</p>

<m:pre jazyk="text"><![CDATA[

SHELL=bash

BASH_FUNC_read_nullbyte%%=() { local IFS=; for v in "$$@"; do export "$$v"; read -r -d '' "$$v"; done }

export BASH_FUNC_read_nullbyte%%]]></m:pre>

	<p>usage example:</p>

<m:pre jazyk="Makefile"><![CDATA[

run_commands: summary.rp

	cat $(<) \

		| relpipe-tr-cut --relation 'ip_nic' --invert-match relation true \

		| relpipe-out-nullbyte \

		| while read_nullbyte ip_address interface mac_address; do\

			echo "network interface $$interface ($$mac_address) has IP address $$ip_address"; \

		done;

]]></m:pre>

		<p>

			Both approaches – the shell script and the Makefile – have pros and cons.

			With Makefile, we usually create some temporary files containing intermediate results.

			That avoids streaming. But on the other hand, we process (parse, transform, filter, format etc.) only data that have been changed.

		</p>

	</text>

</stránka>