relpipe-data/examples-in-sql-reading-sql.xml
author František Kučera <franta-hg@frantovo.cz>
Mon, 21 Feb 2022 00:43:11 +0100
branchv_0
changeset 329 5bc2bb8b7946
parent 327 66070d82e12b
permissions -rw-r--r--
Release v0.18
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
278
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     1
<stránka
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     2
	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     3
	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     4
	
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     5
	<nadpis>Reading SQL scripts</nadpis>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     6
	<perex>read an SQL file and display it in a tabular way</perex>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     7
	<m:pořadí-příkladu>03200</m:pořadí-příkladu>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     8
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     9
	<text xmlns="http://www.w3.org/1999/xhtml">
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    10
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    11
		<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    12
			SQL scripts containing
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    13
			DDL (Data Definition Language) and DML (Data Manipulation Language)
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    14
			contain both data structures (relations) and data (records).
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    15
			Simple example:
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    16
		</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    17
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    18
		<m:pre jazyk="sql" src="examples/relpipe-in-sql-1.sql"/>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    19
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    20
		<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    21
			We can read such data using the <code>relpipe-in-sql</code> command 
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    22
			in a similar way we read CSV, XML or Recfile streams – just pipe the stream into the particular input filter
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    23
			and let it convert data to the relational format:
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    24
		</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    25
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    26
		<m:pre jazyk="bash"><![CDATA[cat relpipe-in-sql-1.sql | relpipe-in-sql | relpipe-out-tabular]]></m:pre>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    27
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    28
		<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    29
			And in the next step we use an output filter and covert relational data to some other format e.g. the tabular output displayed in our terminal:
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    30
		</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    31
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    32
		<m:pre jazyk="text" src="examples/relpipe-in-sql-1.txt"/>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    33
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    34
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    35
		<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    36
			Of course, we can add further steps in our pipeline and use any transformation tool for filtering or modifying data:
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    37
		</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    38
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    39
		<m:pre jazyk="bash"><![CDATA[# AWK transformation:
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    40
cat relpipe-in-sql-1.sql \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    41
	| relpipe-in-sql \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    42
	| relpipe-tr-awk \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    43
		--relation 'a' --where 'message == "Hello,"' \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    44
		--relation '.*' --drop \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    45
	| relpipe-out-tabular
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    46
316
d7ae02390fac relpipe-tr-guile.cpp → relpipe-tr-scheme.cpp
František Kučera <franta-hg@frantovo.cz>
parents: 278
diff changeset
    47
# Scheme transformation:
278
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    48
cat relpipe-in-sql-1.sql \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    49
	| relpipe-in-sql \
316
d7ae02390fac relpipe-tr-guile.cpp → relpipe-tr-scheme.cpp
František Kučera <franta-hg@frantovo.cz>
parents: 278
diff changeset
    50
	| relpipe-tr-scheme \
278
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    51
		--relation 'a' --where '(string= $message "Hello,")' \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    52
		--relation '.*' --drop \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    53
	| relpipe-out-tabular]]></m:pre>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    54
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    55
		<p>and get filtered output:</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    56
		<m:pre jazyk="text" src="examples/relpipe-in-sql-1.filtered.txt"/>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    57
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    58
		<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    59
			However, it is usually not necessary, because once we have data in an in-memory database (which happens on-the-fly in the <code>relpipe-in-sql</code> step),
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    60
			we can use the SQL language for filtering and transformations and get the same output as above:
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    61
		</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    62
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    63
		<m:pre jazyk="bash"><![CDATA[cat relpipe-in-sql-1.sql \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    64
	| relpipe-in-sql \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    65
		--relation 'a' "SELECT * FROM a WHERE message = 'Hello,'" \
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    66
	| relpipe-out-tabular]]></m:pre>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    67
	
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    68
	<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    69
		Actually, the <code>relpipe-in-sql</code> is just a symbolic link to the <code>relpipe-tr-sql</code> 
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    70
		and have the same capabilities (with just bit different default behavior to match general logic of the input filters).
316
d7ae02390fac relpipe-tr-guile.cpp → relpipe-tr-scheme.cpp
František Kučera <franta-hg@frantovo.cz>
parents: 278
diff changeset
    71
		So if we do not need special feautres of Scheme, AWK or other transformation tool, it is better to stay with SQL (when we already use the SQL module).
278
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    72
	</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    73
	
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    74
	<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    75
		The difference between <code>relpipe-tr-sql</code> and <code>relpipe-in-sql</code> is that
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    76
		the SQL input filter reads SQL scripts from the STDIN while the SQL transformation reads relational data from the STDIN.
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    77
		And the input filter has implicit <code>--copy '.*'</code> option, if executed without any arguments (so it passes through all relations from the input without need of writing any SELECTs).
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    78
		We can override this default behavior by using any argument – we can e.g. copy only certain relations using <code>--copy 'a|c'</code> (it is a regular expression for matching relation names)
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    79
		or rename them: <code>--copy-renamed 'a|c' 'copy_of_$0'</code> or run arbitrary SELECT: <code>--relation 'a' "SELECT * FROM …"</code> as we have seen above.
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    80
	</p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    81
	
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    82
	<p>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    83
		SQL scripts can be used for storing relational data as an alternative to other human-readable and human-editable text formats like XML, CSV or Recfiles.
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    84
		And compared to the other formats, SQL scripts may contain even some logic (e.g. call SQL functions) or views.
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    85
	</p>
327
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    86
	
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    87
	<p>
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    88
		n.b. The SQL script does not contain only data – it is an executable script and running it might be dangerous.
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    89
		Depending on SQL engine used (the default one is SQLite, but others like PostgreSQL or MySQL/MariaDB can be used),
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    90
		such script may call various functions and some of them might read or write local files or do some other unsafe operations.
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    91
		Thus the SQL scripts comming from untrusted sources must be carefully reviewed or executed in an isolated environment (sandbox).
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    92
		We can run <code>relpipe-in-sql</code> using <code>sudo</code> under an unprivileged account or using <code>ssh</code> doing the same even on a remote machine (virtual od physical one or a container dedicated for such dirty work).
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    93
		Or we can use the <code>--data-source-name</code> or <code>--data-source-string</code> options and run such script on a remote DBMS under an unprivileged database account or on a sandbox database that will be destroyed or refreshed after use.
66070d82e12b relpipe-in-sql: security note
František Kučera <franta-hg@frantovo.cz>
parents: 316
diff changeset
    94
	</p>
278
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    95
		
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    96
	</text>
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    97
ae17db13569c examples: Reading SQL scripts
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    98
</stránka>