|
1 <stránka |
|
2 xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" |
|
3 xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> |
|
4 |
|
5 <nadpis>Integrating Relational pipes with GNU Recutils</nadpis> |
|
6 <perex>using recfile format as input and output + filtering</perex> |
|
7 <m:pořadí-příkladu>01900</m:pořadí-příkladu> |
|
8 |
|
9 <text xmlns="http://www.w3.org/1999/xhtml"> |
|
10 |
|
11 <p> |
|
12 Recfile is the native format of <a href="https://www.gnu.org/software/recutils/">GNU Recutils</a>. |
|
13 Recfiles are text files that contain records of various types. |
|
14 They are human-editable and serve as a simple databases. |
|
15 <m:name/> support input and output in this format since v0.11. |
|
16 </p> |
|
17 |
|
18 |
|
19 <p> |
|
20 We can convert any relational data to the recfile format by using <code>relpipe-out-recfile</code> – e.g. our <code>fstab</code> will look like this: |
|
21 </p> |
|
22 |
|
23 <m:pre jazyk="text" src="examples/relpipe-out-fstab.rec.txt"/> |
|
24 |
|
25 <p> |
|
26 Then we can edit this data (e.g. in GNU Emacs which has mode for this format) or store it in a version control system like Mercurial or Git. |
|
27 Because it is a text format (like XML, which is also supported and good for this purpose), |
|
28 we can efficiently track changes in data across versions, do <code>diff</code> or (with some care) even <code>patch</code>. |
|
29 And we can use whole GNU Recutils toolchain while working with such data. |
|
30 </p> |
|
31 |
|
32 <p> |
|
33 Obligatory example of filtering our <code>fstab</code>: |
|
34 </p> |
|
35 |
|
36 <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab | relpipe-out-recfile | recsel -e "type = 'btrfs' || type = 'xfs'"]]></m:pre> |
|
37 |
|
38 <p>Will give us a recfile:</p> |
|
39 |
|
40 <m:pre jazyk="text"><![CDATA[scheme: UUID |
|
41 device: a2b5f230-a795-4f6f-a39b-9b57686c86d5 |
|
42 mount_point: /home |
|
43 type: btrfs |
|
44 options: relatime |
|
45 dump: 0 |
|
46 pass: 2 |
|
47 |
|
48 scheme: |
|
49 device: /dev/mapper/sdf_crypt |
|
50 mount_point: /mnt/private |
|
51 type: xfs |
|
52 options: relatime |
|
53 dump: 0 |
|
54 pass: 2]]></m:pre> |
|
55 |
|
56 <p>And we can convert it back to the relational format using <code>relpipe-in-recfile</code>:</p> |
|
57 <m:pre jazyk="bash"><![CDATA[relpipe-in-fstab \ |
|
58 | relpipe-out-recfile \ |
|
59 | recsel -e "type = 'btrfs' || type = 'xfs'" \ |
|
60 | relpipe-in-recfile \ |
|
61 | relpipe-out-tabular]]></m:pre> |
|
62 |
|
63 <p>and print as a table in our terminal:</p> |
|
64 <m:pre jazyk="text"><![CDATA[recfile: |
|
65 ╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬──────────────────┬───────────────┬───────────────╮ |
|
66 │ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (string) │ pass (string) │ |
|
67 ├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼──────────────────┼───────────────┼───────────────┤ |
|
68 │ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │ |
|
69 │ │ /dev/mapper/sdf_crypt │ /mnt/private │ xfs │ relatime │ 0 │ 2 │ |
|
70 ╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴──────────────────┴───────────────┴───────────────╯ |
|
71 Record count: 2]]></m:pre> |
|
72 |
|
73 <p> |
|
74 n.b. in the v0.11 the conversion to recfiles and back is not 100% lossless (unlike XML) |
|
75 because <m:name/> support only three data types (string, unsigned integer and boolean) in this version; |
|
76 this will be improved in later releases (more data types are planned before v1.0) |
|
77 </p> |
|
78 |
|
79 <p> |
|
80 <m:name/> can be also used together with <a href="https://sql-dk.globalcode.info/">SQL-DK</a> (in 2019-03-05 development version) |
|
81 to pipe data from big relational databases like PostgreSQL or MariaDB to other formats like recfiles. |
|
82 Having a script: |
|
83 </p> |
|
84 |
|
85 <m:pre jazyk="text" src="examples/sql-dk_pg_1.sh" odkaz="ano"/> |
|
86 |
|
87 <p> |
|
88 We can convert result sets from any SQL queries to relational format and then work with such data without connection to the original database. |
|
89 Thus we can cache (<em>materialize</em>) the results locally in a file and use them even offline. |
|
90 Or we can run the SQL query each time and have fresh data: |
|
91 </p> |
|
92 |
|
93 <m:pre jazyk="text"><![CDATA[sql-dk_pg_1.sh | relpipe-out-recfile]]></m:pre> |
|
94 |
|
95 <p>Will result in:</p> |
|
96 <m:pre jazyk="text" src="examples/sql-dk_pg_1.rec.txt"/> |
|
97 |
|
98 <p>Or we can view the data in classic tabular way using <code>relpipe-out-tabular</code>:</p> |
|
99 <m:pre jazyk="text" src="examples/sql-dk_pg_1.tabular.txt"/> |
|
100 |
|
101 <p> |
|
102 Materialized (or fresh) data from the database can be further transformed |
|
103 using <code>relpipe-tr-*</code> commands like grep, sed, cut, guile, |
|
104 or (through the recfile conversion) by the <code>recsel</code> command from GNU Recutils. |
|
105 </p> |
|
106 |
|
107 <p> |
|
108 The <code>relpipe-in-recfile</code> will help with conversion of recfiles to various formats like XHTML, |
|
109 pretty-printing or with xargs-like processing |
|
110 (using <code>relpipe-out-nullbyte</code> and regular <code>xargs</code> or <code>read_nullbyte</code> function |
|
111 as described in the <m:a href="examples-out-bash">Writing an output filter in Bash</m:a> example). |
|
112 Thus we can have data-driven Bash scripts based on our recfiles. |
|
113 </p> |
|
114 |
|
115 |
|
116 </text> |
|
117 |
|
118 </stránka> |