author | František Kučera <franta-hg@frantovo.cz> |
Sun, 25 Nov 2018 19:58:06 +0100 | |
branch | v_0 |
changeset 144 | ee7e96151673 |
parent 143 | 297da74fcab2 |
child 145 | 42bbbccd87f3 |
permissions | -rw-r--r-- |
23
0d2729ed16ed
zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents:
18
diff
changeset
|
1 |
<stránka |
0d2729ed16ed
zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents:
18
diff
changeset
|
2 |
xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" |
0d2729ed16ed
zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents:
18
diff
changeset
|
3 |
xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> |
0d2729ed16ed
zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents:
18
diff
changeset
|
4 |
|
137 | 5 |
<nadpis>Relational pipes</nadpis> |
6 |
<perex>Official homepage of Relational pipes.</perex> |
|
4
1bb39595a51c
genrování hlavní nabídky #1
František Kučera <franta-hg@frantovo.cz>
parents:
2
diff
changeset
|
7 |
<pořadí>10</pořadí> |
1bb39595a51c
genrování hlavní nabídky #1
František Kučera <franta-hg@frantovo.cz>
parents:
2
diff
changeset
|
8 |
|
2
ab9099ff88fa
vkládání zápatí, jmenné prostory, saxon
František Kučera <franta-hg@frantovo.cz>
parents:
1
diff
changeset
|
9 |
<text xmlns="http://www.w3.org/1999/xhtml"> |
ab9099ff88fa
vkládání zápatí, jmenné prostory, saxon
František Kučera <franta-hg@frantovo.cz>
parents:
1
diff
changeset
|
10 |
<p> |
143 | 11 |
One of the great parts of the <m:unix/> |
12 |
<m:podČarou> |
|
13 |
<m:unix tvar="vysvětlivka"/> |
|
14 |
</m:podČarou> culture is the invention<m:podČarou>which is attributed to Doug McIlroy, see <a href="http://www.catb.org/~esr/writings/taoup/html/ch07s02.html#plumbing">The Art of Unix Programming: Pipes, Redirection, and Filters</a></m:podČarou> |
|
15 |
of <em>pipes</em> and the idea<m:podČarou>see <a href="http://www.catb.org/~esr/writings/taoup/html/ch01s06.html">The Art of Unix Programming: Basics of the Unix Philosophy</a></m:podČarou> |
|
16 |
that <em>one program should do one thing and do it well</em>. |
|
87
25dec6931f18
Lepší odsazení, tabulátory.
František Kučera <franta-hg@frantovo.cz>
parents:
23
diff
changeset
|
17 |
</p> |
138 | 18 |
|
143 | 19 |
<p> |
20 |
Each running program (process) has one input stream (called standard input or STDIN) and one output stream (called standard output or STDOUT) and also one additional output stream for logging/errors/warnings (STDERR). |
|
21 |
We can connect programs and pass the STDOUT of first one to the STDIN of the second one (etc.) using pipes. |
|
22 |
</p> |
|
144
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
23 |
|
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
24 |
<p> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
25 |
A classic pipeline example (<m:a href="classic-example">explained</m:a>): |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
26 |
</p> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
27 |
|
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
28 |
<m:classic-example/> |
143 | 29 |
|
30 |
<!-- |
|
31 |
<m:diagram orientace="vodorovně"> |
|
32 |
node[shape=box]; |
|
33 |
||
34 |
cat [label="cat /etc/fstab"]; |
|
35 |
dd []; |
|
36 |
grep [label="grep tmpfs"]; |
|
37 |
log [label="/tmp/dd.log"]; |
|
38 |
||
39 |
cat -> dd [label="STDOUT → STDIN"]; |
|
40 |
dd -> grep [label="STDOUT → STDIN"]; |
|
41 |
dd -> log [label="STDERR → file"]; |
|
42 |
</m:diagram> |
|
43 |
--> |
|
44 |
||
45 |
<p> |
|
46 |
According to this principle we can build complex and powerful programs (pipelines) by composing several simple, single-purpose and reusable programs. |
|
47 |
Such single-purpose programs (often called <em>filters</em>) are much easier to create, test and optimize and their authors don't have to bother about the complexity of the final pipeline. |
|
48 |
They even don't have to know, how their programs will be used in the future by others. |
|
144
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
49 |
This is a great design principle that brings us advanced flexibility, reusability, efficiency and reliability. |
143 | 50 |
Being in any role (author of a filter, builder of a pipeline etc.), we can always focus on our task only and do it well. |
51 |
And we can collaborate with others even if we don't know about them and we don't know that we are collaborating. |
|
52 |
Now think about putting this together with the free software ideas... How very! |
|
53 |
</p> |
|
54 |
||
55 |
<!-- |
|
138 | 56 |
<m:diagram orientace="vodorovně"> |
57 |
compound=true; |
|
58 |
node[shape=box]; |
|
59 |
||
60 |
subgraph cluster_in { |
|
61 |
label = "Inputs:"; |
|
62 |
cli; |
|
63 |
fstab; |
|
64 |
} |
|
65 |
||
66 |
subgraph cluster_tr { |
|
67 |
label = "Transformations:"; |
|
68 |
grep; |
|
69 |
sed; |
|
70 |
} |
|
71 |
||
72 |
subgraph cluster_out { |
|
73 |
label = "Outputs:"; |
|
74 |
xml; |
|
75 |
tabular; |
|
76 |
gui; |
|
77 |
} |
|
78 |
||
79 |
cli -> grep [ltail=cluster_in, lhead=cluster_tr]; |
|
80 |
grep -> xml [ltail=cluster_tr, lhead=cluster_out]; |
|
81 |
// cli -> xml [ltail=cluster_in, lhead=cluster_out]; |
|
82 |
||
83 |
</m:diagram> |
|
143 | 84 |
--> |
85 |
||
144
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
86 |
|
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
87 |
<p>Bytes, text, structured data? XML, YAML, JSON, ASN.1</p> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
88 |
|
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
89 |
<p>Rules:</p> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
90 |
|
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
91 |
<ul> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
92 |
<li>a stream contains zero or more relations</li> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
93 |
<li>a relation has a name</li> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
94 |
<li>a relation has one or more attributes</li> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
95 |
<li>a relation contains zero or more records</li> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
96 |
</ul> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
97 |
|
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
98 |
|
143 | 99 |
<h2>What <m:name/> are?</h2> |
100 |
||
101 |
<p> |
|
102 |
<m:name/> are an open <em>data format</em> designed for streaming structured data between two processes. |
|
103 |
Simultaneously with the format specification, we are also developing a <em>reference implementation</em> (libraries and tools) as a free software. |
|
104 |
Although we believe in the specification-first (or contract-first) approach, we always look and check, whether the theoretic concepts are feasible and whether they can be reasonably and reliably implemented. |
|
105 |
So befeore publishing any new specification or its version, we will verify it by creating a reference implementation at least in one programming language. |
|
106 |
</p> |
|
107 |
<p> |
|
108 |
More generally, <m:name/> are a philosophical continuation of the classic <m:unix/> pipelines and the relational model. |
|
109 |
</p> |
|
110 |
||
111 |
||
112 |
<h2>What <m:name/> are not?</h2> |
|
113 |
||
114 |
<p> |
|
115 |
<m:name/> respect the existing ecosystem and are rather an improvement or supplement than a replacement. |
|
116 |
So <m:name/> are not a: |
|
117 |
</p> |
|
118 |
||
119 |
<ul> |
|
120 |
<li>Shell – we use existing shells (e.g. GNU Bash), work with any shell and even without a shell (e.g. as a stream format passed through a network or stored in a file).</li> |
|
144
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
121 |
<li>Terminal emulator – same as with shells, we use existing terminals and we can use <m:name/> also outside any terminal; if we interact with the terminal, we use standard means as Unicode, ANSI escape sequences etc.</li> |
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
122 |
<li>IDE – we can use standard <m:unix/> tools as an IDE (GNU Screen, Make etc.) or any other IDE.</li> |
143 | 123 |
<li>Programming language – <m:name/> are language-independent data format and can be produced or consumed in any programming language.</li> |
124 |
<li>Query language – although some of our tools are doing queries, filtering or transformations, we are not inventing a new query language – instead, we use existing languages like SQL, XPath or regular expressions.</li> |
|
125 |
<!--<li>Text editor – </li>--> |
|
144
ee7e96151673
classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents:
143
diff
changeset
|
126 |
<li>Database system, DBMS – we focus on the stream processing rather than data storage. Although sometimes it makes sense to redirect data to a file and continue with the processing later.</li> |
143 | 127 |
</ul> |
128 |
||
129 |
||
130 |
<h2>Project status</h2> |
|
131 |
||
132 |
<p> |
|
133 |
The main ideas and the roadmap are quite clear, but many things will change (including the format internals and interfaces of the libraries and tools). |
|
134 |
Because we know how important the API and ABI stability is, we are not ready to publish the version 1.0 yet. |
|
135 |
</p> |
|
136 |
<p> |
|
137 |
On the other hand, the already published tools (tagged as v0.x in v_0 branch) should work quite well (should compile, should run, should not segfault often, should not wipe your hard drive or kill your cat), |
|
138 |
so they might be useful for someone who likes our ideas and who is prepared to update own programs and scripts when the new version is ready. |
|
139 |
</p> |
|
138 | 140 |
|
141 |
||
87
25dec6931f18
Lepší odsazení, tabulátory.
František Kučera <franta-hg@frantovo.cz>
parents:
23
diff
changeset
|
142 |
</text> |
4
1bb39595a51c
genrování hlavní nabídky #1
František Kučera <franta-hg@frantovo.cz>
parents:
2
diff
changeset
|
143 |
|
1 | 144 |
</stránka> |
87
25dec6931f18
Lepší odsazení, tabulátory.
František Kučera <franta-hg@frantovo.cz>
parents:
23
diff
changeset
|
145 |