18 |
18 |
19 <p> |
19 <p> |
20 Each running program (process) has one input stream (called standard input or STDIN) and one output stream (called standard output or STDOUT) and also one additional output stream for logging/errors/warnings (STDERR). |
20 Each running program (process) has one input stream (called standard input or STDIN) and one output stream (called standard output or STDOUT) and also one additional output stream for logging/errors/warnings (STDERR). |
21 We can connect programs and pass the STDOUT of first one to the STDIN of the second one (etc.) using pipes. |
21 We can connect programs and pass the STDOUT of first one to the STDIN of the second one (etc.) using pipes. |
22 </p> |
22 </p> |
|
23 |
|
24 <p> |
|
25 A classic pipeline example (<m:a href="classic-example">explained</m:a>): |
|
26 </p> |
|
27 |
|
28 <m:classic-example/> |
23 |
29 |
24 <!-- |
30 <!-- |
25 <pre>cat /etc/fstab | dd 2>/tmp/dd.log | grep tmpfs</pre> |
|
26 <p></p> |
|
27 <m:diagram orientace="vodorovně"> |
31 <m:diagram orientace="vodorovně"> |
28 node[shape=box]; |
32 node[shape=box]; |
29 |
33 |
30 cat [label="cat /etc/fstab"]; |
34 cat [label="cat /etc/fstab"]; |
31 dd []; |
35 dd []; |
40 |
44 |
41 <p> |
45 <p> |
42 According to this principle we can build complex and powerful programs (pipelines) by composing several simple, single-purpose and reusable programs. |
46 According to this principle we can build complex and powerful programs (pipelines) by composing several simple, single-purpose and reusable programs. |
43 Such single-purpose programs (often called <em>filters</em>) are much easier to create, test and optimize and their authors don't have to bother about the complexity of the final pipeline. |
47 Such single-purpose programs (often called <em>filters</em>) are much easier to create, test and optimize and their authors don't have to bother about the complexity of the final pipeline. |
44 They even don't have to know, how their programs will be used in the future by others. |
48 They even don't have to know, how their programs will be used in the future by others. |
45 This is a great design principle that brings us advanced flexibility, reusability, efficiency and reliability. Simply: awesome. |
49 This is a great design principle that brings us advanced flexibility, reusability, efficiency and reliability. |
46 Being in any role (author of a filter, builder of a pipeline etc.), we can always focus on our task only and do it well. |
50 Being in any role (author of a filter, builder of a pipeline etc.), we can always focus on our task only and do it well. |
47 And we can collaborate with others even if we don't know about them and we don't know that we are collaborating. |
51 And we can collaborate with others even if we don't know about them and we don't know that we are collaborating. |
48 Now think about putting this together with the free software ideas... How very! |
52 Now think about putting this together with the free software ideas... How very! |
49 </p> |
53 </p> |
50 |
54 |
77 // cli -> xml [ltail=cluster_in, lhead=cluster_out]; |
81 // cli -> xml [ltail=cluster_in, lhead=cluster_out]; |
78 |
82 |
79 </m:diagram> |
83 </m:diagram> |
80 --> |
84 --> |
81 |
85 |
|
86 |
|
87 <p>Bytes, text, structured data? XML, YAML, JSON, ASN.1</p> |
|
88 |
|
89 <p>Rules:</p> |
|
90 |
|
91 <ul> |
|
92 <li>a stream contains zero or more relations</li> |
|
93 <li>a relation has a name</li> |
|
94 <li>a relation has one or more attributes</li> |
|
95 <li>a relation contains zero or more records</li> |
|
96 </ul> |
|
97 |
|
98 |
82 <h2>What <m:name/> are?</h2> |
99 <h2>What <m:name/> are?</h2> |
83 |
100 |
84 <p> |
101 <p> |
85 <m:name/> are an open <em>data format</em> designed for streaming structured data between two processes. |
102 <m:name/> are an open <em>data format</em> designed for streaming structured data between two processes. |
86 Simultaneously with the format specification, we are also developing a <em>reference implementation</em> (libraries and tools) as a free software. |
103 Simultaneously with the format specification, we are also developing a <em>reference implementation</em> (libraries and tools) as a free software. |
99 So <m:name/> are not a: |
116 So <m:name/> are not a: |
100 </p> |
117 </p> |
101 |
118 |
102 <ul> |
119 <ul> |
103 <li>Shell – we use existing shells (e.g. GNU Bash), work with any shell and even without a shell (e.g. as a stream format passed through a network or stored in a file).</li> |
120 <li>Shell – we use existing shells (e.g. GNU Bash), work with any shell and even without a shell (e.g. as a stream format passed through a network or stored in a file).</li> |
104 <li>Terminal emulator – same as with shells, we use existing terminals and we can use <m:name/> also outside any terminal; if we interact with any terminal, we use standard means as Unicode, ANSI escape sequences etc.</li> |
121 <li>Terminal emulator – same as with shells, we use existing terminals and we can use <m:name/> also outside any terminal; if we interact with the terminal, we use standard means as Unicode, ANSI escape sequences etc.</li> |
105 <li>IDE – we use standard <m:unix/> tools as an IDE (GNU Screen, Make etc.) or any other IDE.</li> |
122 <li>IDE – we can use standard <m:unix/> tools as an IDE (GNU Screen, Make etc.) or any other IDE.</li> |
106 <li>Programming language – <m:name/> are language-independent data format and can be produced or consumed in any programming language.</li> |
123 <li>Programming language – <m:name/> are language-independent data format and can be produced or consumed in any programming language.</li> |
107 <li>Query language – although some of our tools are doing queries, filtering or transformations, we are not inventing a new query language – instead, we use existing languages like SQL, XPath or regular expressions.</li> |
124 <li>Query language – although some of our tools are doing queries, filtering or transformations, we are not inventing a new query language – instead, we use existing languages like SQL, XPath or regular expressions.</li> |
108 <!--<li>Text editor – </li>--> |
125 <!--<li>Text editor – </li>--> |
109 <li>Database system, DBMS – we focus on the stream processing rather than data storage. Although sometimes it makes sense to pipe data to a file and continue with the processing later.</li> |
126 <li>Database system, DBMS – we focus on the stream processing rather than data storage. Although sometimes it makes sense to redirect data to a file and continue with the processing later.</li> |
110 </ul> |
127 </ul> |
111 |
128 |
112 |
129 |
113 <h2>Project status</h2> |
130 <h2>Project status</h2> |
114 |
131 |