relpipe-data/index.xml
author František Kučera <franta-hg@frantovo.cz>
Sun, 25 Nov 2018 19:58:06 +0100
branchv_0
changeset 144 ee7e96151673
parent 143 297da74fcab2
child 145 42bbbccd87f3
permissions -rw-r--r--
classic pipeline example
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
23
0d2729ed16ed zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents: 18
diff changeset
     1
<stránka
0d2729ed16ed zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents: 18
diff changeset
     2
	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
0d2729ed16ed zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents: 18
diff changeset
     3
	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
0d2729ed16ed zkouška interního odkazu
František Kučera <franta-hg@frantovo.cz>
parents: 18
diff changeset
     4
	
137
ecfe2980ae1f logo and css
František Kučera <franta-hg@frantovo.cz>
parents: 136
diff changeset
     5
	<nadpis>Relational pipes</nadpis>
ecfe2980ae1f logo and css
František Kučera <franta-hg@frantovo.cz>
parents: 136
diff changeset
     6
	<perex>Official homepage of Relational pipes.</perex>
4
1bb39595a51c genrování hlavní nabídky #1
František Kučera <franta-hg@frantovo.cz>
parents: 2
diff changeset
     7
	<pořadí>10</pořadí>
1bb39595a51c genrování hlavní nabídky #1
František Kučera <franta-hg@frantovo.cz>
parents: 2
diff changeset
     8
2
ab9099ff88fa vkládání zápatí, jmenné prostory, saxon
František Kučera <franta-hg@frantovo.cz>
parents: 1
diff changeset
     9
	<text xmlns="http://www.w3.org/1999/xhtml">
ab9099ff88fa vkládání zápatí, jmenné prostory, saxon
František Kučera <franta-hg@frantovo.cz>
parents: 1
diff changeset
    10
		<p>
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    11
			One of the great parts of the <m:unix/>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    12
			<m:podČarou> 
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    13
				<m:unix tvar="vysvětlivka"/> 
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    14
			</m:podČarou> culture is the invention<m:podČarou>which is attributed to Doug McIlroy, see <a href="http://www.catb.org/~esr/writings/taoup/html/ch07s02.html#plumbing">The Art of Unix Programming: Pipes, Redirection, and Filters</a></m:podČarou>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    15
			of <em>pipes</em> and the idea<m:podČarou>see <a href="http://www.catb.org/~esr/writings/taoup/html/ch01s06.html">The Art of Unix Programming: Basics of the Unix Philosophy</a></m:podČarou> 
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    16
			that <em>one program should do one thing and do it well</em>.
87
25dec6931f18 Lepší odsazení, tabulátory.
František Kučera <franta-hg@frantovo.cz>
parents: 23
diff changeset
    17
		</p>
138
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    18
		
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    19
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    20
			Each running program (process) has one input stream (called standard input or STDIN) and one output stream (called standard output or STDOUT) and also one additional output stream for logging/errors/warnings (STDERR).
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    21
			We can connect programs and pass the STDOUT of first one to the STDIN of the second one (etc.) using pipes.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    22
		</p>
144
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    23
		
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    24
		<p>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    25
			A classic pipeline example (<m:a href="classic-example">explained</m:a>):
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    26
		</p>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    27
		
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    28
		<m:classic-example/>
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    29
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    30
		<!--		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    31
		<m:diagram orientace="vodorovně">
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    32
			node[shape=box];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    33
			
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    34
			cat  [label="cat /etc/fstab"];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    35
			dd   [];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    36
			grep [label="grep tmpfs"];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    37
			log  [label="/tmp/dd.log"];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    38
			
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    39
			cat -> dd  [label="STDOUT → STDIN"];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    40
			dd -> grep [label="STDOUT → STDIN"];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    41
			dd -> log  [label="STDERR → file"];
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    42
		</m:diagram>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    43
		-->
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    44
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    45
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    46
			According to this principle we can build complex and powerful programs (pipelines) by composing several simple, single-purpose and reusable programs.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    47
			Such single-purpose programs (often called <em>filters</em>) are much easier to create, test and optimize and their authors don't have to bother about the complexity of the final pipeline.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    48
			They even don't have to know, how their programs will be used in the future by others.
144
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    49
			This is a great design principle that brings us advanced flexibility, reusability, efficiency and reliability.
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    50
			Being in any role (author of a filter, builder of a pipeline etc.), we can always focus on our task only and do it well.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    51
			And we can collaborate with others even if we don't know about them and we don't know that we are collaborating.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    52
			Now think about putting this together with the free software ideas...  How very!
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    53
		</p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    54
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    55
		<!--
138
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    56
		<m:diagram orientace="vodorovně">
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    57
			compound=true;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    58
			node[shape=box];
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    59
			
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    60
			subgraph cluster_in {
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    61
			label = "Inputs:";
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    62
			cli;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    63
			fstab;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    64
			}
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    65
			
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    66
			subgraph cluster_tr {
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    67
			label = "Transformations:";
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    68
			grep;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    69
			sed;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    70
			}
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    71
			
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    72
			subgraph cluster_out {
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    73
			label = "Outputs:";
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    74
			xml;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    75
			tabular;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    76
			gui;
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    77
			}
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    78
			
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    79
			cli -> grep  [ltail=cluster_in, lhead=cluster_tr];
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    80
			grep -> xml [ltail=cluster_tr, lhead=cluster_out];
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    81
			// cli -> xml [ltail=cluster_in, lhead=cluster_out];
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    82
			
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
    83
		</m:diagram>
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    84
		-->
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    85
		
144
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    86
		
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    87
		<p>Bytes, text, structured data? XML, YAML, JSON, ASN.1</p>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    88
		
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    89
		<p>Rules:</p>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    90
		
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    91
		<ul>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    92
			<li>a stream contains zero or more relations</li>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    93
			<li>a relation has a name</li>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    94
			<li>a relation has one or more attributes</li>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    95
			<li>a relation contains zero or more records</li>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    96
		</ul>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    97
		
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
    98
		
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
    99
		<h2>What <m:name/> are?</h2>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   100
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   101
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   102
			<m:name/> are an open <em>data format</em> designed for streaming structured data between two processes. 
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   103
			Simultaneously with the format specification, we are also developing a <em>reference implementation</em> (libraries and tools) as a free software.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   104
			Although we believe in the specification-first (or contract-first) approach, we always look and check, whether the theoretic concepts are feasible and whether they can be reasonably and reliably implemented.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   105
			So befeore publishing any new specification or its version, we will verify it by creating a reference implementation at least in one programming language.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   106
		</p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   107
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   108
			More generally, <m:name/> are a philosophical continuation of the classic <m:unix/> pipelines and the relational model.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   109
		</p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   110
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   111
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   112
		<h2>What <m:name/> are not?</h2>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   113
			
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   114
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   115
			<m:name/> respect the existing ecosystem and are rather an improvement or supplement than a replacement.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   116
			So <m:name/> are not a:
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   117
		</p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   118
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   119
		<ul>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   120
			<li>Shell – we use existing shells (e.g. GNU Bash), work with any shell and even without a shell (e.g. as a stream format passed through a network or stored in a file).</li>
144
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
   121
			<li>Terminal emulator – same as with shells, we use existing terminals and we can use <m:name/> also outside any terminal; if we interact with the terminal, we use standard means as Unicode, ANSI escape sequences etc.</li>
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
   122
			<li>IDE – we can use standard <m:unix/> tools as an IDE (GNU Screen, Make etc.) or any other IDE.</li>
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   123
			<li>Programming language – <m:name/> are language-independent data format and can be produced or consumed in any programming language.</li>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   124
			<li>Query language – although some of our tools are doing queries, filtering or transformations, we are not inventing a new query language – instead, we use existing languages like SQL, XPath or regular expressions.</li>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   125
			<!--<li>Text editor – </li>-->
144
ee7e96151673 classic pipeline example
František Kučera <franta-hg@frantovo.cz>
parents: 143
diff changeset
   126
			<li>Database system, DBMS – we focus on the stream processing rather than data storage. Although sometimes it makes sense to redirect data to a file and continue with the processing later.</li>
143
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   127
		</ul>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   128
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   129
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   130
		<h2>Project status</h2>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   131
		
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   132
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   133
			The main ideas and the roadmap are quite clear, but many things will change (including the format internals and interfaces of the libraries and tools).
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   134
			Because we know how important the API and ABI stability is, we are not ready to publish the version 1.0 yet.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   135
		</p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   136
		<p>
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   137
			On the other hand, the already published tools (tagged as v0.x in v_0 branch) should work quite well (should compile, should run, should not segfault often, should not wipe your hard drive or kill your cat),
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   138
			so they might be useful for someone who likes our ideas and who is prepared to update own programs and scripts when the new version is ready.
297da74fcab2 introduction
František Kučera <franta-hg@frantovo.cz>
parents: 138
diff changeset
   139
		</p>
138
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
   140
5d892b169e32 metadata, footer
František Kučera <franta-hg@frantovo.cz>
parents: 137
diff changeset
   141
		
87
25dec6931f18 Lepší odsazení, tabulátory.
František Kučera <franta-hg@frantovo.cz>
parents: 23
diff changeset
   142
	</text>
4
1bb39595a51c genrování hlavní nabídky #1
František Kučera <franta-hg@frantovo.cz>
parents: 2
diff changeset
   143
1
a05c6f3cbc3e základ, první verze
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
   144
</stránka>
87
25dec6931f18 Lepší odsazení, tabulátory.
František Kučera <franta-hg@frantovo.cz>
parents: 23
diff changeset
   145