<stránka

	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"

	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">

	<nadpis>Classic pipeline example</nadpis>

	<perex>Explained example of classic pipeline</perex>

	<text xmlns="http://www.w3.org/1999/xhtml">

		<p>

			Assume that we have a text file containing a list of animals and their properties:

		</p>

		<m:pre jazyk="text" src="animals.txt"/>

		<p>

			We can pass this file through a pipeline:

		</p>

		<m:classic-example/>

		<p>

			Particular steps of the pipeline are separated by the | pipe symbol.

			In the first step, we just read the file and print it on STDOUT.<m:podČarou>Of course, this is an <a href="http://porkmail.org/era/unix/award.html" title="Useless Use of Cat">UUoC</a>, but in examples the right order makes it easier to read than usage of &lt; file redirections.</m:podČarou>

			In the second step, we filter only dogs and get:

		</p>

		<pre><![CDATA[big yellow dog

small white dog]]></pre>

		<p>

			In the third step, we select second <em>field</em> (fields are separated by spaces) and get colours of our dogs:

		</p>

		<pre><![CDATA[yellow

white]]></pre>

		<p>

			In the fourth step, we translate the values to uppercase and get:

		</p>

		<pre><![CDATA[YELLOW

WHITE]]></pre>

		<p>

			So we have a list of colors of our dogs printed in big letters. 

			In case we have several dogs of same color, we could avoid duplicates simply by adding <code>| sort -u</code> in the pipeline (after the <code>cut</code> step).

		</p>

		<h2>The great parts</h2>

		<p>

			They can focus on their tasks which are reading files, filtering by regular expressions, doing some substrings and text conversions. And they do it well without being distracted by any animals.

		</p>

		<p>

			We just simply build a pipeline in a shell (e.g. GNU Bash) from existing programs and focus on our business logic.

			And we do it well without being distracted by any low-level issues.

		</p>

		<p>

			Each program used in the pipeline can be written in different programming language and they will work together.

			Tools written in C, C++, Java, Lisp, Perl, Python, Rust or any other language can be combined together.

			Thus optimal language can be used for each task.

		</p>

		<p>

			The pipeline is reusable which is a big advantage compared to ad-hoc operations done in CLI or GUI tools.

			Thus we can feed different data (with same structure of course) into the pipeline and get desired result.

		</p>

		<p>

			Particular steps in the pipeline can be added, removed or exchanged. 

			And we can also debug the pipeline and check what each step produces (e.g. use <code>tee</code> to duplicate the intermediary outputs to a file or just execute only some first steps of the pipeline).

		</p>

		<h2>The pitfalls</h2>

		<p>

			This simple example looks quite flawless.

			But actually it is very brittle.

		</p>

		<p>

			What if we have a very big cat that can be described by this line in our file?

		</p>

		<pre>dog-sized red cat</pre>

		<p>In the second step of the pipeline (<code>grep</code>) we will include this record and the final result will be:</p>

		<pre><![CDATA[RED

YELLOW

WHITE]]></pre>

		<p>

			This problem is caused by the fact that the <code>grep dog</code> filters lines containing the word <em>dog</em> regardless its position (first, second or third field).

		</p>

		<p>

			What if we have a turtle that has lighter color than other turtles?

		</p>

		<pre>small light green turtle</pre>

		<p>

			If we do <code>grep turtle</code> it will work well in this case, but our pipeline will fail in the third step where the <code>cut</code> will select only <em>light</em> (instead of <em>light green</em>).

			And the final result will be:

		</p>

		<pre><![CDATA[GREEN

LIGHT]]></pre>

		<p>

			Which is definitively wrong because the second turtle is not LIGHT, it is LIGHT GREEN.

			Sometimes we could avoid such problems by restrictions/presumptions e.g. <em>the color must not contain a space character</em> (we could replace spaces by hyphens).

			Or we could use some other field delimiter e.g. ; or | or ,. But still we would not be able to use such character in the field values.

			or add some quotes/apostrophes (which still requires escaping, because what if we have e.g. name field containing an apostrophe?).

			And parsing such inputs by classic tools and regular expressions is not easy and sometimes even not possible.

		</p>

		<p>

			There are also other problems like character encoding, missing meta-data (e.g. field names and types), joining multiple files (Is there always a new-line character at the end of the file? Or is there some nasty BOM at the beginning of the file?)

			or passing several types of data in a single stream (we have list of animals and we can have e.g. also a list of foods or list of our staff where each list has different fields).

		</p>

	</text>

</stránka>