# HG changeset patch # User František Kučera # Date 1601037504 -7200 # Node ID d7ae02390fac36bae625c8e14f57a49ce729812f # Parent d4c2968a391f2640ccdd79ca30e3bf8814c80d6f relpipe-tr-guile.cpp → relpipe-tr-scheme.cpp diff -r d4c2968a391f -r d7ae02390fac relpipe-data/download.xml --- a/relpipe-data/download.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/download.xml Fri Sep 25 14:38:24 2020 +0200 @@ -39,9 +39,9 @@ hg clone https://hg.globalcode.info/relpipe/relpipe-tr-awk.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-tr-cut.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-tr-grep.cpp; -hg clone https://hg.globalcode.info/relpipe/relpipe-tr-guile.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-tr-python.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-tr-sed.cpp; +hg clone https://hg.globalcode.info/relpipe/relpipe-tr-scheme.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-tr-sql.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-tr-validator.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-web;]]> diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples-guile-aggregations.xml --- a/relpipe-data/examples-guile-aggregations.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples-guile-aggregations.xml Fri Sep 25 14:38:24 2020 +0200 @@ -2,15 +2,15 @@ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> - Aggregating data with Guile + Aggregating data with Scheme counting records and computing sum 01700

- In relpipe-tr-guile we can generate new records – not only modify records from the input. - There is --has-more-records option which – if evaluated as true – says: „read one more record from the Guile context and call me again“. + In relpipe-tr-scheme we can generate new records – not only modify records from the input. + There is --has-more-records option which – if evaluated as true – says: „read one more record from the Scheme context and call me again“. We can also suppress all original records by --where '#f'. And we can also change the structure of the relation (see previous examples). Thus we can iterate through a relation but completely replace its structure and content. @@ -49,7 +49,7 @@

- One day, there might also be a translator that parses SQL code and generates Guile code, + One day, there might also be a translator that parses SQL code and generates Scheme code, so we could have advantages of both worlds a) concise and declarative syntax of SQL and b) streaming – which means no need for putting all the data in the RAM or on the disk. diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples-guile-filtering.xml --- a/relpipe-data/examples-guile-filtering.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples-guile-filtering.xml Fri Sep 25 14:38:24 2020 +0200 @@ -2,7 +2,7 @@ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> - Complex filtering with Guile + Complex filtering with Scheme filtering records with AND, OR and functions 01400 @@ -12,12 +12,12 @@ For simple filtering, we can use relpipe-tr-grep. But what if we need to write some complex query that contains AND and OR operators? What if we need e.g. compare numbers – not only match texts against regular expressions? - There is a tool capable to do this and much more: relpipe-tr-guile! + There is a tool capable to do this and much more: relpipe-tr-scheme!

Guile is the GNU implementation of Scheme language (something like Lisp and also full of parenthesis). - The relpipe-tr-guile uses GNU Guile as a library, puts data in the Guile context and evaluates Guile expressions and then reads data from the Guile context back and generates relational output from them. + The relpipe-tr-scheme reference implementation uses GNU Guile as a library, puts data in the Scheme context and evaluates Scheme expressions and then reads data from the Scheme context back and generates relational output from them. Good news are that it is not necessary to know Lisp/Scheme to use this tool. For the first steps, it can be used just as a query language – like SQL, just a bit Polish.

@@ -29,7 +29,7 @@

Well, well… here we are:

@@ -54,7 +54,7 @@ ╰───────────────────────────────────────────────────────────────────────┴───────────────┴────────────────┴────────────────┴────────────────╯ Record count: 13]]> -

The --relation 'files.*' is a regular expression that says which relations should be processed in Guile – others are passed through unchanged.

+

The --relation 'files.*' is a regular expression that says which relations should be processed in Scheme – others are passed through unchanged.

The --where '(= $size 666)' is our condition. @@ -65,11 +65,11 @@

- Relational attributes are mapped to Guile variables with same name, just prefixed with $. + Relational attributes are mapped to Scheme variables with same name, just prefixed with $. (we considered symbol, but $ seems to be still more common on keyboards in 2019) - While relational attribute name is an arbitrary string, Guile variable names have some limitations, thus not all attributes can be mapped – those with spaces and some special characters are currently unsupported (this will be fixed in later versions by some kind of encoding/escaping). + While relational attribute name is an arbitrary string, Scheme variable names have some limitations, thus not all attributes can be mapped – those with spaces and some special characters are currently unsupported (this will be fixed in later versions by some kind of encoding/escaping).

@@ -88,7 +88,7 @@

The Btrfs filesystems in our fstab:

@@ -106,7 +106,7 @@

So we can find filesystems mounted somewhere under /mnt:

@@ -130,7 +130,7 @@

Like in SQL, we can join multiple conditions together with logical operators AND and OR. - In Guile/Scheme these operators are also functions – they are written in the same (fashion). + In Scheme these operators are also functions – they are written in the same (fashion).

@@ -139,7 +139,7 @@

Files with sizes 666 bytes or 1984 bytes:

@@ -172,7 +172,7 @@ diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples-guile-multiple-relations.xml --- a/relpipe-data/examples-guile-multiple-relations.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples-guile-multiple-relations.xml Fri Sep 25 14:38:24 2020 +0200 @@ -2,14 +2,14 @@ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> - Processing multiple relations with Guile + Processing multiple relations with Scheme filter some relations and others keep unaffected 01800

- The relpipe-tr-guile is capable to process multiple relations in a single pass. + The relpipe-tr-scheme is capable to process multiple relations in a single pass. So we can filter some relations and let others flow unaffected through this pipeline step.

@@ -21,9 +21,9 @@ # put them together in a single stream function: sample-data() { r1; r2; r3; } -# let them flow through our Guile transformation: +# let them flow through our Scheme transformation: sample-data \ - | relpipe-tr-guile \ + | relpipe-tr-scheme \ --relation fstab \ --where '(or (string= $type "btrfs") (string-prefix? "/mnt/" $mount_point) )' \ --relation filesystem \ @@ -74,7 +74,7 @@

- In Guile, we have filtered the fstab and filesystem relations + In Scheme, we have filtered the fstab and filesystem relations while the seq relation was kept intact.

@@ -85,7 +85,7 @@

- If we define some variables in the Guile context, they will stay there – so we can pass data across relations. + If we define some variables in the Scheme context, they will stay there – so we can pass data across relations. Thus we can do even JOIN, if we really want.

diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples-guile-parametrized-queries.xml --- a/relpipe-data/examples-guile-parametrized-queries.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples-guile-parametrized-queries.xml Fri Sep 25 14:38:24 2020 +0200 @@ -2,7 +2,7 @@ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> - Parametrized queries with Guile + Parametrized queries with Scheme passing input parameters and avoiding code-injections 01600 @@ -20,7 +20,7 @@

- But it is fundamentally wrong. The input parameter is blindly pasted in middle of the Guile code. + But it is fundamentally wrong. The input parameter is blindly pasted in middle of the Scheme code. So if we call e.g. fstab-where-type 'ext4"', it crashes terribly. Do you remember SQL injections in your first PHP scripts when you were 14? Do not do it again!

- The relpipe-tr-guile tool has a safe way for passing parameters from the outside. And such parameters are even strongly typed. + The relpipe-tr-scheme tool has a safe way for passing parameters from the outside. And such parameters are even strongly typed. So this is, how our program should be written:

- Doing projections with Guile + Doing projections with Scheme modifying attribute values and adding new attributes or removing them 01500

- The relpipe-tr-guile can not only filter records, + The relpipe-tr-scheme can not only filter records, but can also modify them and even modify the structure of the relation – add or remove attributes.

@@ -55,11 +55,11 @@

Modifying attribute values

- Then, we can modify such relation using Guile – e.g. we can make the name uppercase and increase id by 1000: + Then, we can modify such relation using Scheme – e.g. we can make the name uppercase and increase id by 1000:

@@ -85,7 +85,7 @@

@@ -68,7 +68,7 @@

Actually, the relpipe-in-sql is just a symbolic link to the relpipe-tr-sql and have the same capabilities (with just bit different default behavior to match general logic of the input filters). - So if we do not need special feautres of Guile, AWK or other transformation tool, it is better to stay with SQL (when we already use the SQL module). + So if we do not need special feautres of Scheme, AWK or other transformation tool, it is better to stay with SQL (when we already use the SQL module).

diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples-parallel-hashes.xml --- a/relpipe-data/examples-parallel-hashes.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples-parallel-hashes.xml Fri Sep 25 14:38:24 2020 +0200 @@ -56,7 +56,7 @@

  • aggregate – uses SQL to order the records and SQL window function to show, how many files have the same content; - in this step we could use also relpipe-tr-awk or relpipe-tr-guile if we prefer AWK or Guile/Scheme to SQL + in this step we could use also relpipe-tr-awk or relpipe-tr-scheme if we prefer AWK or Scheme to SQL
  • relpipe-out-tabular diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples-runnable-jars.xml --- a/relpipe-data/examples-runnable-jars.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples-runnable-jars.xml Fri Sep 25 14:38:24 2020 +0200 @@ -43,7 +43,7 @@
  • filterRunable – uses AWK to skip the records (files) that does not have a main class; - in this step we could use also relpipe-tr-sql or relpipe-tr-guile if we prefer SQL or Guile/Scheme to AWK + in this step we could use also relpipe-tr-sql or relpipe-tr-scheme if we prefer SQL or Scheme to AWK
  • shortenPath diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples/awk-through-xml.sh --- a/relpipe-data/examples/awk-through-xml.sh Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples/awk-through-xml.sh Fri Sep 25 14:38:24 2020 +0200 @@ -41,8 +41,8 @@ # "SELECT * FROM policy WHERE policy = 'allow' AND user <> 'root'" # --copy 'limit' - # Or using Guile (Scheme): - # relpipe-tr-guile \ + # Or using Scheme: + # relpipe-tr-scheme \ # --relation policy \ # --where '(and (string= $policy "allow") (not (string= $user "root")) )' } diff -r d4c2968a391f -r d7ae02390fac relpipe-data/examples/guile-file-count-size-sum.sh --- a/relpipe-data/examples/guile-file-count-size-sum.sh Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/examples/guile-file-count-size-sum.sh Fri Sep 25 14:38:24 2020 +0200 @@ -7,7 +7,7 @@ | relpipe-in-filesystem \ --file path \ --file size \ - | relpipe-tr-guile \ + | relpipe-tr-scheme \ --relation 'f.*' \ --output-attribute 'count' integer \ --output-attribute 'sum' integer \ diff -r d4c2968a391f -r d7ae02390fac relpipe-data/implementation.xml --- a/relpipe-data/implementation.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/implementation.xml Fri Sep 25 14:38:24 2020 +0200 @@ -38,7 +38,7 @@ relpipe-tr-awk.cpp executable transformation c++ GNU GPLv3 relpipe-tr-cut.cpp executable transformation c++ GNU GPLv3 relpipe-tr-grep.cpp executable transformation c++ GNU GPLv3 - relpipe-tr-guile.cpp executable transformation c++ GNU GPLv3 + relpipe-tr-scheme.cpp executable transformation c++ GNU GPLv3 relpipe-tr-python.cpp executable transformation c++ GNU GPLv3 relpipe-tr-sed.cpp executable transformation c++ GNU GPLv3 relpipe-tr-sql.cpp executable transformation c++ GNU GPLv3 diff -r d4c2968a391f -r d7ae02390fac relpipe-data/index.xml --- a/relpipe-data/index.xml Sat Sep 12 13:20:21 2020 +0200 +++ b/relpipe-data/index.xml Fri Sep 25 14:38:24 2020 +0200 @@ -148,7 +148,7 @@
  • Terminal emulator – same as with shells, we use existing terminals and we can use also outside any terminal; if we interact with the terminal, we use standard means like Unicode, ANSI escape sequences etc.
  • IDE – we can use standard tools as an IDE (GNU Screen, Emacs, Make etc.) or any other IDE.
  • Programming language – are language-independent data format and can be produced or consumed in any programming language.
  • -
  • Query language – although some of our tools are doing queries, filtering or transformations, we are not inventing a new query language – instead, we use existing languages like SQL, XPath, Guile/Scheme, AWK or regular expressions.
  • +
  • Query language – although some of our tools are doing queries, filtering or transformations, we are not inventing a new query language – instead, we use existing languages like SQL, XPath, Scheme, AWK or regular expressions.
  • Database system, DBMS – we focus on the stream processing rather than data storage. Although sometimes it makes sense to redirect data to a file and continue with the processing later.