Mercurial Mercurial > relpipe > relpipe-web / changeset
summary | shortlog | changelog | graph | tags | bookmarks | branches | files | changeset | raw | help
Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Release v0.11 v_0 relpipe-v0.11
authorFrantišek Kučera <franta-hg@frantovo.cz>
Tue, 09 Apr 2019 17:14:05 +0200
branchv_0
changeset 256 822ffd23d679
parent 255 94b533007e77
child 257 a39066264509
Release v0.11
relpipe-data/download.xml file | annotate | diff | comparison | revisions
relpipe-data/examples-recfile.xml file | annotate | diff | comparison | revisions
relpipe-data/examples/release-v0.11.sh file | annotate | diff | comparison | revisions
relpipe-data/implementation.xml file | annotate | diff | comparison | revisions
relpipe-data/release-v0.10.xml file | annotate | diff | comparison | revisions
relpipe-data/release-v0.11.xml file | annotate | diff | comparison | revisions
relpipe-data/roadmap.xml file | annotate | diff | comparison | revisions 
--- a/relpipe-data/download.xml	Mon Apr 08 13:37:35 2019 +0200
+++ b/relpipe-data/download.xml	Tue Apr 09 17:14:05 2019 +0200
@@ -18,6 +18,7 @@
 hg clone https://hg.globalcode.info/relpipe/relpipe-in-csv.cpp;
 hg clone https://hg.globalcode.info/relpipe/relpipe-in-filesystem.cpp;
 hg clone https://hg.globalcode.info/relpipe/relpipe-in-fstab.cpp;
+hg clone https://hg.globalcode.info/relpipe/relpipe-in-recfile.cpp;
 hg clone https://hg.globalcode.info/relpipe/relpipe-in-xml.cpp;
 hg clone https://hg.globalcode.info/relpipe/relpipe-lib-cli.cpp;
 hg clone https://hg.globalcode.info/relpipe/relpipe-lib-protocol.cpp;
@@ -56,6 +57,7 @@
 			<li>2018-12-20: <m:a href="release-v0.8">v0.8</m:a></li>
 			<li>2019-01-18: <m:a href="release-v0.9">v0.9</m:a></li>
 			<li>2019-02-20: <m:a href="release-v0.10">v0.10</m:a></li>
+			<li>2019-04-08: <m:a href="release-v0.11">v0.11</m:a></li>
 		</ul>
 		
 	</text>
--- a/relpipe-data/examples-recfile.xml	Mon Apr 08 13:37:35 2019 +0200
+++ b/relpipe-data/examples-recfile.xml	Tue Apr 09 17:14:05 2019 +0200
@@ -11,7 +11,7 @@
 		<p>
 			Recfile is the native format of <a href="https://www.gnu.org/software/recutils/">GNU Recutils</a>.
 			Recfiles are text files that contain records of various types.
-			They are human-editable and serve as a simple databases.
+			They are human-editable and serve as simple databases.
 			<m:name/> support input and output in this format since v0.11.
 		</p>
 		
@@ -77,12 +77,24 @@
 		</p>
 		
 		<p>
+			Because some web browsers or tools can store the original URL in extended attributes while downloading a file,
+			we can use <code>recsel</code> to find files downloaded from some particular domain:
+		</p>
+		<m:pre jazyk="bash"><![CDATA[find -print0 | relpipe-in-filesystem \
+	--file path \
+	--file size \
+	--file type \
+	--xattr xdg.origin.url --as url \
+	| relpipe-out-recfile \
+	| recsel -e 'url ~ "^https?://([^/]*\.)?archive\.org/"']]></m:pre>
+		
+		<p>
 			<m:name/> can be also used together with <a href="https://sql-dk.globalcode.info/">SQL-DK</a> (in 2019-03-05 development version)
 			to pipe data from big relational databases like PostgreSQL or MariaDB to other formats like recfiles.
 			Having a script:
 		</p>
 				
-		<m:pre jazyk="text" src="examples/sql-dk_pg_1.sh" odkaz="ano"/>
+		<m:pre jazyk="bash" src="examples/sql-dk_pg_1.sh" odkaz="ano"/>
 		
 		<p>
 			We can convert result sets from any SQL queries to relational format and then work with such data without connection to the original database.
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples/release-v0.11.sh	Tue Apr 09 17:14:05 2019 +0200
@@ -0,0 +1,43 @@
+# Install dependencies as root:
+su -c "apt install g++ make cmake mercurial pkg-config"
+su -c "apt install libxerces-c-dev" # needed only for relpipe-in-xml module
+su -c "apt install guile-2.2-dev"   # needed only for relpipe-tr-guile module; guile-2.0-dev also works but requires a patch (see below)
+
+# Run rest of installation as a non-root user:
+export RELPIPE_VERSION="v0.11"
+export RELPIPE_SRC=~/src
+export RELPIPE_BUILD=~/build
+export RELPIPE_INSTALL=~/install
+export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
+export PATH="$RELPIPE_INSTALL/bin:$PATH"
+
+rm -rf "$RELPIPE_BUILD"/relpipe-*
+mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
+
+# Helper functions:
+relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
+relpipe_install()  { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
+
+# Download all sources:
+relpipe_download lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter in-cli in-fstab in-xml in-csv in-filesystem in-recfile out-gui.qt out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-python tr-sed tr-validator tr-guile
+
+# Optional: At this point, we have all dependencies and sources downloaded, so we can disconnect this computer from the internet in order to verify that our build process is sane, deterministic and does not depend on any external resources.
+
+# Build and install libraries:
+relpipe_install lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter
+
+# Build and install tools:
+relpipe_install in-fstab in-cli in-fstab in-xml in-csv in-recfile tr-cut tr-grep tr-sed tr-guile out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile
+
+# relpipe_install in-filesystem # requires GCC 8 or patching (see below)
+
+# Clean-up:
+unset -f relpipe_install
+unset -f relpipe_download
+unset -v RELPIPE_VERSION
+unset -v RELPIPE_SRC
+unset -v RELPIPE_BUILD
+unset -v RELPIPE_INSTALL
+
+# Filter your your fstab using Guile Scheme and view it like on an 80s green screen terminal!
+relpipe-in-fstab | relpipe-tr-guile --relation 'fstab' --where '(or (= $pass 1) (string= $type "swap") )' | relpipe-out-tabular
--- a/relpipe-data/implementation.xml	Mon Apr 08 13:37:35 2019 +0200
+++ b/relpipe-data/implementation.xml	Tue Apr 09 17:14:05 2019 +0200
@@ -18,6 +18,7 @@
 			relpipe-in-csv.cpp	executable	input	c++	GNU GPLv3+
 			relpipe-in-filesystem.cpp	executable	input	c++	GNU GPLv3+
 			relpipe-in-fstab.cpp	executable	input	c++	GNU GPLv3+
+			relpipe-in-recfile.cpp	executable	input	c++	GNU GPLv3+
 			relpipe-in-xml.cpp	executable	input	c++	GNU GPLv3+
 			relpipe-lib-cli.cpp	library	header-only	c++	GNU GPLv3+
 			relpipe-lib-protocol.cpp	library	header-only	c++	GNU LGPLv3+ or GPLv2+
--- a/relpipe-data/release-v0.10.xml	Mon Apr 08 13:37:35 2019 +0200
+++ b/relpipe-data/release-v0.10.xml	Tue Apr 09 17:14:05 2019 +0200
@@ -3,7 +3,7 @@
 	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
 	
 	<nadpis>Release v0.10</nadpis>
-	<perex>second public release of Relational pipes</perex>
+	<perex>third public release of Relational pipes</perex>
 	<m:release>v0.10</m:release>
 
 	<text xmlns="http://www.w3.org/1999/xhtml">
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/release-v0.11.xml	Tue Apr 09 17:14:05 2019 +0200
@@ -0,0 +1,112 @@
+<stránka
+	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+	
+	<nadpis>Release v0.11</nadpis>
+	<perex>fourth public release of Relational pipes</perex>
+	<m:release>v0.11</m:release>
+
+	<text xmlns="http://www.w3.org/1999/xhtml">
+		<p>
+			We are pleased to introduce you the new development version of <m:name/>.
+			This release brings Recfile input and output and ASN.1 BER output:
+		</p>
+		
+		<ul>
+			<li>
+				<strong>Recfile format support</strong>:
+				Recfile is the native format of <a href="https://www.gnu.org/software/recutils/">GNU Recutils</a>.
+				Recfiles are text files that contain records of various types.
+				They are human-editable and serve as simple databases.
+				<m:name/> now contain tools for converting relational data streams to and from the recfile format.
+				This format is good for editing by hand or for storing in some version control system.
+				GNU Recutils and <m:name/> can be used together for data processing or e.g. for data-driven scripting.
+			</li>
+			
+			<li>
+				<strong>ASN.1 BER output</strong>:
+				Abstract Syntax Notation One (ASN.1) is traditional and well established language for describing interfaces and data structures in particular.
+				This format/language is widespread and actively used in telecommunications, cryptography and other industries.
+				<abbr title="Basic Encoding Rules">BER</abbr> is one of ASN.1 encodings.
+				This release of <m:name/> bring <abbr title="This support will be further improved and extended in further releases.">basic</abbr> support for output in the BER format.
+				Now it can serve as a preview (such data can be opened in dumpasn1 or wireshark) or a template (the source code can be modified to generate BER data in some particular format)
+				and help with interaction with ASN.1 <em>native speakers</em>.
+			</li>
+		</ul>
+		
+		<p>
+			See the <m:a href="examples">examples</m:a> and <m:a href="screenshots">screenshots</m:a> pages for details.
+		</p>
+		
+		<p>
+			Please note that this is still a development relasease and thus the API (libraries, CLI arguments, formats) might and will change.
+			Any suggestions, ideas and bug reports are welcome in our <m:a href="contact">mailing list</m:a>.
+		</p>
+		
+		<h3>Data types</h3>
+		<ul>
+			<li m:since="v0.8">boolean</li>
+			<li m:since="v0.8">variable unsigned integer (prototype)</li>
+			<li m:since="v0.8">string in UTF-8</li>
+		</ul>
+		<h3>Inputs</h3>
+		<ul>
+			<li m:since="v0.11">Recfile</li>
+			<li m:since="v0.9">XML</li>
+			<li m:since="v0.9">CSV</li>
+			<li m:since="v0.9">file system</li>
+			<li m:since="v0.8">CLI</li>
+			<li m:since="v0.8">fstab</li>
+		</ul>
+		<h3>Transformations</h3>
+		<ul>
+			<li m:since="v0.10">guile: filtering and transformations defined in the Scheme language using GNU Guile</li>
+			<li m:since="v0.8">grep: regular expression filter, removes unwanted records from the relation</li>
+			<li m:since="v0.8">cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation)</li>
+			<li m:since="v0.8">sed: regular expression replacer</li>
+			<li m:since="v0.8">validator: just a pass-through filter that crashes on invalid data</li>
+			<li m:since="v0.8">python: highly experimental</li>
+		</ul>
+		<h3>Outputs</h3>
+		<ul>
+			<li m:since="v0.11">ASN.1 BER</li>
+			<li m:since="v0.11">Recfile</li>
+			<li m:since="v0.9">CSV</li>
+			<li m:since="v0.8">tabular</li>
+			<li m:since="v0.8">XML</li>
+			<li m:since="v0.8">nullbyte</li>
+			<li m:since="v0.8">GUI in Qt</li>
+			<li m:since="v0.8">ODS (LibreOffice)</li>
+		</ul>
+		
+		<p>
+			Instalation was tested on Debian GNU/Linux 9.6.
+			The process should be similar on other distributions.
+		</p>
+		
+		<m:pre src="examples/release-v0.11.sh" jazyk="bash" odkaz="ano"/>
+		
+		<p>
+			<m:name/> are modular thus you can download and install only parts you need (the libraries are needed always).
+			Tools <code>out-gui.qt</code> and <code>tr-python</code> require additional libraries and are not built by default.
+		</p>
+		
+		<p>
+			The module <code>relpipe-in-filesystem</code> uses C++ filesystem API which is supported since GCC 8.
+			This module can be compiled and seems usable even with GCC 6, but requires some patching (switch to the experimental API):
+		</p>
+		
+		<m:pre jazyk="bash"><![CDATA[sed 's@#include <filesystem>@#include <experimental/filesystem>@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
+sed 's@std::filesystem@std::experimental::filesystem@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
+sed 's/.*PROPERTY CXX_STANDARD.*/#\0/g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/CMakeLists.txt]]></m:pre>
+
+		<p>
+			The module <code>relpipe-tr-guile</code> uses GNU Guile 2.2 but can also work with 2.0.
+			In such case, it requires this patch:
+		</p>
+		
+		<m:pre jazyk="bash"><![CDATA[sed 's/guile-2\.2/guile-2.0/g' -i "$RELPIPE_SRC"/relpipe-tr-guile.cpp/src/CMakeLists.txt]]></m:pre>
+
+	</text>
+
+</stránka>
--- a/relpipe-data/roadmap.xml	Mon Apr 08 13:37:35 2019 +0200
+++ b/relpipe-data/roadmap.xml	Tue Apr 09 17:14:05 2019 +0200
@@ -16,7 +16,7 @@
 			Released versions are described on the <m:a href="download">download</m:a> page.
 		</p>
 		
-		<h2>v0.11, v0.12, v0.13 etc.</h2>
+		<h2>v0.12, v0.13, v0.14 etc.</h2>
 		
 		<p>
 			Releases for discussion and verification of the format and API design.
@@ -128,7 +128,6 @@
 			<li>YAML, JSON, INI</li>
 			<li>CSV, DSV</li>
 			<li>Fsdb</li>
-			<li>recfiles</li>
 			<li>Java .properties</li>
 			<li>Gettext / .po files</li>
 			<li>sqlite (full database dump)</li>
@@ -160,7 +159,6 @@
 		<ul>
 			<li>GUI in GTK</li>
 			<li>(La)TeX</li>
-			<li>Record, Single (like in <a href="https://sql-dk.globalcode.info/">SQL-DK</a>)</li>
 			<li>XML tree</li>
 			<li>iCalendar</li>
 			<li>vCard</li>
@@ -168,10 +166,8 @@
 			<li>YAML, JSON, INI</li>
 			<li>CSV, DSV</li>
 			<li>Fsdb</li>
-			<li>recfiles</li>
 			<li>Java .properties</li>
 			<li>Gettext / .po files</li>
-			<li>ASN.1</li>
 			<li>XHTML: tables, graphs</li>
 			<li>GraphViz: .dot files</li>
 			<li>SQL: CREATE TABLE, INSERT</li>
relpipe/relpipe-web