# HG changeset patch # User František Kučera # Date 1543345659 -3600 # Node ID 5697a01db3882f48219050b7c8e444bdf1bb44bd # Parent 7d7d4e1f293f80230354a469f176e3d145eb03ab roadmap diff -r 7d7d4e1f293f -r 5697a01db388 relpipe-data/classic-example.xml --- a/relpipe-data/classic-example.xml Tue Nov 27 17:12:12 2018 +0100 +++ b/relpipe-data/classic-example.xml Tue Nov 27 20:07:39 2018 +0100 @@ -116,7 +116,7 @@

- There are also other problems like character encoding, missing meta-data (e.g. field names and types), joining multiple files (Is there always a new-line character at the end of the file? Or is there a BOM at the beginning of the file?) + There are also other problems like character encoding, missing meta-data (e.g. field names and types), joining multiple files (Is there always a new-line character at the end of the file? Or is there some nasty BOM at the beginning of the file?) or passing several types of data in a single stream (we have list of animals and we can have e.g. also a list of foods or list of our staff where each list has different fields).

diff -r 7d7d4e1f293f -r 5697a01db388 relpipe-data/faq.xml --- a/relpipe-data/faq.xml Tue Nov 27 17:12:12 2018 +0100 +++ b/relpipe-data/faq.xml Tue Nov 27 20:07:39 2018 +0100 @@ -16,7 +16,7 @@ So it is not a big difference whether it will be released this month or the next one. We understand the release early, release often rule. But it fits better to application software than to standards and APIs. - Of course, we expect some evolution after the v1.0.0 release, but we need to stabilize and verify many things before the release in order to be able to maintain hackward compatibility in future. + Of course, we expect some evolution after the v1.0.0 release, but we need to stabilize and verify many things before the release in order to be able to maintain backward compatibility in future.

diff -r 7d7d4e1f293f -r 5697a01db388 relpipe-data/roadmap.xml --- a/relpipe-data/roadmap.xml Tue Nov 27 17:12:12 2018 +0100 +++ b/relpipe-data/roadmap.xml Tue Nov 27 20:07:39 2018 +0100 @@ -8,8 +8,218 @@

- The world is relational! + Releases before v1.0.0 are development and are not intended for production use. + Releases after v1.0.0 should follow the rules of Semantic versioning. +

+ +

v0.8

+ +

+ First public release for brave and courageous. +

+ +

Data types

boolean
variable unsigned integer (prototype)
string in UTF-8

Inputs

CLI
fstab

Transformations

grep: regular expression filter
sed: regular expression replacer

Outputs

tabular
XML
GUI in Qt
ODS (LibreOffice)

+ + +

v0.9, v0.10, v0.11 etc.

+ +

+ Releases for discussion and verification of the format and API design. +

+ +

Data types

arrays/streams of bytes (octets)
arrays of other types
arrays of other types with support for NULL values
nested relations
date and time
variable integer (probably LEB128 or similar)
decimal numbers
support NULL values (introduce bitmaps)
support long strings (chunked, similar to octet streams)
fixed size integers
precise BigDecimal
fractions
string in UTF-16
string in ASCII
string in ISO 8859-1
string in ISO 8859-2

+ +

Inputs

XML
DSV
regular expression

Transformations

SQL: sqlite

Outputs

CLI: null-byte separated values

+ + +

v1.0

+ +

+ First version for production use. + Brings no new features, just stabilized result of previous development. + Stable must be: +

+ +

format specification
relpipe-lib-writer
relpipe-lib-reader
relpipe-in-cli
relpipe-out-tabular
relpipe-out-xml

+ +

+ Other parts might be released as stable later. +

+ After this point, all components (the format specification, particular libraries and particular tools) + will be versioned independently and the compatibility matrix will be maintained.

+ +

Further versions

+ Plans for next decades: +

+ +

Data types

intervals of various types
IPv4 address and subnet
IPv6 address and subnet
UUID
e-mail
URL / URI
geographic locations
OID: Object identifier

Inputs

Systems, commands:

files: ls, find, grep
network information: ip, iptables, netstat, ss, dhcp-lease-list
network interaction: ping, host, wget, curl
system information: ps, lsof
versioning systems (Mercurial, Git, Subversion, Monotone, Bazaar)
SELECTs from sqlite, SQL-DK

+ +

Formats:

XML: as xmlTable() in SQL
ODS (LibreOffice)
YAML, JSON, INI
CSV, DSV
Fsdb
recfiles
Java .properties
Gettext / .po files
sqlite (full database dump)
STDIO log: captured STDIN, STDOUT and STDERR of other process with precise timing of all events
Inverse tabular
Wikipedia / MediaWiki: harvest tables from given article

Transformations

iconv: character encoding converter
rename: relations, attributes
filter/skip: relations, attributes
add constant attribute
WHERE-like filter
JOIN
ORDER BY
UNION, UNION ALL, explicit or implicit (relation interleaving)
split single relation into multiple relations according to an attribute value
union multiple relations and add an attribute with original relation name
XPath, XSLT, XQuery
statistics: compute/add aggregated values like min(), max(), avg(), sum() or percentiles
function calls (probably should be part od SQL or XPath)
pack and unpack: single record with arrays vs. multiple records with scalars
Guile, Lua, Perl, Python
system command executor

Outputs

GUI in GTK
(La)TeX
Record, Single (like in SQL-DK)
XML tree
YAML, JSON, INI
CSV, DSV
Fsdb
recfiles
Java .properties
Gettext / .po files
ASN.1
XHTML: tables, graphs
GraphViz: .dot files
SQL: CREATE TABLE, INSERT
SMTP, IMAP, files, HTTP, SOAP, ZeroMQ, TCP, UDP
source code / literals in Java, C, C++, Bash
return/exit code (single numeric value)

+ +

Libraries and tools

Readers (SAX-like parsers) and writers (generators) for:

C++
C
Java
Python
Perl
Rust
Go
PHP

Other libraries and tools:

ORM API: mapping between classes/objects and relations/records
schemas (XSD-like)
generators, compilers, validators, comparators
transformer helpers
Ragel input helpers
PEG input helpers
repair tools (for corrupted data)
pull parsers
SAX and DOM parsers (read relational data like it was XML)
visual editor for pipeline design

+ +