diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-ini.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-ini.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,333 @@ + + + Reading and writing INI and unix configs + work with INI, classic key=value configurations, Java .properties or Manifests + 05300 + + + +

+ INI is very common simple text format used for configuration files. + It extends classic key=value config files and adds one more level – sections – for structuring. +

+ +

+ Unfortunatelly, there is no single INI standard and we have to deal with various INI dialects. + Some organizations specify their own INI dialect, but often INI files lacking any formal specification are used. + Our tools support various specifics and dialects through the --parser-option options. + Like any other CLI options, they are supported in our Bash completion scripts. +

+ + + +

+ Classic (unix) key=value config files can be considered a dialect or a subset of the INI format + and thus be processed by INI input and output filters. + It is like INI without any sections or INI having only global properties (key=value at the beginning of the file). + Java .properties and MANIFEST.MF files are similar case. They also can be processed by our tools. +

+ +

Simple INI example

+ +

This is a simple INI file:

+ + + +

We read such INI file using this command:

+ + + +

and get:

+ + + +

+ Of course, we can do more that listing the entries as a table. + We can convert them to other formats or e.g. run some command on each entry. + Or modify certain values or add/remove entries and save as new INI file. +

+ +

More complex INI example

+ +

+ The INI format is just seemingly simple. There is much more than section/key/value. + Our parser is by default configured to eat as much INI as possible. + It can be tuned by --parser-option CLI options. + Sometimes, such tuning is necessary, because some INI features are conflicting – some INI dialects are mutually exclusive. +

+ + +

(please note that syntax highlighting does not support advanced INI features and is sometimes broken)

+ + + +

We read such INI file using this command:

+ + + +

and get:

+ + + +

+ If we omit the --enable-sub-keys true option, the sub_key attribute disappears + and we get a[x] and a[y] in the key attribute + i.e. brackets are not interpreted in any special way – they are considered just part of the key. +

+ +

See Bash completion for complete list of options and dialects.

+ + +

Keeping comments and whitespace while modifying the INI files

+ +

+ Comments and empty lines are ignored by default and does not generate any output. + This is desired behavior in most cases. + But sometimes we want to pass them through. + We may e.g. generate some XHTML report or other user-friendly output containing the comments, + or we may want to modify existing INI without losing original comments and formatting. + The tool offers the --enable-comments and --enable-whitespace options. + They allows lossless or almost lossless round-trip conversion from INI to relations and back. +

+ +

+ Besides that, we have also the --enable-event-numbers and --enable-line-numbers options. + But they are useful mostly for debugging purposes. +

+ +

If we enable all that options, we get this output:

+ + + +

+ So we can take the simple INI file (see above) and pass it through a filter + that changes (on-the-fly) the x value in the first-section: +

+ + + +

and get an INI file that contains original comments and modified values:

+ + + +

+ Both the comment at the beginning of the file + and the x value in second-section + were kept intact. +

+ + + +

+ This AWK filter will generate the same output. + And we can simply add comments just by adding e.g. comment = "the script changed this value"; after the value = "256"; + which results in: +

+ + + +

This way we can automatically modify our configuration files.

+ + +

Generating INI files

+ +

+ The relpipe-out-ini tool serves not only for recreating already existing INI files but also for creating new ones. + Data may come from various sources – CSV editor, script, SQL query etc. – and may come in diferent shapes. + While on the INI side we use the term „dialect“, on the relational side we use the term „style“. +

+ + +

So in relpipe-out-ini we can chose from several styles:

+ +

standard: expects same structure as generated by relpipe-in-ini; can generate also comments and whitespace
literal: relation name → section name; attribute name → key; attribute value → value; multiple records → duplicate section names
section-first: like literal, just section names are taken from the first attribute
dropped: this relation is ignored/skipped
auto: use standard style if the relation has key and value attributes; otherwise use literal style

+ +

Each relation might be processed in different way – use the --relation and regular expression describing the relation(s) name.

+ +

+ We may for example run: +

+ + + +

and get INI like this:

+ + + + +

+ This way, we can generate INI files to be loaded as configuration files by other programs. + We can also use relpipe-out-ini for displaying data rather vertically than horizontally (like in common relpipe-out-tabular). + This is useful for data with long values or many attributes. + The relpipe-out-recfile tool can also server this purpose. +

+ +

Generating XHTML report from several INI files

+ +

+ INI format is used by many programs like Mercurial, Git, KDE, Gnome… and also by ODBC. + Its configuration is spread across several INI files (system-wide driver configuration and connections and user-specific connections). + We can collect all these INI files, use their names as relation names and generate a nice XHTML report: +

+ + odbc-report.xhtml]]> + +

+ n.b. It might contain also sensitive information like passwords. + We may use some relpipe-tr-* filter to remove such records or replace such values by ********. +

+ + +

Java .properties and MANIFEST.MF

+ +

+ Java has built-in support for simple key=value format – so called .properties files. + They are often used for storing flat data where more advanced format (like XML) is not necessary. + This format supports comments and some escaping sequences. + Escaping of non-ASCII characters was mandatory but since Java 9 the UTF-8 encoding is used by default (no need to call native2ascii). + In we consider .properties files to be a dialect of the INI format so it is supported in relpipe-in-ini and relpipe-out-ini. + The same applies to the MANIFEST.MF which is another key=value format used in the Java world. + Thus we can read and write e.g. Java resource bundles (localization), Java EE or Spring application configs or MANIFEST.MF containing OSGi metadata and other information. +

+ + example.properties +… | relpipe-out-ini --writer-option dialect java-manifest-mf > MANIFEST.MF]]> + +

+ We can pass such data through various transformation (SQL, AWK, Scheme, XPath etc.) + and do conversions to/from various formats (CSV, INI, XML, XHTML, YAML, Recfile, ASN.1 etc.) + or convert .properties to MANIFEST.MF and vice versa. +

+ + +

Classic unix key=value config files

+ +

+ Many unix or GNU/Linux programs use simple key=value configuration files with # comments + and some basic escaping (like \\ → \, \n → new line etc.) + It is very similar to Java .properties mentioned above. +

+ +

+ The configuration is often spread across many small files heavily stuffed with comments. + If we want to collect just the valid configuration entries from all the files, + we may somehow misuse the INI input filter and the fact that given files contain no sections – turn the filenames into section names + and feed the result to the relpipe-in-ini: +

+ + + (.*) <==/[\1]/g' \ + | relpipe-in-ini --relation 'sysctl' \ + | relpipe-out-tabular]]> + +

This gives us nice overview of our system configuration:

+ + + +

We can also do this:

+ + +

Or create a reusable function:

+ (.*) <==/[\1]/g' \ + | relpipe-in-ini --relation "configs" \ + | relpipe-out-tabular --write-types false; +}]]> + +

and call it this way:

+ + +

It is a bit dirty hack (what if the file name, key or value contains <== or […]?) but it will work quite well.

+ +

And like with Java .properties and MANIFEST-MF, we can use the relpipe-out-ini to generate the configuration files.

+ + + +