<stránka
xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
<nadpis>Reading and writing INI and unix configs</nadpis>
<perex>work with INI, classic key=value configurations, Java .properties or Manifests</perex>
<m:pořadí-příkladu>05300</m:pořadí-příkladu>
<text xmlns="http://www.w3.org/1999/xhtml">
<p>
INI is very common simple text format used for configuration files.
It extends classic <i>key=value</i> config files and adds one more level – sections – for structuring.
</p>
<p>
Unfortunatelly, there is no single INI standard and we have to deal with various INI dialects.
Some organizations specify their own INI dialect, but often INI files lacking any formal specification are used.
Our tools support various specifics and dialects through the <code>--parser-option</code> options.
Like any other CLI options, they are supported in our Bash completion scripts.
</p>
<!--
<p>(please note that OS/2 uses binary INI files that are currently not supported in <m:name/> and it is unsure whether they will ever be)</p>
-->
<p>
Classic (<i>unix</i>) <i>key=value</i> config files can be considered a dialect or a subset of the INI format
and thus be processed by INI input and output filters.
It is like INI without any sections or INI having only global properties (<i>key=value</i> at the beginning of the file).
Java <i>.properties</i> and <i>MANIFEST.MF</i> files are similar case. They also can be processed by our tools.
</p>
<h3>Simple INI example</h3>
<p>This is a simple INI file:</p>
<m:pre jazyk="ini" src="examples/ini-simple.ini"/>
<p>We read such INI file using this command:</p>
<m:pre jazyk="text"><![CDATA[cat ini-simple.ini | relpipe-in-ini | relpipe-out-tabular]]></m:pre>
<p>and get:</p>
<m:pre jazyk="text" src="examples/ini-simple.tabular"/>
<p>
Of course, we can do more that listing the entries as a table.
We can convert them to other formats or e.g. run some command on each entry.
Or modify certain values or add/remove entries and save as new INI file.
</p>
<h3>More complex INI example</h3>
<p>
The INI format is just seemingly simple. There is much more than <i>section/key/value</i>.
Our parser is by default configured to eat as much INI as possible.
It can be tuned by <code>--parser-option</code> CLI options.
Sometimes, such tuning is necessary, because some INI features are conflicting – some INI dialects are mutually exclusive.
</p>
<p>(please note that syntax highlighting does not support advanced INI features and is sometimes broken)</p>
<m:pre jazyk="ini" src="examples/ini-complex.ini"/>
<p>We read such INI file using this command:</p>
<m:pre jazyk="text"><![CDATA[cat ini-simple.ini | relpipe-in-ini --enable-sub-keys true | relpipe-out-tabular]]></m:pre>
<p>and get:</p>
<m:pre jazyk="text" src="examples/ini-complex.tabular"/>
<p>
If we omit the <code>--enable-sub-keys true</code> option, the <code>sub_key</code> attribute disappears
and we get <code>a[x]</code> and <code>a[y]</code> in the <code>key</code> attribute
i.e. brackets are not interpreted in any special way – they are considered just part of the key.
</p>
<p>See Bash completion for complete list of options and dialects.</p>
<h3>Keeping comments and whitespace while modifying the INI files</h3>
<p>
Comments and empty lines are ignored by default and does not generate any output.
This is desired behavior in most cases.
But sometimes we want to pass them through.
We may e.g. generate some XHTML report or other user-friendly output containing the comments,
or we may want to modify existing INI without losing original comments and formatting.
The tool offers the <code>--enable-comments</code> and <code>--enable-whitespace</code> options.
They allows lossless or almost lossless round-trip conversion from INI to relations and back.
</p>
<p>
Besides that, we have also the <code>--enable-event-numbers</code> and <code>--enable-line-numbers</code> options.
But they are useful mostly for debugging purposes.
</p>
<p>If we enable all that options, we get this output:</p>
<m:pre jazyk="text" src="examples/ini-complex.full.tabular"/>
<p>
So we can take the simple INI file (see above) and pass it through a filter
that changes (on-the-fly) the <code>x</code> value in the <code>first-section</code>:
</p>
<m:pre jazyk="bash"><![CDATA[cat ini-simple.ini \
| relpipe-in-ini --enable-comments true \
| relpipe-tr-scheme \
--relation 'ini' \
--for-each '
(if (and (string= $section "first-section") (string= $key "x"))
(set! $value "256"))' \
| relpipe-out-ini]]></m:pre>
<p>and get an INI file that contains original comments and modified values:</p>
<m:pre jazyk="ini" src="examples/ini-simple.modified.ini"/>
<p>
Both the comment at the beginning of the file
and the <code>x</code> value in <code>second-section</code>
were kept intact.
</p>
<m:pre jazyk="bash"><![CDATA[cat ini-simple.ini \
| relpipe-in-ini --enable-comments true \
| relpipe-tr-awk \
--relation '.*' \
--for-each '(section == "first-section" && key == "x") { value = "256"; }; record();' \
| relpipe-out-ini]]></m:pre>
<p>
This AWK filter will generate the same output.
And we can simply add comments just by adding e.g. <code>comment = "the script changed this value";</code> after the <code>value = "256";</code>
which results in:
</p>
<m:pre jazyk="ini"><![CDATA[x = 256 ; the script changed this value]]></m:pre>
<p>This way we can automatically modify our configuration files.</p>
<h3>Generating INI files</h3>
<p>
The <code>relpipe-out-ini</code> tool serves not only for recreating already existing INI files but also for creating new ones.
Data may come from various sources – CSV editor, script, SQL query etc. – and may come in diferent shapes.
While on the INI side we use the term „dialect“, on the relational side we use the term „style“.
</p>
<p>So in <code>relpipe-out-ini</code> we can chose from several <i>styles</i>:</p>
<ul>
<li><code>standard</code>: expects same structure as generated by <code>relpipe-in-ini</code>; can generate also comments and whitespace</li>
<li><code>literal</code>: relation name → section name; attribute name → key; attribute value → value; multiple records → duplicate section names</li>
<li><code>section-first</code>: like <code>literal</code>, just section names are taken from the first attribute</li>
<li><code>dropped</code>: this relation is ignored/skipped</li>
<li><code>auto</code>: use <code>standard</code> style if the relation has <code>key</code> and <code>value</code> attributes; otherwise use <code>literal</code> style</li>
</ul>
<p>Each relation might be processed in different way – use the <code>--relation</code> and regular expression describing the relation(s) name.</p>
<p>
We may for example run:
</p>
<m:pre jazyk="shell"><![CDATA[relpipe-in-x11 --list-input-devices true \
| relpipe-tr-awk \
--relation '.*' \
--output-attribute 'type' string \
--output-attribute 'id' integer \
--output-attribute 'name' string \
--where 'type == "mouse" || type == "keyboard"' \
| relpipe-out-ini --style section-first]]></m:pre>
<p>and get INI like this:</p>
<m:pre jazyk="ini"><![CDATA[[keyboard]
id = 6
name = Power Button
[keyboard]
id = 7
name = Video Bus
[keyboard]
id = 8
name = Power Button
[mouse]
id = 10
name = Logitech USB Trackball
[keyboard]
id = 16
name = AT Translated Set 2 keyboard
[keyboard]
id = 12
name = ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard
[mouse]
id = 9
name = 3Dconnexion 3Dconnexion Universal Receiver Mouse]]></m:pre>
<p>
This way, we can generate INI files to be loaded as configuration files by other programs.
We can also use <code>relpipe-out-ini</code> for displaying data rather vertically than horizontally (like in common <code>relpipe-out-tabular</code>).
This is useful for data with long values or many attributes.
The <code>relpipe-out-recfile</code> tool can also server this purpose.
</p>
<h3>Generating XHTML report from several INI files</h3>
<p>
INI format is used by many programs like Mercurial, Git, KDE, Gnome… and also by <m:a href="examples-tr-sql-odbc">ODBC</m:a>.
Its configuration is spread across several INI files (system-wide driver configuration and connections and user-specific connections).
We can collect all these INI files, use their names as relation names and generate a nice XHTML report:
</p>
<m:pre jazyk="bash"><![CDATA[for file in /etc/odbcinst.ini /etc/odbc.ini ~/.odbc.ini ; do
cat "$file" | relpipe-in-ini --relation "$(basename $file)";
done | relpipe-out-xhtml > odbc-report.xhtml]]></m:pre>
<p>
n.b. It might contain also sensitive information like passwords.
We may use some <code>relpipe-tr-*</code> filter to remove such records or replace such values by <code>********</code>.
</p>
<h3>Java .properties and MANIFEST.MF</h3>
<p>
Java has built-in support for simple key=value format – so called <i>.properties files</i>.
They are often used for storing flat data where more advanced format (like XML) is not necessary.
This format supports comments and some escaping sequences.
Escaping of non-ASCII characters was mandatory but since Java 9 the UTF-8 encoding is used by default (no need to call <code>native2ascii</code>).
In <m:name/> we consider .properties files to be a dialect of the INI format so it is supported in <code>relpipe-in-ini</code> and <code>relpipe-out-ini</code>.
The same applies to the MANIFEST.MF which is another key=value format used in the Java world.
Thus we can read and write e.g. Java resource bundles (localization), Java EE or Spring application configs or MANIFEST.MF containing OSGi metadata and other information.
</p>
<m:pre jazyk="bash"><![CDATA[# read .properties or MANIFEST.MF:
cat example.properties | relpipe-in-ini --parser-option dialect java-properties | …
cat MANIFEST.MF | relpipe-in-ini --parser-option dialect java-manifest-mf | …
# write .properties or MANIFEST.MF:
… | relpipe-out-ini --writer-option dialect java-properties > example.properties
… | relpipe-out-ini --writer-option dialect java-manifest-mf > MANIFEST.MF]]></m:pre>
<p>
We can pass such data through various transformation (SQL, AWK, Scheme, XPath etc.)
and do conversions to/from various formats (CSV, INI, XML, XHTML, YAML, Recfile, ASN.1 etc.)
or convert .properties to MANIFEST.MF and vice versa.
</p>
<h3>Classic <i>unix</i> key=value config files</h3>
<p>
Many <i>unix</i> or GNU/Linux programs use simple key=value configuration files with <code>#</code> comments
and some basic escaping (like <code>\\</code> → <code>\</code>, <code>\n</code> → <i>new line</i> etc.)
It is very similar to Java .properties mentioned above.
</p>
<p>
The configuration is often spread across many small files heavily stuffed with comments.
If we want to collect just the valid configuration entries from all the files,
we may somehow misuse the INI input filter and the fact that given files contain no sections – turn the filenames into section names
and feed the result to the <code>relpipe-in-ini</code>:
</p>
<m:pre jazyk="bash"><![CDATA[head -n -0 /etc/sysctl.d/*.conf \
| sed -E 's/==> (.*) <==/[\1]/g' \
| relpipe-in-ini --relation 'sysctl' \
| relpipe-out-tabular]]></m:pre>
<p>This gives us nice overview of our system configuration:</p>
<m:pre jazyk="text"><![CDATA[sysctl:
╭─────────────────────────────────────────┬────────────────────────────────────┬────────────────╮
│ section (string) │ key (string) │ value (string) │
├─────────────────────────────────────────┼────────────────────────────────────┼────────────────┤
│ /etc/sysctl.d/uhd-usrp2.conf │ net.core.rmem_max │ 50000000 │
│ /etc/sysctl.d/uhd-usrp2.conf │ net.core.wmem_max │ 1048576 │
│ /etc/sysctl.d/10-console-messages.conf │ kernel.printk │ 4 4 1 7 │
│ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.all.use_tempaddr │ 2 │
│ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.default.use_tempaddr │ 2 │
│ /etc/sysctl.d/10-kernel-hardening.conf │ kernel.kptr_restrict │ 1 │
│ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_hardlinks │ 1 │
│ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_symlinks │ 1 │
│ /etc/sysctl.d/10-magic-sysrq.conf │ kernel.sysrq │ 176 │
│ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.default.rp_filter │ 1 │
│ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.all.rp_filter │ 1 │
│ /etc/sysctl.d/10-network-security.conf │ net.ipv4.tcp_syncookies │ 1 │
│ /etc/sysctl.d/10-ptrace.conf │ kernel.yama.ptrace_scope │ 1 │
│ /etc/sysctl.d/10-zeropage.conf │ vm.mmap_min_addr │ 65536 │
│ /etc/sysctl.d/99-sysctl.conf │ kernel.sysrq │ 1 │
╰─────────────────────────────────────────┴────────────────────────────────────┴────────────────╯
Record count: 15]]></m:pre>
<p>We can also do this:</p>
<m:pre jazyk="bash"><![CDATA[find /etc/sysctl.d/ -name '*.conf' \
| while read f; do echo "[$f]"; cat "$f"; done \
| relpipe-in-ini \
| relpipe-out-tabular]]></m:pre>
<p>Or create a reusable function:</p>
<m:pre jazyk="bash"><![CDATA[list-config() {
head -n -0 "$@" \
| sed -E 's/==> (.*) <==/[\1]/g' \
| relpipe-in-ini --relation "configs" \
| relpipe-out-tabular --write-types false;
}]]></m:pre>
<p>and call it this way:</p>
<m:pre jazyk="bash"><![CDATA[list-config /etc/sysctl.d/*.conf]]></m:pre>
<p>It is a bit dirty hack (what if the file name, key or value contains <code><==</code> or <code>[…]</code>?) but it will work quite well.</p>
<p>And like with Java .properties and MANIFEST-MF, we can use the <code>relpipe-out-ini</code> to generate the configuration files.</p>
</text>
</stránka>