--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/relpipe-data/examples-ini.xml Mon Feb 21 00:43:11 2022 +0100
@@ -0,0 +1,333 @@
+<stránka
+ xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
+ xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
+
+ <nadpis>Reading and writing INI and unix configs</nadpis>
+ <perex>work with INI, classic key=value configurations, Java .properties or Manifests</perex>
+ <m:pořadí-příkladu>05300</m:pořadí-příkladu>
+
+ <text xmlns="http://www.w3.org/1999/xhtml">
+
+ <p>
+ INI is very common simple text format used for configuration files.
+ It extends classic <i>key=value</i> config files and adds one more level – sections – for structuring.
+ </p>
+
+ <p>
+ Unfortunatelly, there is no single INI standard and we have to deal with various INI dialects.
+ Some organizations specify their own INI dialect, but often INI files lacking any formal specification are used.
+ Our tools support various specifics and dialects through the <code>--parser-option</code> options.
+ Like any other CLI options, they are supported in our Bash completion scripts.
+ </p>
+
+ <!--
+ <p>(please note that OS/2 uses binary INI files that are currently not supported in <m:name/> and it is unsure whether they will ever be)</p>
+ -->
+
+ <p>
+ Classic (<i>unix</i>) <i>key=value</i> config files can be considered a dialect or a subset of the INI format
+ and thus be processed by INI input and output filters.
+ It is like INI without any sections or INI having only global properties (<i>key=value</i> at the beginning of the file).
+ Java <i>.properties</i> and <i>MANIFEST.MF</i> files are similar case. They also can be processed by our tools.
+ </p>
+
+ <h3>Simple INI example</h3>
+
+ <p>This is a simple INI file:</p>
+
+ <m:pre jazyk="ini" src="examples/ini-simple.ini"/>
+
+ <p>We read such INI file using this command:</p>
+
+ <m:pre jazyk="text"><![CDATA[cat ini-simple.ini | relpipe-in-ini | relpipe-out-tabular]]></m:pre>
+
+ <p>and get:</p>
+
+ <m:pre jazyk="text" src="examples/ini-simple.tabular"/>
+
+ <p>
+ Of course, we can do more that listing the entries as a table.
+ We can convert them to other formats or e.g. run some command on each entry.
+ Or modify certain values or add/remove entries and save as new INI file.
+ </p>
+
+ <h3>More complex INI example</h3>
+
+ <p>
+ The INI format is just seemingly simple. There is much more than <i>section/key/value</i>.
+ Our parser is by default configured to eat as much INI as possible.
+ It can be tuned by <code>--parser-option</code> CLI options.
+ Sometimes, such tuning is necessary, because some INI features are conflicting – some INI dialects are mutually exclusive.
+ </p>
+
+
+ <p>(please note that syntax highlighting does not support advanced INI features and is sometimes broken)</p>
+
+ <m:pre jazyk="ini" src="examples/ini-complex.ini"/>
+
+ <p>We read such INI file using this command:</p>
+
+ <m:pre jazyk="text"><![CDATA[cat ini-simple.ini | relpipe-in-ini --enable-sub-keys true | relpipe-out-tabular]]></m:pre>
+
+ <p>and get:</p>
+
+ <m:pre jazyk="text" src="examples/ini-complex.tabular"/>
+
+ <p>
+ If we omit the <code>--enable-sub-keys true</code> option, the <code>sub_key</code> attribute disappears
+ and we get <code>a[x]</code> and <code>a[y]</code> in the <code>key</code> attribute
+ i.e. brackets are not interpreted in any special way – they are considered just part of the key.
+ </p>
+
+ <p>See Bash completion for complete list of options and dialects.</p>
+
+
+ <h3>Keeping comments and whitespace while modifying the INI files</h3>
+
+ <p>
+ Comments and empty lines are ignored by default and does not generate any output.
+ This is desired behavior in most cases.
+ But sometimes we want to pass them through.
+ We may e.g. generate some XHTML report or other user-friendly output containing the comments,
+ or we may want to modify existing INI without losing original comments and formatting.
+ The tool offers the <code>--enable-comments</code> and <code>--enable-whitespace</code> options.
+ They allows lossless or almost lossless round-trip conversion from INI to relations and back.
+ </p>
+
+ <p>
+ Besides that, we have also the <code>--enable-event-numbers</code> and <code>--enable-line-numbers</code> options.
+ But they are useful mostly for debugging purposes.
+ </p>
+
+ <p>If we enable all that options, we get this output:</p>
+
+ <m:pre jazyk="text" src="examples/ini-complex.full.tabular"/>
+
+ <p>
+ So we can take the simple INI file (see above) and pass it through a filter
+ that changes (on-the-fly) the <code>x</code> value in the <code>first-section</code>:
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[cat ini-simple.ini \
+ | relpipe-in-ini --enable-comments true \
+ | relpipe-tr-scheme \
+ --relation 'ini' \
+ --for-each '
+ (if (and (string= $section "first-section") (string= $key "x"))
+ (set! $value "256"))' \
+ | relpipe-out-ini]]></m:pre>
+
+ <p>and get an INI file that contains original comments and modified values:</p>
+
+ <m:pre jazyk="ini" src="examples/ini-simple.modified.ini"/>
+
+ <p>
+ Both the comment at the beginning of the file
+ and the <code>x</code> value in <code>second-section</code>
+ were kept intact.
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[cat ini-simple.ini \
+ | relpipe-in-ini --enable-comments true \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --for-each '(section == "first-section" && key == "x") { value = "256"; }; record();' \
+ | relpipe-out-ini]]></m:pre>
+
+ <p>
+ This AWK filter will generate the same output.
+ And we can simply add comments just by adding e.g. <code>comment = "the script changed this value";</code> after the <code>value = "256";</code>
+ which results in:
+ </p>
+
+ <m:pre jazyk="ini"><![CDATA[x = 256 ; the script changed this value]]></m:pre>
+
+ <p>This way we can automatically modify our configuration files.</p>
+
+
+ <h3>Generating INI files</h3>
+
+ <p>
+ The <code>relpipe-out-ini</code> tool serves not only for recreating already existing INI files but also for creating new ones.
+ Data may come from various sources – CSV editor, script, SQL query etc. – and may come in diferent shapes.
+ While on the INI side we use the term „dialect“, on the relational side we use the term „style“.
+ </p>
+
+
+ <p>So in <code>relpipe-out-ini</code> we can chose from several <i>styles</i>:</p>
+
+ <ul>
+ <li><code>standard</code>: expects same structure as generated by <code>relpipe-in-ini</code>; can generate also comments and whitespace</li>
+ <li><code>literal</code>: relation name → section name; attribute name → key; attribute value → value; multiple records → duplicate section names</li>
+ <li><code>section-first</code>: like <code>literal</code>, just section names are taken from the first attribute</li>
+ <li><code>dropped</code>: this relation is ignored/skipped</li>
+ <li><code>auto</code>: use <code>standard</code> style if the relation has <code>key</code> and <code>value</code> attributes; otherwise use <code>literal</code> style</li>
+ </ul>
+
+ <p>Each relation might be processed in different way – use the <code>--relation</code> and regular expression describing the relation(s) name.</p>
+
+ <p>
+ We may for example run:
+ </p>
+
+ <m:pre jazyk="shell"><![CDATA[relpipe-in-x11 --list-input-devices true \
+ | relpipe-tr-awk \
+ --relation '.*' \
+ --output-attribute 'type' string \
+ --output-attribute 'id' integer \
+ --output-attribute 'name' string \
+ --where 'type == "mouse" || type == "keyboard"' \
+ | relpipe-out-ini --style section-first]]></m:pre>
+
+ <p>and get INI like this:</p>
+
+
+ <m:pre jazyk="ini"><![CDATA[[keyboard]
+id = 6
+name = Power Button
+
+[keyboard]
+id = 7
+name = Video Bus
+
+[keyboard]
+id = 8
+name = Power Button
+
+[mouse]
+id = 10
+name = Logitech USB Trackball
+
+[keyboard]
+id = 16
+name = AT Translated Set 2 keyboard
+
+[keyboard]
+id = 12
+name = ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard
+
+[mouse]
+id = 9
+name = 3Dconnexion 3Dconnexion Universal Receiver Mouse]]></m:pre>
+
+ <p>
+ This way, we can generate INI files to be loaded as configuration files by other programs.
+ We can also use <code>relpipe-out-ini</code> for displaying data rather vertically than horizontally (like in common <code>relpipe-out-tabular</code>).
+ This is useful for data with long values or many attributes.
+ The <code>relpipe-out-recfile</code> tool can also server this purpose.
+ </p>
+
+ <h3>Generating XHTML report from several INI files</h3>
+
+ <p>
+ INI format is used by many programs like Mercurial, Git, KDE, Gnome… and also by <m:a href="examples-tr-sql-odbc">ODBC</m:a>.
+ Its configuration is spread across several INI files (system-wide driver configuration and connections and user-specific connections).
+ We can collect all these INI files, use their names as relation names and generate a nice XHTML report:
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[for file in /etc/odbcinst.ini /etc/odbc.ini ~/.odbc.ini ; do
+ cat "$file" | relpipe-in-ini --relation "$(basename $file)";
+done | relpipe-out-xhtml > odbc-report.xhtml]]></m:pre>
+
+ <p>
+ n.b. It might contain also sensitive information like passwords.
+ We may use some <code>relpipe-tr-*</code> filter to remove such records or replace such values by <code>********</code>.
+ </p>
+
+
+ <h3>Java .properties and MANIFEST.MF</h3>
+
+ <p>
+ Java has built-in support for simple key=value format – so called <i>.properties files</i>.
+ They are often used for storing flat data where more advanced format (like XML) is not necessary.
+ This format supports comments and some escaping sequences.
+ Escaping of non-ASCII characters was mandatory but since Java 9 the UTF-8 encoding is used by default (no need to call <code>native2ascii</code>).
+ In <m:name/> we consider .properties files to be a dialect of the INI format so it is supported in <code>relpipe-in-ini</code> and <code>relpipe-out-ini</code>.
+ The same applies to the MANIFEST.MF which is another key=value format used in the Java world.
+ Thus we can read and write e.g. Java resource bundles (localization), Java EE or Spring application configs or MANIFEST.MF containing OSGi metadata and other information.
+ </p>
+
+ <m:pre jazyk="bash"><![CDATA[# read .properties or MANIFEST.MF:
+cat example.properties | relpipe-in-ini --parser-option dialect java-properties | …
+cat MANIFEST.MF | relpipe-in-ini --parser-option dialect java-manifest-mf | …
+
+# write .properties or MANIFEST.MF:
+… | relpipe-out-ini --writer-option dialect java-properties > example.properties
+… | relpipe-out-ini --writer-option dialect java-manifest-mf > MANIFEST.MF]]></m:pre>
+
+ <p>
+ We can pass such data through various transformation (SQL, AWK, Scheme, XPath etc.)
+ and do conversions to/from various formats (CSV, INI, XML, XHTML, YAML, Recfile, ASN.1 etc.)
+ or convert .properties to MANIFEST.MF and vice versa.
+ </p>
+
+
+ <h3>Classic <i>unix</i> key=value config files</h3>
+
+ <p>
+ Many <i>unix</i> or GNU/Linux programs use simple key=value configuration files with <code>#</code> comments
+ and some basic escaping (like <code>\\</code> → <code>\</code>, <code>\n</code> → <i>new line</i> etc.)
+ It is very similar to Java .properties mentioned above.
+ </p>
+
+ <p>
+ The configuration is often spread across many small files heavily stuffed with comments.
+ If we want to collect just the valid configuration entries from all the files,
+ we may somehow misuse the INI input filter and the fact that given files contain no sections – turn the filenames into section names
+ and feed the result to the <code>relpipe-in-ini</code>:
+ </p>
+
+
+ <m:pre jazyk="bash"><![CDATA[head -n -0 /etc/sysctl.d/*.conf \
+ | sed -E 's/==> (.*) <==/[\1]/g' \
+ | relpipe-in-ini --relation 'sysctl' \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>This gives us nice overview of our system configuration:</p>
+ <m:pre jazyk="text"><![CDATA[sysctl:
+ ╭─────────────────────────────────────────┬────────────────────────────────────┬────────────────╮
+ │ section (string) │ key (string) │ value (string) │
+ ├─────────────────────────────────────────┼────────────────────────────────────┼────────────────┤
+ │ /etc/sysctl.d/uhd-usrp2.conf │ net.core.rmem_max │ 50000000 │
+ │ /etc/sysctl.d/uhd-usrp2.conf │ net.core.wmem_max │ 1048576 │
+ │ /etc/sysctl.d/10-console-messages.conf │ kernel.printk │ 4 4 1 7 │
+ │ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.all.use_tempaddr │ 2 │
+ │ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.default.use_tempaddr │ 2 │
+ │ /etc/sysctl.d/10-kernel-hardening.conf │ kernel.kptr_restrict │ 1 │
+ │ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_hardlinks │ 1 │
+ │ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_symlinks │ 1 │
+ │ /etc/sysctl.d/10-magic-sysrq.conf │ kernel.sysrq │ 176 │
+ │ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.default.rp_filter │ 1 │
+ │ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.all.rp_filter │ 1 │
+ │ /etc/sysctl.d/10-network-security.conf │ net.ipv4.tcp_syncookies │ 1 │
+ │ /etc/sysctl.d/10-ptrace.conf │ kernel.yama.ptrace_scope │ 1 │
+ │ /etc/sysctl.d/10-zeropage.conf │ vm.mmap_min_addr │ 65536 │
+ │ /etc/sysctl.d/99-sysctl.conf │ kernel.sysrq │ 1 │
+ ╰─────────────────────────────────────────┴────────────────────────────────────┴────────────────╯
+Record count: 15]]></m:pre>
+
+
+ <p>We can also do this:</p>
+ <m:pre jazyk="bash"><![CDATA[find /etc/sysctl.d/ -name '*.conf' \
+ | while read f; do echo "[$f]"; cat "$f"; done \
+ | relpipe-in-ini \
+ | relpipe-out-tabular]]></m:pre>
+
+ <p>Or create a reusable function:</p>
+ <m:pre jazyk="bash"><![CDATA[list-config() {
+ head -n -0 "$@" \
+ | sed -E 's/==> (.*) <==/[\1]/g' \
+ | relpipe-in-ini --relation "configs" \
+ | relpipe-out-tabular --write-types false;
+}]]></m:pre>
+
+ <p>and call it this way:</p>
+ <m:pre jazyk="bash"><![CDATA[list-config /etc/sysctl.d/*.conf]]></m:pre>
+
+ <p>It is a bit dirty hack (what if the file name, key or value contains <code><==</code> or <code>[…]</code>?) but it will work quite well.</p>
+
+ <p>And like with Java .properties and MANIFEST-MF, we can use the <code>relpipe-out-ini</code> to generate the configuration files.</p>
+
+ </text>
+
+</stránka>