|
1 <stránka |
|
2 xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" |
|
3 xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> |
|
4 |
|
5 <nadpis>Reading and writing INI and unix configs</nadpis> |
|
6 <perex>work with INI, classic key=value configurations, Java .properties or Manifests</perex> |
|
7 <m:pořadí-příkladu>05300</m:pořadí-příkladu> |
|
8 |
|
9 <text xmlns="http://www.w3.org/1999/xhtml"> |
|
10 |
|
11 <p> |
|
12 INI is very common simple text format used for configuration files. |
|
13 It extends classic <i>key=value</i> config files and adds one more level – sections – for structuring. |
|
14 </p> |
|
15 |
|
16 <p> |
|
17 Unfortunatelly, there is no single INI standard and we have to deal with various INI dialects. |
|
18 Some organizations specify their own INI dialect, but often INI files lacking any formal specification are used. |
|
19 Our tools support various specifics and dialects through the <code>--parser-option</code> options. |
|
20 Like any other CLI options, they are supported in our Bash completion scripts. |
|
21 </p> |
|
22 |
|
23 <!-- |
|
24 <p>(please note that OS/2 uses binary INI files that are currently not supported in <m:name/> and it is unsure whether they will ever be)</p> |
|
25 --> |
|
26 |
|
27 <p> |
|
28 Classic (<i>unix</i>) <i>key=value</i> config files can be considered a dialect or a subset of the INI format |
|
29 and thus be processed by INI input and output filters. |
|
30 It is like INI without any sections or INI having only global properties (<i>key=value</i> at the beginning of the file). |
|
31 Java <i>.properties</i> and <i>MANIFEST.MF</i> files are similar case. They also can be processed by our tools. |
|
32 </p> |
|
33 |
|
34 <h3>Simple INI example</h3> |
|
35 |
|
36 <p>This is a simple INI file:</p> |
|
37 |
|
38 <m:pre jazyk="ini" src="examples/ini-simple.ini"/> |
|
39 |
|
40 <p>We read such INI file using this command:</p> |
|
41 |
|
42 <m:pre jazyk="text"><![CDATA[cat ini-simple.ini | relpipe-in-ini | relpipe-out-tabular]]></m:pre> |
|
43 |
|
44 <p>and get:</p> |
|
45 |
|
46 <m:pre jazyk="text" src="examples/ini-simple.tabular"/> |
|
47 |
|
48 <p> |
|
49 Of course, we can do more that listing the entries as a table. |
|
50 We can convert them to other formats or e.g. run some command on each entry. |
|
51 Or modify certain values or add/remove entries and save as new INI file. |
|
52 </p> |
|
53 |
|
54 <h3>More complex INI example</h3> |
|
55 |
|
56 <p> |
|
57 The INI format is just seemingly simple. There is much more than <i>section/key/value</i>. |
|
58 Our parser is by default configured to eat as much INI as possible. |
|
59 It can be tuned by <code>--parser-option</code> CLI options. |
|
60 Sometimes, such tuning is necessary, because some INI features are conflicting – some INI dialects are mutually exclusive. |
|
61 </p> |
|
62 |
|
63 |
|
64 <p>(please note that syntax highlighting does not support advanced INI features and is sometimes broken)</p> |
|
65 |
|
66 <m:pre jazyk="ini" src="examples/ini-complex.ini"/> |
|
67 |
|
68 <p>We read such INI file using this command:</p> |
|
69 |
|
70 <m:pre jazyk="text"><![CDATA[cat ini-simple.ini | relpipe-in-ini --enable-sub-keys true | relpipe-out-tabular]]></m:pre> |
|
71 |
|
72 <p>and get:</p> |
|
73 |
|
74 <m:pre jazyk="text" src="examples/ini-complex.tabular"/> |
|
75 |
|
76 <p> |
|
77 If we omit the <code>--enable-sub-keys true</code> option, the <code>sub_key</code> attribute disappears |
|
78 and we get <code>a[x]</code> and <code>a[y]</code> in the <code>key</code> attribute |
|
79 i.e. brackets are not interpreted in any special way – they are considered just part of the key. |
|
80 </p> |
|
81 |
|
82 <p>See Bash completion for complete list of options and dialects.</p> |
|
83 |
|
84 |
|
85 <h3>Keeping comments and whitespace while modifying the INI files</h3> |
|
86 |
|
87 <p> |
|
88 Comments and empty lines are ignored by default and does not generate any output. |
|
89 This is desired behavior in most cases. |
|
90 But sometimes we want to pass them through. |
|
91 We may e.g. generate some XHTML report or other user-friendly output containing the comments, |
|
92 or we may want to modify existing INI without losing original comments and formatting. |
|
93 The tool offers the <code>--enable-comments</code> and <code>--enable-whitespace</code> options. |
|
94 They allows lossless or almost lossless round-trip conversion from INI to relations and back. |
|
95 </p> |
|
96 |
|
97 <p> |
|
98 Besides that, we have also the <code>--enable-event-numbers</code> and <code>--enable-line-numbers</code> options. |
|
99 But they are useful mostly for debugging purposes. |
|
100 </p> |
|
101 |
|
102 <p>If we enable all that options, we get this output:</p> |
|
103 |
|
104 <m:pre jazyk="text" src="examples/ini-complex.full.tabular"/> |
|
105 |
|
106 <p> |
|
107 So we can take the simple INI file (see above) and pass it through a filter |
|
108 that changes (on-the-fly) the <code>x</code> value in the <code>first-section</code>: |
|
109 </p> |
|
110 |
|
111 <m:pre jazyk="bash"><![CDATA[cat ini-simple.ini \ |
|
112 | relpipe-in-ini --enable-comments true \ |
|
113 | relpipe-tr-scheme \ |
|
114 --relation 'ini' \ |
|
115 --for-each ' |
|
116 (if (and (string= $section "first-section") (string= $key "x")) |
|
117 (set! $value "256"))' \ |
|
118 | relpipe-out-ini]]></m:pre> |
|
119 |
|
120 <p>and get an INI file that contains original comments and modified values:</p> |
|
121 |
|
122 <m:pre jazyk="ini" src="examples/ini-simple.modified.ini"/> |
|
123 |
|
124 <p> |
|
125 Both the comment at the beginning of the file |
|
126 and the <code>x</code> value in <code>second-section</code> |
|
127 were kept intact. |
|
128 </p> |
|
129 |
|
130 <m:pre jazyk="bash"><![CDATA[cat ini-simple.ini \ |
|
131 | relpipe-in-ini --enable-comments true \ |
|
132 | relpipe-tr-awk \ |
|
133 --relation '.*' \ |
|
134 --for-each '(section == "first-section" && key == "x") { value = "256"; }; record();' \ |
|
135 | relpipe-out-ini]]></m:pre> |
|
136 |
|
137 <p> |
|
138 This AWK filter will generate the same output. |
|
139 And we can simply add comments just by adding e.g. <code>comment = "the script changed this value";</code> after the <code>value = "256";</code> |
|
140 which results in: |
|
141 </p> |
|
142 |
|
143 <m:pre jazyk="ini"><![CDATA[x = 256 ; the script changed this value]]></m:pre> |
|
144 |
|
145 <p>This way we can automatically modify our configuration files.</p> |
|
146 |
|
147 |
|
148 <h3>Generating INI files</h3> |
|
149 |
|
150 <p> |
|
151 The <code>relpipe-out-ini</code> tool serves not only for recreating already existing INI files but also for creating new ones. |
|
152 Data may come from various sources – CSV editor, script, SQL query etc. – and may come in diferent shapes. |
|
153 While on the INI side we use the term „dialect“, on the relational side we use the term „style“. |
|
154 </p> |
|
155 |
|
156 |
|
157 <p>So in <code>relpipe-out-ini</code> we can chose from several <i>styles</i>:</p> |
|
158 |
|
159 <ul> |
|
160 <li><code>standard</code>: expects same structure as generated by <code>relpipe-in-ini</code>; can generate also comments and whitespace</li> |
|
161 <li><code>literal</code>: relation name → section name; attribute name → key; attribute value → value; multiple records → duplicate section names</li> |
|
162 <li><code>section-first</code>: like <code>literal</code>, just section names are taken from the first attribute</li> |
|
163 <li><code>dropped</code>: this relation is ignored/skipped</li> |
|
164 <li><code>auto</code>: use <code>standard</code> style if the relation has <code>key</code> and <code>value</code> attributes; otherwise use <code>literal</code> style</li> |
|
165 </ul> |
|
166 |
|
167 <p>Each relation might be processed in different way – use the <code>--relation</code> and regular expression describing the relation(s) name.</p> |
|
168 |
|
169 <p> |
|
170 We may for example run: |
|
171 </p> |
|
172 |
|
173 <m:pre jazyk="shell"><![CDATA[relpipe-in-x11 --list-input-devices true \ |
|
174 | relpipe-tr-awk \ |
|
175 --relation '.*' \ |
|
176 --output-attribute 'type' string \ |
|
177 --output-attribute 'id' integer \ |
|
178 --output-attribute 'name' string \ |
|
179 --where 'type == "mouse" || type == "keyboard"' \ |
|
180 | relpipe-out-ini --style section-first]]></m:pre> |
|
181 |
|
182 <p>and get INI like this:</p> |
|
183 |
|
184 |
|
185 <m:pre jazyk="ini"><![CDATA[[keyboard] |
|
186 id = 6 |
|
187 name = Power Button |
|
188 |
|
189 [keyboard] |
|
190 id = 7 |
|
191 name = Video Bus |
|
192 |
|
193 [keyboard] |
|
194 id = 8 |
|
195 name = Power Button |
|
196 |
|
197 [mouse] |
|
198 id = 10 |
|
199 name = Logitech USB Trackball |
|
200 |
|
201 [keyboard] |
|
202 id = 16 |
|
203 name = AT Translated Set 2 keyboard |
|
204 |
|
205 [keyboard] |
|
206 id = 12 |
|
207 name = ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard |
|
208 |
|
209 [mouse] |
|
210 id = 9 |
|
211 name = 3Dconnexion 3Dconnexion Universal Receiver Mouse]]></m:pre> |
|
212 |
|
213 <p> |
|
214 This way, we can generate INI files to be loaded as configuration files by other programs. |
|
215 We can also use <code>relpipe-out-ini</code> for displaying data rather vertically than horizontally (like in common <code>relpipe-out-tabular</code>). |
|
216 This is useful for data with long values or many attributes. |
|
217 The <code>relpipe-out-recfile</code> tool can also server this purpose. |
|
218 </p> |
|
219 |
|
220 <h3>Generating XHTML report from several INI files</h3> |
|
221 |
|
222 <p> |
|
223 INI format is used by many programs like Mercurial, Git, KDE, Gnome… and also by <m:a href="examples-tr-sql-odbc">ODBC</m:a>. |
|
224 Its configuration is spread across several INI files (system-wide driver configuration and connections and user-specific connections). |
|
225 We can collect all these INI files, use their names as relation names and generate a nice XHTML report: |
|
226 </p> |
|
227 |
|
228 <m:pre jazyk="bash"><![CDATA[for file in /etc/odbcinst.ini /etc/odbc.ini ~/.odbc.ini ; do |
|
229 cat "$file" | relpipe-in-ini --relation "$(basename $file)"; |
|
230 done | relpipe-out-xhtml > odbc-report.xhtml]]></m:pre> |
|
231 |
|
232 <p> |
|
233 n.b. It might contain also sensitive information like passwords. |
|
234 We may use some <code>relpipe-tr-*</code> filter to remove such records or replace such values by <code>********</code>. |
|
235 </p> |
|
236 |
|
237 |
|
238 <h3>Java .properties and MANIFEST.MF</h3> |
|
239 |
|
240 <p> |
|
241 Java has built-in support for simple key=value format – so called <i>.properties files</i>. |
|
242 They are often used for storing flat data where more advanced format (like XML) is not necessary. |
|
243 This format supports comments and some escaping sequences. |
|
244 Escaping of non-ASCII characters was mandatory but since Java 9 the UTF-8 encoding is used by default (no need to call <code>native2ascii</code>). |
|
245 In <m:name/> we consider .properties files to be a dialect of the INI format so it is supported in <code>relpipe-in-ini</code> and <code>relpipe-out-ini</code>. |
|
246 The same applies to the MANIFEST.MF which is another key=value format used in the Java world. |
|
247 Thus we can read and write e.g. Java resource bundles (localization), Java EE or Spring application configs or MANIFEST.MF containing OSGi metadata and other information. |
|
248 </p> |
|
249 |
|
250 <m:pre jazyk="bash"><![CDATA[# read .properties or MANIFEST.MF: |
|
251 cat example.properties | relpipe-in-ini --parser-option dialect java-properties | … |
|
252 cat MANIFEST.MF | relpipe-in-ini --parser-option dialect java-manifest-mf | … |
|
253 |
|
254 # write .properties or MANIFEST.MF: |
|
255 … | relpipe-out-ini --writer-option dialect java-properties > example.properties |
|
256 … | relpipe-out-ini --writer-option dialect java-manifest-mf > MANIFEST.MF]]></m:pre> |
|
257 |
|
258 <p> |
|
259 We can pass such data through various transformation (SQL, AWK, Scheme, XPath etc.) |
|
260 and do conversions to/from various formats (CSV, INI, XML, XHTML, YAML, Recfile, ASN.1 etc.) |
|
261 or convert .properties to MANIFEST.MF and vice versa. |
|
262 </p> |
|
263 |
|
264 |
|
265 <h3>Classic <i>unix</i> key=value config files</h3> |
|
266 |
|
267 <p> |
|
268 Many <i>unix</i> or GNU/Linux programs use simple key=value configuration files with <code>#</code> comments |
|
269 and some basic escaping (like <code>\\</code> → <code>\</code>, <code>\n</code> → <i>new line</i> etc.) |
|
270 It is very similar to Java .properties mentioned above. |
|
271 </p> |
|
272 |
|
273 <p> |
|
274 The configuration is often spread across many small files heavily stuffed with comments. |
|
275 If we want to collect just the valid configuration entries from all the files, |
|
276 we may somehow misuse the INI input filter and the fact that given files contain no sections – turn the filenames into section names |
|
277 and feed the result to the <code>relpipe-in-ini</code>: |
|
278 </p> |
|
279 |
|
280 |
|
281 <m:pre jazyk="bash"><![CDATA[head -n -0 /etc/sysctl.d/*.conf \ |
|
282 | sed -E 's/==> (.*) <==/[\1]/g' \ |
|
283 | relpipe-in-ini --relation 'sysctl' \ |
|
284 | relpipe-out-tabular]]></m:pre> |
|
285 |
|
286 <p>This gives us nice overview of our system configuration:</p> |
|
287 <m:pre jazyk="text"><![CDATA[sysctl: |
|
288 ╭─────────────────────────────────────────┬────────────────────────────────────┬────────────────╮ |
|
289 │ section (string) │ key (string) │ value (string) │ |
|
290 ├─────────────────────────────────────────┼────────────────────────────────────┼────────────────┤ |
|
291 │ /etc/sysctl.d/uhd-usrp2.conf │ net.core.rmem_max │ 50000000 │ |
|
292 │ /etc/sysctl.d/uhd-usrp2.conf │ net.core.wmem_max │ 1048576 │ |
|
293 │ /etc/sysctl.d/10-console-messages.conf │ kernel.printk │ 4 4 1 7 │ |
|
294 │ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.all.use_tempaddr │ 2 │ |
|
295 │ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.default.use_tempaddr │ 2 │ |
|
296 │ /etc/sysctl.d/10-kernel-hardening.conf │ kernel.kptr_restrict │ 1 │ |
|
297 │ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_hardlinks │ 1 │ |
|
298 │ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_symlinks │ 1 │ |
|
299 │ /etc/sysctl.d/10-magic-sysrq.conf │ kernel.sysrq │ 176 │ |
|
300 │ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.default.rp_filter │ 1 │ |
|
301 │ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.all.rp_filter │ 1 │ |
|
302 │ /etc/sysctl.d/10-network-security.conf │ net.ipv4.tcp_syncookies │ 1 │ |
|
303 │ /etc/sysctl.d/10-ptrace.conf │ kernel.yama.ptrace_scope │ 1 │ |
|
304 │ /etc/sysctl.d/10-zeropage.conf │ vm.mmap_min_addr │ 65536 │ |
|
305 │ /etc/sysctl.d/99-sysctl.conf │ kernel.sysrq │ 1 │ |
|
306 ╰─────────────────────────────────────────┴────────────────────────────────────┴────────────────╯ |
|
307 Record count: 15]]></m:pre> |
|
308 |
|
309 |
|
310 <p>We can also do this:</p> |
|
311 <m:pre jazyk="bash"><![CDATA[find /etc/sysctl.d/ -name '*.conf' \ |
|
312 | while read f; do echo "[$f]"; cat "$f"; done \ |
|
313 | relpipe-in-ini \ |
|
314 | relpipe-out-tabular]]></m:pre> |
|
315 |
|
316 <p>Or create a reusable function:</p> |
|
317 <m:pre jazyk="bash"><![CDATA[list-config() { |
|
318 head -n -0 "$@" \ |
|
319 | sed -E 's/==> (.*) <==/[\1]/g' \ |
|
320 | relpipe-in-ini --relation "configs" \ |
|
321 | relpipe-out-tabular --write-types false; |
|
322 }]]></m:pre> |
|
323 |
|
324 <p>and call it this way:</p> |
|
325 <m:pre jazyk="bash"><![CDATA[list-config /etc/sysctl.d/*.conf]]></m:pre> |
|
326 |
|
327 <p>It is a bit dirty hack (what if the file name, key or value contains <code><==</code> or <code>[…]</code>?) but it will work quite well.</p> |
|
328 |
|
329 <p>And like with Java .properties and MANIFEST-MF, we can use the <code>relpipe-out-ini</code> to generate the configuration files.</p> |
|
330 |
|
331 </text> |
|
332 |
|
333 </stránka> |