relpipe-data/examples-awk-aggregations.xml
author František Kučera <franta-hg@frantovo.cz>
Mon, 21 Feb 2022 00:43:11 +0100
branchv_0
changeset 329 5bc2bb8b7946
parent 258 2868d772c27e
permissions -rw-r--r--
Release v0.18
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
258
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     1
<stránka
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     2
	xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     3
	xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     4
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     5
	<nadpis>Aggregating data with AWK</nadpis>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     6
	<perex>counting records and computing sum or appending new records</perex>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     7
	<m:pořadí-příkladu>02600</m:pořadí-příkladu>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     8
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
     9
	<text xmlns="http://www.w3.org/1999/xhtml">
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    10
		
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    11
		<p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    12
			We have filtered records, modified attribute values, added and removed attributes, dropped a relation… 
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    13
			and there is one more operation that we can do with AWK: <code>INSERT</code> resp. appending or preppending additional records to the relation
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    14
			– and we can also completely replace the record set by skipping the original records.
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    15
		</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    16
		
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    17
		<h2>Adding records</h2>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    18
		
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    19
		<p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    20
			Using options <code>--before-records</code> and <code>--after-records</code> we can pass additional AWK code that will be executed – once for given relation.
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    21
			The <code>record()</code> function will then generate an additional record (can be called multiple times and generate more records):
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    22
		</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    23
		
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    24
		<m:pre jazyk="bash"><![CDATA[awkCode='
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    25
	scheme = "";
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    26
	device = "/dev/vg1/volume123";
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    27
	mount_point = "/mnt/v123";
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    28
	type = "btrfs";
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    29
	options = "relatime";
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    30
	dump = 0;
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    31
	pass = 2;
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    32
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    33
	record();
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    34
';
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    35
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    36
relpipe-in-fstab \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    37
	| relpipe-tr-awk \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    38
		--relation '.*' \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    39
			--for-each '1' \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    40
			--after-records "$awkCode" \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    41
	| relpipe-out-tabular]]></m:pre>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    42
	
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    43
		<p>Which will <code>INSERT</code> one new record:</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    44
	
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    45
		<pre><![CDATA[fstab:
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    46
 ╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬───────────────────────────────────────┬────────────────┬────────────────╮
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    47
 │ scheme (string) │ device                      (string) │ mount_point (string) │ type (string) │ options                      (string) │ dump (integer) │ pass (integer) │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    48
 ├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼───────────────────────────────────────┼────────────────┼────────────────┤
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    49
 │ UUID            │ 29758270-fd25-4a6c-a7bb-9a18302816af │ /                    │ ext4          │ relatime,user_xattr,errors=remount-ro │              0 │              1 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    50
 │                 │ /dev/sr0                             │ /media/cdrom0        │ udf,iso9660   │ user,noauto                           │              0 │              0 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    51
 │                 │ /dev/sde                             │ /mnt/data            │ ext4          │ relatime,user_xattr,errors=remount-ro │              0 │              2 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    52
 │ UUID            │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home                │ btrfs         │ relatime                              │              0 │              2 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    53
 │                 │ /dev/mapper/sdf_crypt                │ /mnt/private         │ xfs           │ relatime                              │              0 │              2 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    54
 │                 │ /dev/vg1/volume123                   │ /mnt/v123            │ btrfs         │ relatime                              │              0 │              2 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    55
 ╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────┴────────────────┴────────────────╯
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    56
Record count: 6]]></pre>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    57
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    58
		<h2>Counting and summarizing values</h2>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    59
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    60
		<p>We can also compute some statistics like <code>COUNT()</code> and <code>SUM()</code>:</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    61
		
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    62
		<m:pre jazyk="bash"><![CDATA[find -print0 | relpipe-in-filesystem \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    63
	| relpipe-tr-awk \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    64
		--relation '.*' \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    65
			--before-records 'count = 0; total_size = 0;' \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    66
			--for-each       '{ count++; total_size += size; }' \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    67
			--after-records  'record();' \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    68
			--output-attribute count      integer \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    69
			--output-attribute total_size integer \
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    70
	| relpipe-out-tabular]]></m:pre>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    71
	
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    72
		<p>and get result:</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    73
	
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    74
		<pre><![CDATA[filesystem:
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    75
 ╭─────────────────┬──────────────────────╮
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    76
 │ count (integer) │ total_size (integer) │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    77
 ├─────────────────┼──────────────────────┤
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    78
 │               9 │               818747 │
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    79
 ╰─────────────────┴──────────────────────╯
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    80
Record count: 1]]></pre>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    81
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    82
		<p>Where the <code>total_size</code> is the same as will <code>du</code> compute:</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    83
		
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    84
		<pre>find . -type f -print0 | du -b -c --files0-from=-</pre>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    85
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    86
		<p>Analogously we can compute minimum, maximum etc. using AWK transformation.</p>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    87
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    88
	</text>
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    89
	
2868d772c27e Release v0.12 – AWK
František Kučera <franta-hg@frantovo.cz>
parents:
diff changeset
    90
</stránka>