# HG changeset patch # User František Kučera # Date 1564085772 -7200 # Node ID 8bf13358a50affa5eef5ef4937fd18462d630e5a # Parent 846510a73535715c5a418db825dd7f83fafb7b63 examples: relpipe-in-xmltable diff -r 846510a73535 -r 8bf13358a50a relpipe-data/download.xml --- a/relpipe-data/download.xml Wed Jul 24 14:18:42 2019 +0200 +++ b/relpipe-data/download.xml Thu Jul 25 22:16:12 2019 +0200 @@ -20,6 +20,7 @@ hg clone https://hg.globalcode.info/relpipe/relpipe-in-fstab.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-in-recfile.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-in-xml.cpp; +hg clone https://hg.globalcode.info/relpipe/relpipe-in-xmltable.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-lib-cli.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-lib-protocol.cpp; hg clone https://hg.globalcode.info/relpipe/relpipe-lib-reader.cpp; @@ -60,6 +61,7 @@
  • 2019-02-20: v0.10
  • 2019-04-08: v0.11
  • 2019-05-28: v0.12
    • +
  • 2019-07-30: v0.13
    • diff -r 846510a73535 -r 8bf13358a50a relpipe-data/examples-in-xmltable-atom.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-in-xmltable-atom.xml Thu Jul 25 22:16:12 2019 +0200 @@ -0,0 +1,45 @@ + + + Reading an Atom feed using XMLTable + converting arbitrary XML into relational data using XMLTable + 02800 + + + +

    + In this exaple we will achieve the same result as in previous one with XQuery + but we will use a different tool – relpipe-in-xmltable. + This approach differs from XQery one in several aspects: +

    + +
      +
    • no need for writing a script/program – we need just to write an one-liner resp. call one command with parameters
      • +
    • no need for an external tool like Galax, BaseX or XQilla
      • +
    • no serialization to the intermediary XML and its immediate deserialization – the XML input is parsed in the same proces which outputs relational data
      • +
    • simpler (but less powerful) tool – we write only two or more XPath expressions
      • +
    + +

    This is the (shortened) structure of our XML input:

    + + + +

    This pipeline will download the XML data and transform it to two relations:

    + + + +

    The first one contains individual entries and the second one contains the common header:

    + + + +

    + This example shows how to work with namespaces and how to generate multiple relations from a single XML input. + It also shows that the name of the relation do not have to be a literal but might be derived from the input document. +

    + +

    If we add the id attribute to the entries table, we can aggregate entries from various sources and still be able to JOIN them with their metadata.

    + +     + +     diff -r 846510a73535 -r 8bf13358a50a relpipe-data/examples-in-xmltable-libvirt.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-in-xmltable-libvirt.xml Thu Jul 25 22:16:12 2019 +0200 @@ -0,0 +1,139 @@ + + + Reading Libvirt XML files using XMLTable + converting arbitrary XML into one or more relations + 02700 + + + +

    + Libvirt is a popular API/tool for managing virtual machines (KVM/Qemu, LXC etc.) and stores its configuration in XML files. + Thanks to the tool relpipe-in-xmltable we can get aggregated overview of our virtual machines. + This tool does similar job like the xmltable function known from SQL. + It uses the XPath language for selecting parts of the input XML – one XPath expression points to record nodes + and one or more XPath expressions point to attribute nodes/values relatively to particular record node. + Our tool is able to produce one or more relations from a single XML input. + The input is parsed at once and converted to DOM in memory i.e. no streaming – thus processing of huge XML files requires appropriate amounts of RAM, on the other hand: + our expression can access whole XML document and pick values not only from currently processed record node. +

    + + +

    These XML config files contain lot of information describing given virtual machine:

    + + + + + relpipe-1 + 36d1e8b2-97e9-40cb-9fc2-306ebf989282 + 1048576 + 1048576 + 2 + + hvm + + + + + + + + + Opteron_G5 + + + + + + + … + + /usr/bin/kvm-spice + + + + +
    + + + + + +
    + + … + + + + +
    + + … + +]]> + + +

    If we are interested only in certain parts, we can pick them using a command like this one:

    + + + + +

    And get three relations:

    + +
    + +

    + Each record contain ID of the machine, thus if we collect data from several VMs, + we can JOIN relevant records together, do some aggregations or statistics. + If we are sure that the name field is unique, we can use it as a key instead of the UUID. +

    + + + + + diff -r 846510a73535 -r 8bf13358a50a relpipe-data/examples/atom-xmltable.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/atom-xmltable.sh Thu Jul 25 22:16:12 2019 +0200 @@ -0,0 +1,21 @@ +#!/bin/bash + +get_atom() { + wget --quiet --output-document - https://blog.frantovo.cz/agregace/c/?p=10 + # wget --quiet --output-document - https://blog.frantovo.cz/agregace/k/ + # cat atom.xml +} + +get_atom \ + | relpipe-in-xmltable \ + --namespace "a" "http://www.w3.org/2005/Atom" \ + --relation "/a:feed/a:title" --name-is-xpath \ + --records "//a:entry" \ + --attribute "published" string "a:published" \ + --attribute "title" string "a:title" \ + --attribute "url" string "a:link/@href" \ + --relation "metadata" \ + --records "." \ + --attribute "id" string "a:id" \ + --attribute "subtitle" string "a:subtitle" \ + | relpipe-out-tabular diff -r 846510a73535 -r 8bf13358a50a relpipe-data/examples/atom-xmltable.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/atom-xmltable.txt Thu Jul 25 22:16:12 2019 +0200 @@ -0,0 +1,24 @@ +Frantovo.cz – články: + ╭──────────────────────┬──────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ + │ published (string) │ title (string) │ url (string) │ + ├──────────────────────┼──────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ + │ 2019-07-12T08:12:00Z │ OpenPOWER – Blackbird – první spuštění │ https://blog.frantovo.cz/c/375/OpenPOWER%20%E2%80%93%20Blackbird%20%E2%80%93%20prvn%C3%AD%20spu%C5%A1t%C4%9Bn%C3%AD │ + │ 2019-07-11T17:17:24Z │ Siemens Nixdorf: ComfoDesk (1990) │ https://blog.frantovo.cz/c/374/Siemens%20Nixdorf%3A%20ComfoDesk%20%281990%29 │ + │ 2019-07-04T18:12:02Z │ Opravujeme myš: výměna spínačů │ https://blog.frantovo.cz/c/373/Opravujeme%20my%C5%A1%3A%20v%C3%BDm%C4%9Bna%20sp%C3%ADna%C4%8D%C5%AF │ + │ 2019-06-04T16:32:08Z │ Java a unixové doménové sokety, FD, systemd a xinetd │ https://blog.frantovo.cz/c/372/Java%20a%C2%A0unixov%C3%A9%20dom%C3%A9nov%C3%A9%20sokety%2C%20FD%2C%20systemd%20a%C2%A0xinetd │ + │ 2019-04-26T19:48:00Z │ Zálohujeme internet: Zdrojové kódy │ https://blog.frantovo.cz/c/371/Z%C3%A1lohujeme%20internet%3A%20Zdrojov%C3%A9%20k%C3%B3dy │ + │ 2018-12-24T13:37:24Z │ GNU Bash: Vánoční tipy │ https://blog.frantovo.cz/c/370/GNU%20Bash%3A%20V%C3%A1no%C4%8Dn%C3%AD%20tipy │ + │ 2018-08-04T23:23:00Z │ HiFive1 – deska s otevřeným čipem RISC-V │ https://blog.frantovo.cz/c/368/HiFive1%20%E2%80%93%20deska%20s%C2%A0otev%C5%99en%C3%BDm%20%C4%8Dipem%20RISC-V │ + │ 2018-06-30T13:37:08Z │ The Things Network – LoRaWAN – IoT │ https://blog.frantovo.cz/c/366/The%20Things%20Network%20%E2%80%93%20LoRaWAN%20%E2%80%93%C2%A0IoT │ + │ 2018-03-31T19:48:00Z │ Roland Rubix44 – externí zvuková karta │ https://blog.frantovo.cz/c/365/Roland%20Rubix44%20%E2%80%93%20extern%C3%AD%20zvukov%C3%A1%20karta │ + │ 2017-11-25T20:26:49Z │ Přepisování parametrů příkazové řádky │ https://blog.frantovo.cz/c/362/P%C5%99episov%C3%A1n%C3%AD%20parametr%C5%AF%20p%C5%99%C3%ADkazov%C3%A9%20%C5%99%C3%A1dky │ + ╰──────────────────────┴──────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +Record count: 10 + +metadata: + ╭───────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────╮ + │ id (string) │ subtitle (string) │ + ├───────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ + │ urn:uuid:d61ea960-3b36-11e3-bdb9-3085a98fdb88 │ Blog nejen o svobodném softwaru, GNU/Linuxu, Javě, XML, politice, ekonomii, filosofii │ + ╰───────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────╯ +Record count: 1 diff -r 846510a73535 -r 8bf13358a50a relpipe-data/examples/atom-xmltable.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples/atom-xmltable.xml Thu Jul 25 22:16:12 2019 +0200 @@ -0,0 +1,69 @@ + + + Frantovo.cz – články + Blog nejen o svobodném softwaru, GNU/Linuxu, Javě, XML, politice, ekonomii, filosofii + urn:uuid:d61ea960-3b36-11e3-bdb9-3085a98fdb88 + + + 2019-07-12T08:12:00Z + + František Kučera + + + OpenPOWER – Blackbird – první spuštění + + https://blog.frantovo.cz/c/375/ + 2019-07-12T08:12:00Z + 2019-07-12T08:12:00Z + +
    +
    +

    Po včerejším retro článku tu máme návrat do budoucnosti – ze které se ale naštěstí už stává současnost. Zatím jsem udělal jen pár fotek… Ohledně motivace a smyslu doporučuji si přečíst HiFive1 – deska s otevřeným čipem RISC-V.

    +

    + + OpenPOWER CPU a deska Blackbird + +

    +
    +
    +     +     + + Siemens Nixdorf: ComfoDesk (1990) + + https://blog.frantovo.cz/c/374/ + 2019-07-11T20:08:59Z + 2019-07-11T17:17:24Z + +
    +
    +

    Objevil jsem doma staré diskety se softwarem ComfoDesk od firmy Siemens Nixdorf. Zajímalo mne, jestli budou ještě fungovat a jestli je ten software tak hrozný jako tehdy (vzpomínky na to nejsou moc dobré – nikdo u nás s tím neuměl pracovat a nakonec jsme museli přeinstalovat systém, abychom se ComfoDesku zbavili). Navíc je na internetu minimum zmínek o tomto softwaru, takže jsem se rozhodl ho alespoň trochu zdokumentovat pro příští generace.

    +

    + + Siemens Nixdorf: ComfoDesk – diskety + +

    +
    +
    +     +     + + Opravujeme myš: výměna spínačů + + https://blog.frantovo.cz/c/373/ + 2019-07-04T22:39:39Z + 2019-07-04T18:12:02Z + +
    +
    +

    Po letech dobré služby mne začaly zlobit myši. A protože nerad vyhazuji věci a byly to kvalitní kousky hardwaru, pustil jsem se do opravy.

    +

    + + myš resp. trackball Logitech TrackMan Marble (T-BC21) + +

    +
    +
    +     +     +     diff -r 846510a73535 -r 8bf13358a50a relpipe-data/implementation.xml --- a/relpipe-data/implementation.xml Wed Jul 24 14:18:42 2019 +0200 +++ b/relpipe-data/implementation.xml Thu Jul 25 22:16:12 2019 +0200 @@ -20,6 +20,7 @@ relpipe-in-fstab.cpp executable input c++ GNU GPLv3+ relpipe-in-recfile.cpp executable input c++ GNU GPLv3+ relpipe-in-xml.cpp executable input c++ GNU GPLv3+ + relpipe-in-xmltable.cpp executable input c++ GNU GPLv3+ relpipe-lib-cli.cpp library header-only c++ GNU GPLv3+ relpipe-lib-protocol.cpp library header-only c++ GNU LGPLv3+ or GPLv2+ relpipe-lib-reader.cpp library shared c++ GNU LGPLv3+ or GPLv2+