<stránka
xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana"
xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro">
<nadpis>Exploring X11 windows and devices and emulating mouse movements and keystrokes</nadpis>
<perex>connect to X11 server and have some fun</perex>
<m:pořadí-příkladu>04900</m:pořadí-příkladu>
<text xmlns="http://www.w3.org/1999/xhtml">
<p>
The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland).
This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and <a href="https://mouse.globalcode.info/v_0/index.xhtml">mice</a>.
</p>
<p>
Since <m:name/>
<m:a href="release-v0.18">v0.18</m:a> we can interact with this wonderful technology through the <code>relpipe-in-x11</code> and <code>relpipe-out-x11</code> tools.
In this example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements).
</p>
<h2>Listing windows</h2>
<p>
In X11 almost everything is a <i>window</i>. Especially classic applications like XCalc are composed from many <i>windows</i>.
We list all windows of the current display this way:
</p>
<m:pre jazyk="bash"><![CDATA[relpipe-in-x11 --list-windows true | relpipe-out-tabular | less -RSi
relpipe-in-x11 --list-windows true | relpipe-out-recfile | less]]></m:pre>
<p>and realize that there are so many windows (hundreds on a common desktop) with so much metadata (long names, titles etc.).</p>
<m:pre jazyk="text"><![CDATA[%rec: x11_window
%type: id int
%type: root int
%type: parent int
%type: level int
%type: process int
%type: x int
%type: y int
%type: width int
%type: height int
id: 1254
root: 1254
parent: 0
level: 0
process: -1
name:
res_class:
res_name:
x: 0
y: 0
width: 3840
height: 2160
…
id: 125829135
root: 1254
parent: 155189297
level: 3
process: 10046
name: Exploring X11 windows and devices and emulating mouse movements and keystrokes – Relational pipes - Falkon
res_class: Falkon
res_name: Falkon Browser
x: 0
y: 0
width: 3840
height: 2044
…]]></m:pre>
<p>
Windows are identified by their <code>id</code> and organized hierarchically – have <code>parent</code> attribute that contains ID of their ancestor in the tree.
There is also the <code>process</code> attribute that contains the PID (process ID) of the running program that created this window.
However we should be aware that this information is voluntarily provided by the X11 clients, not managed by the X11 server, and thus might be missing (quite often) or even misleading (rarely).
</p>
<p>
In order to not get lost in so many windows, we can run a separate X11 server and put only some programs/windows on its display:
</p>
<m:pre jazyk="bash"><![CDATA[# Run a nested X11 server:
Xephyr :8 -screen 640x480
# Run an application on its display:
DISPLAY=:8 xcalc
# List the windows:
DISPLAY=:8 relpipe-in-x11 --list-windows true | relpipe-out-tabular
]]></m:pre>
<p>This classic calculator consist of just 51 windows:</p>
<m:pre jazyk="text"><![CDATA[x11_window:
╭──────────────┬────────────────┬──────────────────┬─────────────────┬───────────────────┬───────────────┬────────────────────┬───────────────────┬─────────────┬─────────────┬─────────────────┬──────────────────╮
│ id (integer) │ root (integer) │ parent (integer) │ level (integer) │ process (integer) │ name (string) │ res_class (string) │ res_name (string) │ x (integer) │ y (integer) │ width (integer) │ height (integer) │
├──────────────┼────────────────┼──────────────────┼─────────────────┼───────────────────┼───────────────┼────────────────────┼───────────────────┼─────────────┼─────────────┼─────────────────┼──────────────────┤
│ 681 │ 681 │ 0 │ 0 │ -1 │ │ │ │ 0 │ 0 │ 640 │ 480 │
│ 2097174 │ 681 │ 681 │ 1 │ -1 │ Calculator │ XCalc │ xcalc │ 0 │ 0 │ 226 │ 304 │
│ 2097175 │ 681 │ 2097174 │ 2 │ -1 │ │ │ │ 0 │ 0 │ 226 │ 304 │
│ 2097180 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 272 │ 40 │ 26 │
│ 2097181 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 272 │ 40 │ 26 │
│ 2097182 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 272 │ 40 │ 26 │
│ 2097183 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 272 │ 40 │ 26 │
│ 2097184 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 272 │ 40 │ 26 │
│ 2097185 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 242 │ 40 │ 26 │
│ 2097186 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 242 │ 40 │ 26 │
│ 2097187 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 242 │ 40 │ 26 │
│ 2097188 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 242 │ 40 │ 26 │
│ 2097189 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 242 │ 40 │ 26 │
│ 2097190 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 212 │ 40 │ 26 │
│ 2097191 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 212 │ 40 │ 26 │
│ 2097192 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 212 │ 40 │ 26 │
│ 2097193 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 212 │ 40 │ 26 │
│ 2097194 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 212 │ 40 │ 26 │
│ 2097195 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 182 │ 40 │ 26 │
│ 2097196 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 182 │ 40 │ 26 │
│ 2097197 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 182 │ 40 │ 26 │
│ 2097198 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 182 │ 40 │ 26 │
│ 2097199 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 182 │ 40 │ 26 │
│ 2097200 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 152 │ 40 │ 26 │
│ 2097201 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 152 │ 40 │ 26 │
│ 2097202 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 152 │ 40 │ 26 │
│ 2097203 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 152 │ 40 │ 26 │
│ 2097204 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 152 │ 40 │ 26 │
│ 2097205 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 122 │ 40 │ 26 │
│ 2097206 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 122 │ 40 │ 26 │
│ 2097207 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 122 │ 40 │ 26 │
│ 2097208 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 122 │ 40 │ 26 │
│ 2097209 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 122 │ 40 │ 26 │
│ 2097210 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 92 │ 40 │ 26 │
│ 2097211 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 92 │ 40 │ 26 │
│ 2097212 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 92 │ 40 │ 26 │
│ 2097213 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 92 │ 40 │ 26 │
│ 2097214 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 92 │ 40 │ 26 │
│ 2097215 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 62 │ 40 │ 26 │
│ 2097216 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 62 │ 40 │ 26 │
│ 2097217 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 62 │ 40 │ 26 │
│ 2097218 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 62 │ 40 │ 26 │
│ 2097219 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 62 │ 40 │ 26 │
│ 2097220 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 2 │ 216 │ 46 │
│ 2097221 │ 681 │ 2097220 │ 4 │ -1 │ │ │ │ 6 │ 2 │ 204 │ 38 │
│ 2097222 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 127 │ 21 │ 18 │ 15 │
│ 2097223 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 91 │ 21 │ 34 │ 15 │
│ 2097224 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 61 │ 21 │ 26 │ 15 │
│ 2097225 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 31 │ 21 │ 26 │ 15 │
│ 2097226 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 4 │ 23 │ 26 │ 15 │
│ 2097227 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 18 │ 2 │ 186 │ 17 │
│ 2097228 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 4 │ 2 │ 10 │ 15 │
╰──────────────┴────────────────┴──────────────────┴─────────────────┴───────────────────┴───────────────┴────────────────────┴───────────────────┴─────────────┴─────────────┴─────────────────┴──────────────────╯
Record count: 52]]></m:pre>
<h2>Filtering windows with SQL and recursive CTE</h2>
<p>
Another way to deal with the abundance of windows is SQL, especially the recursive CTE (common table expressions).
This kind of SQL expressions allows us to work with tree structures and filter all descendants of given parent window.
</p>
<m:pre jazyk="bash"><![CDATA[#!/bin/bash
get_x11_child_windows() {
SQL="
WITH
RECURSIVE w AS (
SELECT * FROM x11_window WHERE name = ?
UNION ALL
SELECT ch.* FROM x11_window ch JOIN w ON (w.id = ch.parent)
)
SELECT * FROM w
ORDER BY x, y
";
relpipe-in-x11 --list-windows true \
| relpipe-tr-sql \
--relation 'window' "$SQL" \
--parameter "$1"
}
format_result() { [[ -t 1 ]] && relpipe-out-tabular || cat; }
get_x11_child_windows "Calculator" | format_result]]></m:pre>
<p>
When we run this script on a desktop with many windows and running XCalc, we will get a listing of the only 51 windows – similar to the listing above.
The root window (screen) is not included in this case, but we can also add it using another <code>UNION ALL</code>.
</p>
<p>
Thanks to <m:name/> modular design we can use also other transformation tools and languages – e.g. Scheme (<code>relpipe-tr-scheme</code>), AWK (<code>relpipe-tr-awk</code>) or XPath (<code>relpipe-tr-xpath</code>).
</p>
<h2>Embedding and composing windows</h2>
<p>
Once we know the ID of a particular window, we can have some fun and run another application inside it:
</p>
<m:pre jazyk="bash"><![CDATA[/usr/lib/xscreensaver/euphoria --regular -window-id 148897953
mpv --wid=148897935 "film.mkv"]]></m:pre>
<p>
So we can e.g. watch a film<!--, monitor a live camera--> or enjoy a screen saver while calculating:
</p>
<m:img src="img/xcalc-x11-embedding-1.png"/>
<p>And we can setup everything automatically whithout typing the IDs by hand: </p>
<m:pre jazyk="bash"><![CDATA[read_nullbyte() { local IFS=; for v in "$@"; do export "$v"; read -r -d '' "$v"; done }
./the-x11-script-listed-above.sh \
| relpipe-tr-xpath \
--relation '.*' \
--where 'level >= 5' \
--output-attribute 'wid' integer 'id' \
| relpipe-out-nullbyte | while read_nullbyte wid; do
sleep 1;
/usr/lib/xscreensaver/euphoria --regular -window-id "$wid" &
done]]></m:pre>
<p>and experience pure euphoria:</p>
<m:img src="img/xcalc-x11-embedding-2.png"/>
<p>
This X11 feature can be used also in much more useful way for composing GUI from several separate programs that might be even written in different programming languages.
</p>
<h2>Listing input devices</h2>
<p>In order to list our keyboards, mice and similar devices we will invoke:</p>
<m:pre jazyk="bash"><![CDATA[relpipe-in-x11 | relpipe-out-tabular]]></m:pre>
<p>or we can explicitly say <code>--list-input-devices true</code>; in both cases we will get a listing similar to this one:</p>
<m:pre jazyk="text"><![CDATA[x11_input_device:
╭──────────────┬───────────────────────────────────────────────────────────┬───────────────╮
│ id (integer) │ name (string) │ type (string) │
├──────────────┼───────────────────────────────────────────────────────────┼───────────────┤
│ 2 │ Virtual core pointer │ │
│ 3 │ Virtual core keyboard │ │
│ 4 │ Virtual core XTEST pointer │ │
│ 5 │ Virtual core XTEST keyboard │ │
│ 6 │ Power Button │ keyboard │
│ 7 │ Video Bus │ keyboard │
│ 8 │ Power Button │ keyboard │
│ 10 │ Logitech USB Trackball │ mouse │
│ 11 │ ZSA Technology Labs Inc ErgoDox EZ Shine │ keyboard │
│ 12 │ ZSA Technology Labs Inc ErgoDox EZ Shine │ mouse │
│ 13 │ ZSA Technology Labs Inc ErgoDox EZ Shine System Control │ keyboard │
│ 14 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ mouse │
│ 15 │ ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard │ keyboard │
│ 16 │ AT Translated Set 2 keyboard │ keyboard │
│ 17 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ keyboard │
│ 9 │ 3Dconnexion 3Dconnexion Universal Receiver Mouse │ mouse │
╰──────────────┴───────────────────────────────────────────────────────────┴───────────────╯
Record count: 16]]></m:pre>
<p>Device IDs can be used to identify the source of input events and their filtering.</p>
<h2>Capturing input events</h2>
<p>
Once we connect to an X11 server (through the <code>relpipe-in-x11</code> X11 client in our case)
we can listen to the input events and monitor the key codes and mouse movements.
</p>
<m:pre jazyk="bash">relpipe-in-x11 --list-input-events true | relpipe-out-csv</m:pre>
<p>
To see the events immediatelly, it is important to use an output filter that does no buffering (e.g. <code>relpipe-out-csv</code> or <code>relpipe-out-gui</code>).
</p>
<m:pre jazyk="text"><![CDATA["device","type","state","key","button","x","y"
"11","key","released","36","-1","1405","1416"
"11","key","pressed","38","-1","1405","1416"
"11","key","released","38","-1","1405","1416"
"11","key","pressed","43","-1","1405","1416"
"11","key","released","43","-1","1405","1416"
"11","key","pressed","32","-1","1405","1416"
"11","key","released","32","-1","1405","1416"
"11","key","pressed","44","-1","1405","1416"
"11","key","released","44","-1","1405","1416"
"9","motion",,"-1","-1","1405","1416"
"9","motion",,"-1","-1","1405","1417"
"9","motion",,"-1","-1","1406","1418"
"9","motion",,"-1","-1","1406","1419"
"9","button","pressed","-1","1","1406","1420"
"9","button","released","-1","1","1406","1420"
"9","motion",,"-1","-1","1406","1420"
"9","motion",,"-1","-1","1405","1420"]]></m:pre>
<p>
In order to capture events from a different X11 server, we set the <code>DISPLAY</code> environment variable.
This way, we can also capture events remotely over SSH.
</p>
<m:pre jazyk="bash">ssh example.com DISPLAY=:0 relpipe-in-x11 --list-input-events true | relpipe-out-csv</m:pre>
<p>
We can log the events to a file or forward them to another X11 server through <code>relpipe-out-x11</code>.
Or we can multiplex the events and forward them to several X11 servers e.g. to run simultaneous tests of different versions or configurations or certain GUI application.
</p>
<h2>Simulating input events</h2>
<p>
With the <code>relpipe-out-x11</code> client we can connect to an X11 server and emit input events like they come from a keyboard or mouse
(actually they would come from a virtual/simulated one).
This command reads relational data with same structure as produced by its counterpart <code>relpipe-in-x11</code>.
We can use data captured earlier or create some new one using any tool like
<code>relpipe-in-cli</code> or other <code>relpipe-in-*</code> optionally transformed through a <code>relpipe-tr-*</code> filter.
We may also translate other signals like <m:a href="examples-jack-midi-monitoring">MIDI</m:a> events to X11 ones (<code>relpipe-in-jack</code>).
</p>
<p>
For example, the following command moves the cursor to the coordinates 100,200 and then to 640,480.
Such events might be produced also in a shell loop and passed through the pipe to a single <code>relpipe-out-x11</code> process
(i.e. without connecting and disconnecting the X11 server on each event).
</p>
<m:pre jazyk="bash"><![CDATA[relpipe-in-cli \
--relation "x11_input_event" \
--attribute "type" string \
--attribute "x" integer \
--attribute "y" integer \
--records \
"motion" "100" "200" \
"motion" "640" "480" \
| relpipe-out-x11]]></m:pre>
<p>
The <code>relpipe-out-x11</code> tool has two options:
<code>--dry-run true</code> that suppresses the events and just tests validity of the input; and
<code>--debug true</code> that prints the events in a text form (XML) on STDERR (so we can see that something comes through).
</p>
<m:pre jazyk="bash">relpipe-in-x11 --list-input-events true | relpipe-out-x11 --dry-run true --debug true</m:pre>
<p>
Using SSH, DDC, <code>relpipe-in-x11</code> and <code>relpipe-out-x11</code>,
we can build a software replacement of a KVM switch and control several computers from one seat (keyboard, video, mouse).
</p>
</text>
</stránka>