diff -r cc60c8dd7924 -r 5bc2bb8b7946 relpipe-data/examples-x11-basics.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/relpipe-data/examples-x11-basics.xml Mon Feb 21 00:43:11 2022 +0100 @@ -0,0 +1,358 @@ + + + Exploring X11 windows and devices and emulating mouse movements and keystrokes + connect to X11 server and have some fun + 04900 + + + +

+ The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland). + This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice. +

+ +

+ Since + v0.18 we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools. + In this example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements). +

+ +

Listing windows

+ +

+ In X11 almost everything is a window. Especially classic applications like XCalc are composed from many windows. + We list all windows of the current display this way: +

+ + + +

and realize that there are so many windows (hundreds on a common desktop) with so much metadata (long names, titles etc.).

+ + + + +

+ Windows are identified by their id and organized hierarchically – have parent attribute that contains ID of their ancestor in the tree. + There is also the process attribute that contains the PID (process ID) of the running program that created this window. + However we should be aware that this information is voluntarily provided by the X11 clients, not managed by the X11 server, and thus might be missing (quite often) or even misleading (rarely). +

+ +

+ In order to not get lost in so many windows, we can run a separate X11 server and put only some programs/windows on its display: +

+ + + + +

This classic calculator consist of just 51 windows:

+ + + + +

Filtering windows with SQL and recursive CTE

+ +

+ Another way to deal with the abundance of windows is SQL, especially the recursive CTE (common table expressions). + This kind of SQL expressions allows us to work with tree structures and filter all descendants of given parent window. +

+ + + + +

+ When we run this script on a desktop with many windows and running XCalc, we will get a listing of the only 51 windows – similar to the listing above. + The root window (screen) is not included in this case, but we can also add it using another UNION ALL. +

+ + +

+ Thanks to modular design we can use also other transformation tools and languages – e.g. Scheme (relpipe-tr-scheme), AWK (relpipe-tr-awk) or XPath (relpipe-tr-xpath). +

+ +

Embedding and composing windows

+ +

+ Once we know the ID of a particular window, we can have some fun and run another application inside it: +

+ + + +

+ So we can e.g. watch a film or enjoy a screen saver while calculating: +

+ + + +

And we can setup everything automatically whithout typing the IDs by hand:

+ + = 5' \ + --output-attribute 'wid' integer 'id' \ + | relpipe-out-nullbyte | while read_nullbyte wid; do + sleep 1; + /usr/lib/xscreensaver/euphoria --regular -window-id "$wid" & + done]]> + +

and experience pure euphoria:

+ + + +

+ This X11 feature can be used also in much more useful way for composing GUI from several separate programs that might be even written in different programming languages. +

+ + +

Listing input devices

+ +

In order to list our keyboards, mice and similar devices we will invoke:

+ + + +

or we can explicitly say --list-input-devices true; in both cases we will get a listing similar to this one:

+ + + +

Device IDs can be used to identify the source of input events and their filtering.

+ +

Capturing input events

+ +

+ Once we connect to an X11 server (through the relpipe-in-x11 X11 client in our case) + we can listen to the input events and monitor the key codes and mouse movements. +

+ + relpipe-in-x11 --list-input-events true | relpipe-out-csv + +

+ To see the events immediatelly, it is important to use an output filter that does no buffering (e.g. relpipe-out-csv or relpipe-out-gui). +

+ + + +

+ In order to capture events from a different X11 server, we set the DISPLAY environment variable. + This way, we can also capture events remotely over SSH. +

+ + ssh example.com DISPLAY=:0 relpipe-in-x11 --list-input-events true | relpipe-out-csv + +

+ We can log the events to a file or forward them to another X11 server through relpipe-out-x11. + Or we can multiplex the events and forward them to several X11 servers e.g. to run simultaneous tests of different versions or configurations or certain GUI application. +

+ + +

Simulating input events

+ +

+ With the relpipe-out-x11 client we can connect to an X11 server and emit input events like they come from a keyboard or mouse + (actually they would come from a virtual/simulated one). + This command reads relational data with same structure as produced by its counterpart relpipe-in-x11. + We can use data captured earlier or create some new one using any tool like + relpipe-in-cli or other relpipe-in-* optionally transformed through a relpipe-tr-* filter. + We may also translate other signals like MIDI events to X11 ones (relpipe-in-jack). +

+ +

+ For example, the following command moves the cursor to the coordinates 100,200 and then to 640,480. + Such events might be produced also in a shell loop and passed through the pipe to a single relpipe-out-x11 process + (i.e. without connecting and disconnecting the X11 server on each event). +

+ + + +

+ The relpipe-out-x11 tool has two options: + --dry-run true that suppresses the events and just tests validity of the input; and + --debug true that prints the events in a text form (XML) on STDERR (so we can see that something comes through). +

+ + relpipe-in-x11 --list-input-events true | relpipe-out-x11 --dry-run true --debug true + +

+ Using SSH, DDC, relpipe-in-x11 and relpipe-out-x11, + we can build a software replacement of a KVM switch and control several computers from one seat (keyboard, video, mouse). +

+ + + +