|
1 <stránka |
|
2 xmlns="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/strana" |
|
3 xmlns:m="https://trac.frantovo.cz/xml-web-generator/wiki/xmlns/makro"> |
|
4 |
|
5 <nadpis>Exploring X11 windows and devices and emulating mouse movements and keystrokes</nadpis> |
|
6 <perex>connect to X11 server and have some fun</perex> |
|
7 <m:pořadí-příkladu>04900</m:pořadí-příkladu> |
|
8 |
|
9 <text xmlns="http://www.w3.org/1999/xhtml"> |
|
10 |
|
11 <p> |
|
12 The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland). |
|
13 This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and <a href="https://mouse.globalcode.info/v_0/index.xhtml">mice</a>. |
|
14 </p> |
|
15 |
|
16 <p> |
|
17 Since <m:name/> |
|
18 <m:a href="release-v0.18">v0.18</m:a> we can interact with this wonderful technology through the <code>relpipe-in-x11</code> and <code>relpipe-out-x11</code> tools. |
|
19 In this example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements). |
|
20 </p> |
|
21 |
|
22 <h2>Listing windows</h2> |
|
23 |
|
24 <p> |
|
25 In X11 almost everything is a <i>window</i>. Especially classic applications like XCalc are composed from many <i>windows</i>. |
|
26 We list all windows of the current display this way: |
|
27 </p> |
|
28 |
|
29 <m:pre jazyk="bash"><![CDATA[relpipe-in-x11 --list-windows true | relpipe-out-tabular | less -RSi |
|
30 relpipe-in-x11 --list-windows true | relpipe-out-recfile | less]]></m:pre> |
|
31 |
|
32 <p>and realize that there are so many windows (hundreds on a common desktop) with so much metadata (long names, titles etc.).</p> |
|
33 |
|
34 |
|
35 <m:pre jazyk="text"><![CDATA[%rec: x11_window |
|
36 %type: id int |
|
37 %type: root int |
|
38 %type: parent int |
|
39 %type: level int |
|
40 %type: process int |
|
41 %type: x int |
|
42 %type: y int |
|
43 %type: width int |
|
44 %type: height int |
|
45 |
|
46 id: 1254 |
|
47 root: 1254 |
|
48 parent: 0 |
|
49 level: 0 |
|
50 process: -1 |
|
51 name: |
|
52 res_class: |
|
53 res_name: |
|
54 x: 0 |
|
55 y: 0 |
|
56 width: 3840 |
|
57 height: 2160 |
|
58 |
|
59 … |
|
60 |
|
61 id: 125829135 |
|
62 root: 1254 |
|
63 parent: 155189297 |
|
64 level: 3 |
|
65 process: 10046 |
|
66 name: Exploring X11 windows and devices and emulating mouse movements and keystrokes – Relational pipes - Falkon |
|
67 res_class: Falkon |
|
68 res_name: Falkon Browser |
|
69 x: 0 |
|
70 y: 0 |
|
71 width: 3840 |
|
72 height: 2044 |
|
73 |
|
74 …]]></m:pre> |
|
75 |
|
76 <p> |
|
77 Windows are identified by their <code>id</code> and organized hierarchically – have <code>parent</code> attribute that contains ID of their ancestor in the tree. |
|
78 There is also the <code>process</code> attribute that contains the PID (process ID) of the running program that created this window. |
|
79 However we should be aware that this information is voluntarily provided by the X11 clients, not managed by the X11 server, and thus might be missing (quite often) or even misleading (rarely). |
|
80 </p> |
|
81 |
|
82 <p> |
|
83 In order to not get lost in so many windows, we can run a separate X11 server and put only some programs/windows on its display: |
|
84 </p> |
|
85 |
|
86 |
|
87 <m:pre jazyk="bash"><![CDATA[# Run a nested X11 server: |
|
88 Xephyr :8 -screen 640x480 |
|
89 |
|
90 # Run an application on its display: |
|
91 DISPLAY=:8 xcalc |
|
92 |
|
93 # List the windows: |
|
94 DISPLAY=:8 relpipe-in-x11 --list-windows true | relpipe-out-tabular |
|
95 ]]></m:pre> |
|
96 |
|
97 <p>This classic calculator consist of just 51 windows:</p> |
|
98 |
|
99 <m:pre jazyk="text"><![CDATA[x11_window: |
|
100 ╭──────────────┬────────────────┬──────────────────┬─────────────────┬───────────────────┬───────────────┬────────────────────┬───────────────────┬─────────────┬─────────────┬─────────────────┬──────────────────╮ |
|
101 │ id (integer) │ root (integer) │ parent (integer) │ level (integer) │ process (integer) │ name (string) │ res_class (string) │ res_name (string) │ x (integer) │ y (integer) │ width (integer) │ height (integer) │ |
|
102 ├──────────────┼────────────────┼──────────────────┼─────────────────┼───────────────────┼───────────────┼────────────────────┼───────────────────┼─────────────┼─────────────┼─────────────────┼──────────────────┤ |
|
103 │ 681 │ 681 │ 0 │ 0 │ -1 │ │ │ │ 0 │ 0 │ 640 │ 480 │ |
|
104 │ 2097174 │ 681 │ 681 │ 1 │ -1 │ Calculator │ XCalc │ xcalc │ 0 │ 0 │ 226 │ 304 │ |
|
105 │ 2097175 │ 681 │ 2097174 │ 2 │ -1 │ │ │ │ 0 │ 0 │ 226 │ 304 │ |
|
106 │ 2097180 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 272 │ 40 │ 26 │ |
|
107 │ 2097181 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 272 │ 40 │ 26 │ |
|
108 │ 2097182 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 272 │ 40 │ 26 │ |
|
109 │ 2097183 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 272 │ 40 │ 26 │ |
|
110 │ 2097184 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 272 │ 40 │ 26 │ |
|
111 │ 2097185 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 242 │ 40 │ 26 │ |
|
112 │ 2097186 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 242 │ 40 │ 26 │ |
|
113 │ 2097187 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 242 │ 40 │ 26 │ |
|
114 │ 2097188 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 242 │ 40 │ 26 │ |
|
115 │ 2097189 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 242 │ 40 │ 26 │ |
|
116 │ 2097190 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 212 │ 40 │ 26 │ |
|
117 │ 2097191 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 212 │ 40 │ 26 │ |
|
118 │ 2097192 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 212 │ 40 │ 26 │ |
|
119 │ 2097193 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 212 │ 40 │ 26 │ |
|
120 │ 2097194 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 212 │ 40 │ 26 │ |
|
121 │ 2097195 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 182 │ 40 │ 26 │ |
|
122 │ 2097196 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 182 │ 40 │ 26 │ |
|
123 │ 2097197 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 182 │ 40 │ 26 │ |
|
124 │ 2097198 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 182 │ 40 │ 26 │ |
|
125 │ 2097199 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 182 │ 40 │ 26 │ |
|
126 │ 2097200 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 152 │ 40 │ 26 │ |
|
127 │ 2097201 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 152 │ 40 │ 26 │ |
|
128 │ 2097202 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 152 │ 40 │ 26 │ |
|
129 │ 2097203 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 152 │ 40 │ 26 │ |
|
130 │ 2097204 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 152 │ 40 │ 26 │ |
|
131 │ 2097205 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 122 │ 40 │ 26 │ |
|
132 │ 2097206 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 122 │ 40 │ 26 │ |
|
133 │ 2097207 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 122 │ 40 │ 26 │ |
|
134 │ 2097208 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 122 │ 40 │ 26 │ |
|
135 │ 2097209 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 122 │ 40 │ 26 │ |
|
136 │ 2097210 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 92 │ 40 │ 26 │ |
|
137 │ 2097211 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 92 │ 40 │ 26 │ |
|
138 │ 2097212 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 92 │ 40 │ 26 │ |
|
139 │ 2097213 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 92 │ 40 │ 26 │ |
|
140 │ 2097214 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 92 │ 40 │ 26 │ |
|
141 │ 2097215 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 62 │ 40 │ 26 │ |
|
142 │ 2097216 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 62 │ 40 │ 26 │ |
|
143 │ 2097217 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 62 │ 40 │ 26 │ |
|
144 │ 2097218 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 62 │ 40 │ 26 │ |
|
145 │ 2097219 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 62 │ 40 │ 26 │ |
|
146 │ 2097220 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 2 │ 216 │ 46 │ |
|
147 │ 2097221 │ 681 │ 2097220 │ 4 │ -1 │ │ │ │ 6 │ 2 │ 204 │ 38 │ |
|
148 │ 2097222 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 127 │ 21 │ 18 │ 15 │ |
|
149 │ 2097223 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 91 │ 21 │ 34 │ 15 │ |
|
150 │ 2097224 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 61 │ 21 │ 26 │ 15 │ |
|
151 │ 2097225 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 31 │ 21 │ 26 │ 15 │ |
|
152 │ 2097226 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 4 │ 23 │ 26 │ 15 │ |
|
153 │ 2097227 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 18 │ 2 │ 186 │ 17 │ |
|
154 │ 2097228 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 4 │ 2 │ 10 │ 15 │ |
|
155 ╰──────────────┴────────────────┴──────────────────┴─────────────────┴───────────────────┴───────────────┴────────────────────┴───────────────────┴─────────────┴─────────────┴─────────────────┴──────────────────╯ |
|
156 Record count: 52]]></m:pre> |
|
157 |
|
158 |
|
159 <h2>Filtering windows with SQL and recursive CTE</h2> |
|
160 |
|
161 <p> |
|
162 Another way to deal with the abundance of windows is SQL, especially the recursive CTE (common table expressions). |
|
163 This kind of SQL expressions allows us to work with tree structures and filter all descendants of given parent window. |
|
164 </p> |
|
165 |
|
166 |
|
167 <m:pre jazyk="bash"><![CDATA[#!/bin/bash |
|
168 |
|
169 get_x11_child_windows() { |
|
170 SQL=" |
|
171 WITH |
|
172 RECURSIVE w AS ( |
|
173 SELECT * FROM x11_window WHERE name = ? |
|
174 UNION ALL |
|
175 SELECT ch.* FROM x11_window ch JOIN w ON (w.id = ch.parent) |
|
176 ) |
|
177 SELECT * FROM w |
|
178 ORDER BY x, y |
|
179 "; |
|
180 relpipe-in-x11 --list-windows true \ |
|
181 | relpipe-tr-sql \ |
|
182 --relation 'window' "$SQL" \ |
|
183 --parameter "$1" |
|
184 } |
|
185 |
|
186 format_result() { [[ -t 1 ]] && relpipe-out-tabular || cat; } |
|
187 |
|
188 get_x11_child_windows "Calculator" | format_result]]></m:pre> |
|
189 |
|
190 <p> |
|
191 When we run this script on a desktop with many windows and running XCalc, we will get a listing of the only 51 windows – similar to the listing above. |
|
192 The root window (screen) is not included in this case, but we can also add it using another <code>UNION ALL</code>. |
|
193 </p> |
|
194 |
|
195 |
|
196 <p> |
|
197 Thanks to <m:name/> modular design we can use also other transformation tools and languages – e.g. Scheme (<code>relpipe-tr-scheme</code>), AWK (<code>relpipe-tr-awk</code>) or XPath (<code>relpipe-tr-xpath</code>). |
|
198 </p> |
|
199 |
|
200 <h2>Embedding and composing windows</h2> |
|
201 |
|
202 <p> |
|
203 Once we know the ID of a particular window, we can have some fun and run another application inside it: |
|
204 </p> |
|
205 |
|
206 <m:pre jazyk="bash"><![CDATA[/usr/lib/xscreensaver/euphoria --regular -window-id 148897953 |
|
207 mpv --wid=148897935 "film.mkv"]]></m:pre> |
|
208 |
|
209 <p> |
|
210 So we can e.g. watch a film<!--, monitor a live camera--> or enjoy a screen saver while calculating: |
|
211 </p> |
|
212 |
|
213 <m:img src="img/xcalc-x11-embedding-1.png"/> |
|
214 |
|
215 <p>And we can setup everything automatically whithout typing the IDs by hand: </p> |
|
216 |
|
217 <m:pre jazyk="bash"><![CDATA[read_nullbyte() { local IFS=; for v in "$@"; do export "$v"; read -r -d '' "$v"; done } |
|
218 |
|
219 ./the-x11-script-listed-above.sh \ |
|
220 | relpipe-tr-xpath \ |
|
221 --relation '.*' \ |
|
222 --where 'level >= 5' \ |
|
223 --output-attribute 'wid' integer 'id' \ |
|
224 | relpipe-out-nullbyte | while read_nullbyte wid; do |
|
225 sleep 1; |
|
226 /usr/lib/xscreensaver/euphoria --regular -window-id "$wid" & |
|
227 done]]></m:pre> |
|
228 |
|
229 <p>and experience pure euphoria:</p> |
|
230 |
|
231 <m:img src="img/xcalc-x11-embedding-2.png"/> |
|
232 |
|
233 <p> |
|
234 This X11 feature can be used also in much more useful way for composing GUI from several separate programs that might be even written in different programming languages. |
|
235 </p> |
|
236 |
|
237 |
|
238 <h2>Listing input devices</h2> |
|
239 |
|
240 <p>In order to list our keyboards, mice and similar devices we will invoke:</p> |
|
241 |
|
242 <m:pre jazyk="bash"><![CDATA[relpipe-in-x11 | relpipe-out-tabular]]></m:pre> |
|
243 |
|
244 <p>or we can explicitly say <code>--list-input-devices true</code>; in both cases we will get a listing similar to this one:</p> |
|
245 |
|
246 <m:pre jazyk="text"><![CDATA[x11_input_device: |
|
247 ╭──────────────┬───────────────────────────────────────────────────────────┬───────────────╮ |
|
248 │ id (integer) │ name (string) │ type (string) │ |
|
249 ├──────────────┼───────────────────────────────────────────────────────────┼───────────────┤ |
|
250 │ 2 │ Virtual core pointer │ │ |
|
251 │ 3 │ Virtual core keyboard │ │ |
|
252 │ 4 │ Virtual core XTEST pointer │ │ |
|
253 │ 5 │ Virtual core XTEST keyboard │ │ |
|
254 │ 6 │ Power Button │ keyboard │ |
|
255 │ 7 │ Video Bus │ keyboard │ |
|
256 │ 8 │ Power Button │ keyboard │ |
|
257 │ 10 │ Logitech USB Trackball │ mouse │ |
|
258 │ 11 │ ZSA Technology Labs Inc ErgoDox EZ Shine │ keyboard │ |
|
259 │ 12 │ ZSA Technology Labs Inc ErgoDox EZ Shine │ mouse │ |
|
260 │ 13 │ ZSA Technology Labs Inc ErgoDox EZ Shine System Control │ keyboard │ |
|
261 │ 14 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ mouse │ |
|
262 │ 15 │ ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard │ keyboard │ |
|
263 │ 16 │ AT Translated Set 2 keyboard │ keyboard │ |
|
264 │ 17 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ keyboard │ |
|
265 │ 9 │ 3Dconnexion 3Dconnexion Universal Receiver Mouse │ mouse │ |
|
266 ╰──────────────┴───────────────────────────────────────────────────────────┴───────────────╯ |
|
267 Record count: 16]]></m:pre> |
|
268 |
|
269 <p>Device IDs can be used to identify the source of input events and their filtering.</p> |
|
270 |
|
271 <h2>Capturing input events</h2> |
|
272 |
|
273 <p> |
|
274 Once we connect to an X11 server (through the <code>relpipe-in-x11</code> X11 client in our case) |
|
275 we can listen to the input events and monitor the key codes and mouse movements. |
|
276 </p> |
|
277 |
|
278 <m:pre jazyk="bash">relpipe-in-x11 --list-input-events true | relpipe-out-csv</m:pre> |
|
279 |
|
280 <p> |
|
281 To see the events immediatelly, it is important to use an output filter that does no buffering (e.g. <code>relpipe-out-csv</code> or <code>relpipe-out-gui</code>). |
|
282 </p> |
|
283 |
|
284 <m:pre jazyk="text"><![CDATA["device","type","state","key","button","x","y" |
|
285 "11","key","released","36","-1","1405","1416" |
|
286 "11","key","pressed","38","-1","1405","1416" |
|
287 "11","key","released","38","-1","1405","1416" |
|
288 "11","key","pressed","43","-1","1405","1416" |
|
289 "11","key","released","43","-1","1405","1416" |
|
290 "11","key","pressed","32","-1","1405","1416" |
|
291 "11","key","released","32","-1","1405","1416" |
|
292 "11","key","pressed","44","-1","1405","1416" |
|
293 "11","key","released","44","-1","1405","1416" |
|
294 "9","motion",,"-1","-1","1405","1416" |
|
295 "9","motion",,"-1","-1","1405","1417" |
|
296 "9","motion",,"-1","-1","1406","1418" |
|
297 "9","motion",,"-1","-1","1406","1419" |
|
298 "9","button","pressed","-1","1","1406","1420" |
|
299 "9","button","released","-1","1","1406","1420" |
|
300 "9","motion",,"-1","-1","1406","1420" |
|
301 "9","motion",,"-1","-1","1405","1420"]]></m:pre> |
|
302 |
|
303 <p> |
|
304 In order to capture events from a different X11 server, we set the <code>DISPLAY</code> environment variable. |
|
305 This way, we can also capture events remotely over SSH. |
|
306 </p> |
|
307 |
|
308 <m:pre jazyk="bash">ssh example.com DISPLAY=:0 relpipe-in-x11 --list-input-events true | relpipe-out-csv</m:pre> |
|
309 |
|
310 <p> |
|
311 We can log the events to a file or forward them to another X11 server through <code>relpipe-out-x11</code>. |
|
312 Or we can multiplex the events and forward them to several X11 servers e.g. to run simultaneous tests of different versions or configurations or certain GUI application. |
|
313 </p> |
|
314 |
|
315 |
|
316 <h2>Simulating input events</h2> |
|
317 |
|
318 <p> |
|
319 With the <code>relpipe-out-x11</code> client we can connect to an X11 server and emit input events like they come from a keyboard or mouse |
|
320 (actually they would come from a virtual/simulated one). |
|
321 This command reads relational data with same structure as produced by its counterpart <code>relpipe-in-x11</code>. |
|
322 We can use data captured earlier or create some new one using any tool like |
|
323 <code>relpipe-in-cli</code> or other <code>relpipe-in-*</code> optionally transformed through a <code>relpipe-tr-*</code> filter. |
|
324 We may also translate other signals like <m:a href="examples-jack-midi-monitoring">MIDI</m:a> events to X11 ones (<code>relpipe-in-jack</code>). |
|
325 </p> |
|
326 |
|
327 <p> |
|
328 For example, the following command moves the cursor to the coordinates 100,200 and then to 640,480. |
|
329 Such events might be produced also in a shell loop and passed through the pipe to a single <code>relpipe-out-x11</code> process |
|
330 (i.e. without connecting and disconnecting the X11 server on each event). |
|
331 </p> |
|
332 |
|
333 <m:pre jazyk="bash"><![CDATA[relpipe-in-cli \ |
|
334 --relation "x11_input_event" \ |
|
335 --attribute "type" string \ |
|
336 --attribute "x" integer \ |
|
337 --attribute "y" integer \ |
|
338 --records \ |
|
339 "motion" "100" "200" \ |
|
340 "motion" "640" "480" \ |
|
341 | relpipe-out-x11]]></m:pre> |
|
342 |
|
343 <p> |
|
344 The <code>relpipe-out-x11</code> tool has two options: |
|
345 <code>--dry-run true</code> that suppresses the events and just tests validity of the input; and |
|
346 <code>--debug true</code> that prints the events in a text form (XML) on STDERR (so we can see that something comes through). |
|
347 </p> |
|
348 |
|
349 <m:pre jazyk="bash">relpipe-in-x11 --list-input-events true | relpipe-out-x11 --dry-run true --debug true</m:pre> |
|
350 |
|
351 <p> |
|
352 Using SSH, DDC, <code>relpipe-in-x11</code> and <code>relpipe-out-x11</code>, |
|
353 we can build a software replacement of a KVM switch and control several computers from one seat (keyboard, video, mouse). |
|
354 </p> |
|
355 |
|
356 </text> |
|
357 |
|
358 </stránka> |