|
1 <html> |
|
2 <head> |
|
3 <title> |
|
4 Debugging transported core dumps |
|
5 </title> |
|
6 </head> |
|
7 <body> |
|
8 <h1>Debugging transported core dumps</h1> |
|
9 |
|
10 <p> |
|
11 When a core dump is moved to a machine different from the one where it was |
|
12 produced ("transported core dump"), debuggers (dbx, gdb, windbg or SA) do not |
|
13 always successfully open the dump. This is due to kernel, library (shared |
|
14 objects or DLLs) mismatch between core dump machine and debugger machine. |
|
15 </p> |
|
16 |
|
17 <p> |
|
18 In most platforms, core dumps do not contain text (a.k.a) Code pages. |
|
19 There pages are to be read from executable and shared objects (or DLLs). |
|
20 Therefore it is important to have matching executable and shared object |
|
21 files in debugger machine. |
|
22 </p> |
|
23 |
|
24 <h3>Solaris transported core dumps</h3> |
|
25 |
|
26 <p> |
|
27 Debuggers on Solaris (and Linux) use two addtional shared objects |
|
28 <b>rtld_db.so</b> and <b>libthread_db.so</b>. rtld_db.so is used to |
|
29 read information on shared objects from the core dump. libthread_db.so |
|
30 is used to get information on threads from the core dump. rtld_db.so |
|
31 evolves along with rtld.so (the runtime linker library) and libthread_db.so |
|
32 evolves along with libthread.so (user land multithreading library). |
|
33 Hence, debugger machine should have right version of rtld_db.so and |
|
34 libthread_db.so to open the core dump successfully. More details on |
|
35 these debugger libraries can be found in |
|
36 <a href="http://docs.sun.com/app/docs/doc/817-1984/"> |
|
37 Solaris Linkers and Libraries Guide - 817-1984</a> |
|
38 </p> |
|
39 |
|
40 <h3>Solaris SA against transported core dumps</h3> |
|
41 |
|
42 <p> |
|
43 With transported core dumps, you may get "rtld_db failures" or |
|
44 "libthread_db failures" or SA may just throw some other error |
|
45 (hotspot symbol is missing) when opening the core dump. |
|
46 Enviroment variable <b>LIBSAPROC_DEBUG</b> may be set to any value |
|
47 to debug such scenarios. With this env. var set, SA prints many |
|
48 messages in standard error which can be useful for further debugging. |
|
49 SA on Solaris uses <b>libproc.so</b> library. This library also |
|
50 prints debug messages with env. var <b>LIBPROC_DEBUG</b>. But, |
|
51 setting LIBSAPROC_DEBUG results in setting LIBPROC_DEBUG as well. |
|
52 </p> |
|
53 <p> |
|
54 The best possible way to debug a transported core dump is to match the |
|
55 debugger machine to that of core dump machine. i.e., have same Kernel |
|
56 and libthread patch level between the machines. mdb (Solaris modular |
|
57 debugger) may be used to find the Kernel patch level of core dump |
|
58 machine and debugger machine may be brought to the same level. |
|
59 </p> |
|
60 <p> |
|
61 If the matching machine is "far off" in your network, then |
|
62 <ul> |
|
63 <li>consider using rlogin and <a href="clhsdb.html">CLHSDB - SA command line HSDB interface</a> or |
|
64 <li>use SA remote debugging and debug the core from core machine remotely. |
|
65 </ul> |
|
66 </p> |
|
67 |
|
68 <p> |
|
69 But, it may not be feasible to find matching machine to debug. |
|
70 If so, you can copy all application shared objects (and libthread_db.so, if needed) from the core dump |
|
71 machine into your debugger machine's directory, say, /export/applibs. Now, set <b>SA_ALTROOT</b> |
|
72 environment variable to point to /export/applibs directory. Note that /export/applibs should either |
|
73 contain matching 'full path' of libraries. i.e., /usr/lib/libthread_db.so from core |
|
74 machine should be under /export/applibs/use/lib directory and /use/java/jre/lib/sparc/client/libjvm.so |
|
75 from core machine should be under /export/applibs/use/java/jre/lib/sparc/client so on or /export/applibs |
|
76 should just contain libthread_db.so, libjvm.so etc. directly. |
|
77 </p> |
|
78 |
|
79 <p> |
|
80 Support for transported core dumps is <b>not</b> built into the standard version of libproc.so. You need to |
|
81 set <b>LD_LIBRARY_PATH</b> env var to point to the path of a specially built version of libproc.so. |
|
82 Note that this version of libproc.so has a special symbol to support transported core dump debugging. |
|
83 In future, we may get this feature built into standard libproc.so -- if that happens, this step (of |
|
84 setting LD_LIBRARY_PATH) can be skipped. |
|
85 </p> |
|
86 |
|
87 <h3>Ignoring libthread_db.so failures</h3> |
|
88 <p> |
|
89 If you are okay with missing thread related information, you can set |
|
90 <b>SA_IGNORE_THREADDB</b> environment variable to any value. With this |
|
91 set, SA ignores libthread_db failure, but you won't be able to get any |
|
92 thread related information. But, you would be able to use SA and get |
|
93 other information. |
|
94 </p> |
|
95 |
|
96 <h3>Linux SA against transported core dumps</h3> |
|
97 <p> |
|
98 On Linux, SA parses core and shared library ELF files. SA <b>does not</b> use |
|
99 libthread_db.so or rtld_db.so for core dump debugging (although |
|
100 libthread_db.so is used for live process debugging). But, you |
|
101 may still face problems with transported core dumps, because matching shared |
|
102 objects may not be in the path(s) specified in core dump file. To |
|
103 workaround this, you can define environment variable <b>SA_ALTROOT</b> |
|
104 to be the directory where shared libraries are kept. The semantics of |
|
105 this env. variable is same as that for Solaris (please refer above). |
|
106 </p> |
|
107 |
|
108 |
|
109 </body> |
|
110 </html> |