1
+ − 1 <html>
+ − 2 <head>
+ − 3 <title>
+ − 4 Debugging transported core dumps
+ − 5 </title>
+ − 6 </head>
+ − 7 <body>
+ − 8 <h1>Debugging transported core dumps</h1>
+ − 9
+ − 10 <p>
+ − 11 When a core dump is moved to a machine different from the one where it was
+ − 12 produced ("transported core dump"), debuggers (dbx, gdb, windbg or SA) do not
+ − 13 always successfully open the dump. This is due to kernel, library (shared
+ − 14 objects or DLLs) mismatch between core dump machine and debugger machine.
+ − 15 </p>
+ − 16
+ − 17 <p>
+ − 18 In most platforms, core dumps do not contain text (a.k.a) Code pages.
+ − 19 There pages are to be read from executable and shared objects (or DLLs).
+ − 20 Therefore it is important to have matching executable and shared object
+ − 21 files in debugger machine.
+ − 22 </p>
+ − 23
+ − 24 <h3>Solaris transported core dumps</h3>
+ − 25
+ − 26 <p>
+ − 27 Debuggers on Solaris (and Linux) use two addtional shared objects
+ − 28 <b>rtld_db.so</b> and <b>libthread_db.so</b>. rtld_db.so is used to
+ − 29 read information on shared objects from the core dump. libthread_db.so
+ − 30 is used to get information on threads from the core dump. rtld_db.so
+ − 31 evolves along with rtld.so (the runtime linker library) and libthread_db.so
+ − 32 evolves along with libthread.so (user land multithreading library).
+ − 33 Hence, debugger machine should have right version of rtld_db.so and
+ − 34 libthread_db.so to open the core dump successfully. More details on
+ − 35 these debugger libraries can be found in
+ − 36 <a href="http://docs.sun.com/app/docs/doc/817-1984/">
+ − 37 Solaris Linkers and Libraries Guide - 817-1984</a>
+ − 38 </p>
+ − 39
+ − 40 <h3>Solaris SA against transported core dumps</h3>
+ − 41
+ − 42 <p>
+ − 43 With transported core dumps, you may get "rtld_db failures" or
+ − 44 "libthread_db failures" or SA may just throw some other error
+ − 45 (hotspot symbol is missing) when opening the core dump.
+ − 46 Enviroment variable <b>LIBSAPROC_DEBUG</b> may be set to any value
+ − 47 to debug such scenarios. With this env. var set, SA prints many
+ − 48 messages in standard error which can be useful for further debugging.
+ − 49 SA on Solaris uses <b>libproc.so</b> library. This library also
+ − 50 prints debug messages with env. var <b>LIBPROC_DEBUG</b>. But,
+ − 51 setting LIBSAPROC_DEBUG results in setting LIBPROC_DEBUG as well.
+ − 52 </p>
+ − 53 <p>
+ − 54 The best possible way to debug a transported core dump is to match the
+ − 55 debugger machine to that of core dump machine. i.e., have same Kernel
+ − 56 and libthread patch level between the machines. mdb (Solaris modular
+ − 57 debugger) may be used to find the Kernel patch level of core dump
+ − 58 machine and debugger machine may be brought to the same level.
+ − 59 </p>
+ − 60 <p>
+ − 61 If the matching machine is "far off" in your network, then
+ − 62 <ul>
+ − 63 <li>consider using rlogin and <a href="clhsdb.html">CLHSDB - SA command line HSDB interface</a> or
+ − 64 <li>use SA remote debugging and debug the core from core machine remotely.
+ − 65 </ul>
+ − 66 </p>
+ − 67
+ − 68 <p>
+ − 69 But, it may not be feasible to find matching machine to debug.
+ − 70 If so, you can copy all application shared objects (and libthread_db.so, if needed) from the core dump
+ − 71 machine into your debugger machine's directory, say, /export/applibs. Now, set <b>SA_ALTROOT</b>
+ − 72 environment variable to point to /export/applibs directory. Note that /export/applibs should either
+ − 73 contain matching 'full path' of libraries. i.e., /usr/lib/libthread_db.so from core
+ − 74 machine should be under /export/applibs/use/lib directory and /use/java/jre/lib/sparc/client/libjvm.so
+ − 75 from core machine should be under /export/applibs/use/java/jre/lib/sparc/client so on or /export/applibs
+ − 76 should just contain libthread_db.so, libjvm.so etc. directly.
+ − 77 </p>
+ − 78
+ − 79 <p>
+ − 80 Support for transported core dumps is <b>not</b> built into the standard version of libproc.so. You need to
+ − 81 set <b>LD_LIBRARY_PATH</b> env var to point to the path of a specially built version of libproc.so.
+ − 82 Note that this version of libproc.so has a special symbol to support transported core dump debugging.
+ − 83 In future, we may get this feature built into standard libproc.so -- if that happens, this step (of
+ − 84 setting LD_LIBRARY_PATH) can be skipped.
+ − 85 </p>
+ − 86
+ − 87 <h3>Ignoring libthread_db.so failures</h3>
+ − 88 <p>
+ − 89 If you are okay with missing thread related information, you can set
+ − 90 <b>SA_IGNORE_THREADDB</b> environment variable to any value. With this
+ − 91 set, SA ignores libthread_db failure, but you won't be able to get any
+ − 92 thread related information. But, you would be able to use SA and get
+ − 93 other information.
+ − 94 </p>
+ − 95
+ − 96 <h3>Linux SA against transported core dumps</h3>
+ − 97 <p>
+ − 98 On Linux, SA parses core and shared library ELF files. SA <b>does not</b> use
+ − 99 libthread_db.so or rtld_db.so for core dump debugging (although
+ − 100 libthread_db.so is used for live process debugging). But, you
+ − 101 may still face problems with transported core dumps, because matching shared
+ − 102 objects may not be in the path(s) specified in core dump file. To
+ − 103 workaround this, you can define environment variable <b>SA_ALTROOT</b>
+ − 104 to be the directory where shared libraries are kept. The semantics of
+ − 105 this env. variable is same as that for Solaris (please refer above).
+ − 106 </p>
+ − 107
+ − 108
+ − 109 </body>
+ − 110 </html>