7197666: java -d64 -version core dumps in a box with lots of memory
Summary: Allow task queues to be mmapped instead of malloced on Solaris
Reviewed-by: coleenp, jmasa, johnc, tschatzl
<HTML>
<HEAD>
<TITLE>
Using The HotSpot Serviceability Agent
</TITLE>
</HEAD>
<BODY TEXT="#000000" BGCOLOR="#FFFFFF">
<H1>
Using The HotSpot Serviceability Agent
</H1>
<P>
<H2>
Contents
</H2>
</P>
<UL>
<LI> <A HREF = "#INTRODUCTION">Introduction</A>
<LI> <A HREF = "#SOURCES">Organization of the sources</A>
<LI> <A HREF = "#RUNNING">Running HSDB</A>
<LI> <A HREF = "#NOTES">Notes</A>
</UL>
<H2>
<A NAME="INTRODUCTION">
Introduction
</A>
</H2>
<P>
The HotSpot Serviceability Agent (SA) is a set of Java APIs which
mirror the internal APIs of the HotSpot VM and which can be used to
examine the state of a HotSpot VM.
</P>
<P>
The system understands the layout of certain VM data structures and is
able to traverse these structures in an examination-only fashion; that
is, it does not rely on being able to run code in the target VM. For
this reason it transparently works with either a running VM or a core
file.
</P>
<P>
The system can reconstruct information about Java frames on the stack
and objects in the heap. Many of the important data structures in the
VM like the CodeCache, Universe, StubQueue, Frames, and VFrames have
been exposed and have relatively complete (or at least useful)
implementations.
</P>
<P>
A small graphical debugger called HSDB (the "HotSpot Debugger") has
been written using these APIs. It provides stack memory dumps
annotated with method invocations, compiled-code inlining (if
present), interpreted vs. compiled code, interpreter codelets (if
interpreted), and live oops from oop-map information. It also provides
a tree-based oop inspector. More information will be added as
necessary; please <A HREF = "mailto:kenneth.russell@eng">send
email</A> with suggestions on what would be useful.
</P>
<P>
The SA currently only works on Solaris. It uses dbx to connect to the
remote process or core file and communicates with a small piece of
code (an "import module") loaded into the debugger.
</P>
<H2>
<A NAME="SOURCES">
Organization of the sources
</A>
</H2>
<P>
The Java-side source code, which is the bulk of the SA, is in
src/share/vm/agent. The organization of the sun.jvm.hotspot package
hierarchy mirrors the organization in the VM. This should allow
engineers familiar with the HotSpot sources to quickly understand how
the SA works and to make modifications if necessary. To build these
sources, cd to src/share/vm/agent and type "make".
</P>
<P>
The SA on Solaris works by communicating with a small piece of code
(an "import module") loaded into dbx. The source code for this import
module is in src/os/solaris/agent. To build this library, cd to
src/os/solaris/agent and type "make 32bit" or "make 64bit". The
distinction is necessary because the SPARC version of dbx ships with
two versions of its executable, and depending on which architecture
(v8 or v9) the debugger is running on selects the appropriate
executable. The SA tries the v8 version first, but if you are running
on a v9 machine you must provide both versions to the SA.
</P>
<P>
The system is currently hardwired to look on jano for its dbx
executable and import module. The relevant directory structure looks
like this:
<UL>
<LI> .../hotspot/sa/
<UL>
<LI> solaris/
<UL>
<LI> sparc/
<UL>
<LI> bin/
<UL>
<LI> dbx: symlink to (v8) dbx 7.0 executable
</UL>
</UL>
<UL>
<LI> lib/
<UL>
<LI> libsvc_agent_dbx.so: 32-bit version of import module
</UL>
</UL>
<LI> sparcv9/
<UL>
<LI> lib/
<UL>
<LI> libsvc_agent_dbx.so: 32-bit version of import module
</UL>
</UL>
</UL>
</UL>
</UL>
</P>
<P>
The code which builds up path names to these executables is contained
in sun.jvm.hotspot.HotSpotAgent.java. There are hardcoded paths in
this file to jano, but the rest of the system is isolated from this.
</P>
<P>
(It would be nice to have a panel in the debugger allowing
configuration of all of the known operating systems' options; right
now Solaris is the only supported OS, making that easier.)
</P>
<H2>
<A NAME="RUNNING">
Running HSDB
</A>
</H2>
<P>
An installation of HSDB has been placed on jano. To access it, add the
following directory to your PATH:
</P>
<P>
<PRE>
/net/jano/export/disk05/hotspot/sa/bin/common
</PRE>
</P>
<P>
To start the debugger, type "hsdb".
</P>
<P>
Alternatively, you can start a local build of the debugger by building
the sources in src/share/vm/agent, cd'ing to that directory, and
typing "java sun.jvm.hotspot.HSDB".
</P>
<P>
There are three modes for the debugger: attaching to a local process,
opening a core file, and attaching to a remote "debug server". The
remote case requires two programs to be running on the remote machine:
the rmiregistry (see the script "start-rmiregistry" in this directory;
run this in the background) and the debug server (see the script
"start-debug-server"), in that order. start-rmiregistry takes no
arguments; start-debug-server takes as argument the process ID or the
executable and core file names to allow remote debugging of. Make sure
you do NOT have a CLASSPATH environment variable set when you run
these scripts. (The classes put into the rmiregistry are in sun.*, and
there are permissions problems if they aren't placed on the boot
classpath.)
</P>
<P>
NOTE that the SA currently only works against VMs on Solaris/SPARC.
Remote debugging of Solaris/SPARC VMs on arbitrary platforms is
possible using the debug server; select "Connect to debug server..."
in HSDB.
</P>
<P>
Once the debugger has been launched, the threads list is displayed.
The current set of functionality allows:
</P>
<UL>
<LI> Browsing of the annotated stack memory ("Stack Memory" button). It
is currently annotated with the following information:
<UL>
<LI> Method names of the Java frames and their extents (supporting
inlined compiled methods)
<LI> Locations and types of oops, found using the oop map information
from compiled methods (interpreter oop maps coming soon)
<LI> If a Java frame was interrupted by a signal (e.g., because of a
crash), annotates the frame with the signal name and number
<LI> Interpreter codelet descriptions for interpreted frames
</UL>
<LI> Finding which thread or threads caused a crash (currently
identified by the presence of a signal handler frame)
<LI> Browsing of oops using the Oop Inspector.
<LI> Browsing of the java.lang.Thread object's oop.
<LI> Object histogram and inspection of objects therein.
</UL>
</P>
<P>
More functionality is planned. Please <A HREF =
"mailto:kenneth.russell@eng">send email</A> with suggestions on what
would be useful, with any questions or comments, or if the debugger
crashes.
</P>
<H2>
<A NAME="NOTES">
Notes
</A>
</H2>
<P>
HSDB does not suspend the system at a safepoint, but at an arbitrary
point. This means that many of the invariants in the VM's code are not
followed.
</P>
<P>
As it happens, most of the places where the code ported over from the
VM has failed involve the topmost frame on the stack. Some
modifications have been made to allow the system to recognize
problematic situations.
</P>
<P>
Certainly, not all of the failure modes of the debugger have been
found. Please <A HREF = "mailto:kenneth.russell@eng">send email</A> if
HSDB throws an exception. The best debugging aid in these situations
is a core file since it is a static view of the VM to which we can
then adapt the debugger code, as opposed to having to try to suspend
the VM over and over to reproduce the failure. gcore (1) is a useful
tool. (NOTE: do not try gcore with any application using the DGA X
server extension (example: Java2Demo); the kernel will panic. See bug
4343237.)
</P>
</BODY>
</HTML>