| author | redestad |
| Tue, 16 Oct 2018 17:24:44 +0200 | |
| branch | JEP-230-microbenchmarks-branch |
| changeset 56980 | 26e61f9ceec4 |
| parent 51644 | 0df09dadd445 |
| permissions | -rw-r--r-- |
| 45227 | 1 |
<!DOCTYPE html> |
| 51644 | 2 |
<html> |
| 44511 | 3 |
<head> |
| 51644 | 4 |
<meta charset="utf-8"> |
5 |
<meta name="generator" content="pandoc"> |
|
6 |
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes"> |
|
|
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
7 |
<title>Testing the JDK</title> |
| 51644 | 8 |
<style type="text/css">code{white-space: pre;}</style>
|
| 47217 | 9 |
<link rel="stylesheet" href="../make/data/docs-resources/resources/jdk-default.css"> |
| 45227 | 10 |
<!--[if lt IE 9]> |
11 |
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script> |
|
12 |
<![endif]--> |
|
| 44511 | 13 |
<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>
|
14 |
</head> |
|
15 |
<body> |
|
| 45227 | 16 |
<header> |
|
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
17 |
<h1 class="title">Testing the JDK</h1> |
| 45227 | 18 |
</header> |
19 |
<nav id="TOC"> |
|
| 44511 | 20 |
<ul> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
21 |
<li><a href="#using-the-run-test-framework">Using the run-test framework</a><ul> |
|
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
22 |
<li><a href="#configuration">Configuration</a></li> |
|
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
23 |
</ul></li> |
| 44511 | 24 |
<li><a href="#test-selection">Test selection</a><ul> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
25 |
<li><a href="#jtreg">JTReg</a></li> |
| 44511 | 26 |
<li><a href="#gtest">Gtest</a></li> |
| 56980 | 27 |
<li><a href="#microbenchmarks">Microbenchmarks</a></li> |
| 44511 | 28 |
</ul></li> |
29 |
<li><a href="#test-results-and-summary">Test results and summary</a></li> |
|
30 |
<li><a href="#test-suite-control">Test suite control</a><ul> |
|
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
31 |
<li><a href="#jtreg-keywords">JTReg keywords</a></li> |
| 44511 | 32 |
<li><a href="#gtest-keywords">Gtest keywords</a></li> |
| 56980 | 33 |
<li><a href="#microbenchmark-keywords">Microbenchmark keywords</a></li> |
| 44511 | 34 |
</ul></li> |
35 |
</ul> |
|
| 45227 | 36 |
</nav> |
| 44511 | 37 |
<h2 id="using-the-run-test-framework">Using the run-test framework</h2> |
|
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
38 |
<p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p> |
| 51644 | 39 |
<p>The main target "run-test" uses the jdk-image as the tested product. There is also an alternate target "exploded-run-test" that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p> |
| 44511 | 40 |
<p>Some example command-lines:</p> |
41 |
<pre><code>$ make run-test-tier1 |
|
42 |
$ make run-test-jdk_lang JTREG="JOBS=8" |
|
43 |
$ make run-test TEST=jdk_lang |
|
44 |
$ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
|
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
45 |
$ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
|
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
46 |
$ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" |
| 56980 | 47 |
$ make run-test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2" |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
48 |
$ make exploded-run-test TEST=tier2</code></pre> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
49 |
<h3 id="configuration">Configuration</h3> |
| 51644 | 50 |
<p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> |
| 56980 | 51 |
<p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=<path to JMH jars></code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p> |
| 44511 | 52 |
<h2 id="test-selection">Test selection</h2> |
53 |
<p>All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.</p> |
|
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
54 |
<p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make run-test-tier1</code> is equivalent to <code>make run-test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>run-test TEST="x"</code> solution needs to be used.</p> |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
55 |
<p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
56 |
<h3 id="jtreg">JTReg</h3> |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
57 |
<p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p> |
|
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
58 |
<p>JTReg test groups can be specified either without a test root, e.g. <code>:tier1</code> (or <code>tier1</code>, the initial colon is optional), or with, e.g. <code>hotspot:tier1</code>, <code>test/jdk:jdk_util</code> or <code>$(TOPDIR)/test/hotspot/jtreg:hotspot_all</code>. The test root can be specified either as an absolute path, or a path relative to the JDK top directory, or the <code>test</code> directory. For simplicity, the hotspot JTReg test root, which really is <code>hotspot/jtreg</code> can be abbreviated as just <code>hotspot</code>.</p> |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
59 |
<p>When specified without a test root, all matching groups from all test roots will be added. Otherwise, only the group from the specified test root will be added.</p> |
|
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
60 |
<p>Individual JTReg tests or directories containing JTReg tests can also be specified, like <code>test/hotspot/jtreg/native_sanity/JniVersion.java</code> or <code>hotspot/jtreg/native_sanity</code>. Just like for test root selection, you can either specify an absolute path (which can even point to JTReg tests outside the source tree), or a path relative to either the JDK top directory or the <code>test</code> directory. <code>hotspot</code> can be used as an alias for <code>hotspot/jtreg</code> here as well.</p> |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
61 |
<p>As long as the test groups or test paths can be uniquely resolved, you do not need to enter the <code>jtreg:</code> prefix. If this is not possible, or if you want to use a fully qualified test descriptor, add <code>jtreg:</code>, e.g. <code>jtreg:test/hotspot/jtreg/native_sanity</code>.</p> |
| 44511 | 62 |
<h3 id="gtest">Gtest</h3> |
63 |
<p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p> |
|
64 |
<p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p> |
|
| 51644 | 65 |
<p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/<variant></code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p> |
| 56980 | 66 |
<h3 id="microbenchmarks">Microbenchmarks</h3> |
67 |
<p>Which microbenchmarks to run is selected using a regular expression following the <code>micro:</code> test descriptor, e.g., <code>micro:java.lang.reflect</code>. This delegates the test selection to JMH, meaning package name, class name and even benchmark method names can be used to select tests.</p> |
|
68 |
<p>Using special characters like <code>|</code> in the regular expression is possible, but needs to be escaped multiple times: <code>micro:ArrayCopy\\\\\|reflect</code>.</p> |
|
| 44511 | 69 |
<h2 id="test-results-and-summary">Test results and summary</h2> |
70 |
<p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p> |
|
71 |
<pre><code>============================== |
|
72 |
Test summary |
|
73 |
============================== |
|
74 |
TEST TOTAL PASS FAIL ERROR |
|
75 |
>> jtreg:jdk/test:tier1 1867 1865 2 0 << |
|
76 |
jtreg:langtools/test:tier1 4711 4711 0 0 |
|
77 |
jtreg:nashorn/test:tier1 133 133 0 0 |
|
78 |
============================== |
|
79 |
TEST FAILURE</code></pre> |
|
80 |
<p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>>> ... <<</code> marker for easy identification.</p> |
|
81 |
<p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p> |
|
82 |
<p>In case of test failures, <code>make run-test</code> will exit with a non-zero exit value.</p> |
|
| 51644 | 83 |
<p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> |
| 44511 | 84 |
<p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> |
85 |
<h2 id="test-suite-control">Test suite control</h2> |
|
86 |
<p>It is possible to control various aspects of the test suites using make control variables.</p> |
|
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
87 |
<p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> |
|
49293
9e1c1889136e
8200180: fix a typo in run-test framework documentation
iignatyev
parents:
48058
diff
changeset
|
88 |
<p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
| 51644 | 89 |
<p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
| 44511 | 90 |
<p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
91 |
<h3 id="jtreg-keywords">JTReg keywords</h3> |
| 44511 | 92 |
<h4 id="jobs">JOBS</h4> |
93 |
<p>The test concurrency (<code>-concurrency</code>).</p> |
|
94 |
<p>Defaults to TEST_JOBS (if set by <code>--with-test-jobs=</code>), otherwise it defaults to JOBS, except for Hotspot, where the default is <em>number of CPU cores/2</em>, but never more than 12.</p> |
|
95 |
<h4 id="timeout">TIMEOUT</h4> |
|
96 |
<p>The timeout factor (<code>-timeoutFactor</code>).</p> |
|
97 |
<p>Defaults to 4.</p> |
|
98 |
<h4 id="test_mode">TEST_MODE</h4> |
|
99 |
<p>The test mode (<code>-agentvm</code>, <code>-samevm</code> or <code>-othervm</code>).</p> |
|
100 |
<p>Defaults to <code>-agentvm</code>.</p> |
|
101 |
<h4 id="assert">ASSERT</h4> |
|
102 |
<p>Enable asserts (<code>-ea -esa</code>, or none).</p> |
|
103 |
<p>Set to <code>true</code> or <code>false</code>. If true, adds <code>-ea -esa</code>. Defaults to true, except for hotspot.</p> |
|
104 |
<h4 id="verbose">VERBOSE</h4> |
|
105 |
<p>The verbosity level (<code>-verbose</code>).</p> |
|
106 |
<p>Defaults to <code>fail,error,summary</code>.</p> |
|
107 |
<h4 id="retain">RETAIN</h4> |
|
108 |
<p>What test data to retain (<code>-retain</code>).</p> |
|
109 |
<p>Defaults to <code>fail,error</code>.</p> |
|
110 |
<h4 id="max_mem">MAX_MEM</h4> |
|
111 |
<p>Limit memory consumption (<code>-Xmx</code> and <code>-vmoption:-Xmx</code>, or none).</p> |
|
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
112 |
<p>Limit memory consumption for JTReg test framework and VM under test. Set to 0 to disable the limits.</p> |
| 44511 | 113 |
<p>Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).</p> |
114 |
<h4 id="options">OPTIONS</h4> |
|
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
115 |
<p>Additional options to the JTReg test framework.</p> |
|
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
116 |
<p>Use <code>JTREG="OPTIONS=--help all"</code> to see all available JTReg options.</p> |
| 44511 | 117 |
<h4 id="java_options">JAVA_OPTIONS</h4> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
118 |
<p>Additional Java options to JTReg (<code>-javaoption</code>).</p> |
| 44511 | 119 |
<h4 id="vm_options">VM_OPTIONS</h4> |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
45227
diff
changeset
|
120 |
<p>Additional VM options to JTReg (<code>-vmoption</code>).</p> |
| 44511 | 121 |
<h3 id="gtest-keywords">Gtest keywords</h3> |
122 |
<h4 id="repeat">REPEAT</h4> |
|
123 |
<p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p> |
|
124 |
<p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p> |
|
125 |
<h4 id="options-1">OPTIONS</h4> |
|
126 |
<p>Additional options to the Gtest test framework.</p> |
|
127 |
<p>Use <code>GTEST="OPTIONS=--help"</code> to see all available Gtest options.</p> |
|
| 56980 | 128 |
<h3 id="microbenchmark-keywords">Microbenchmark keywords</h3> |
129 |
<h4 id="fork">FORK</h4> |
|
130 |
<p>Override the number of benchmark forks to spawn. Same as specifying <code>-f <num></code>.</p> |
|
131 |
<h4 id="iter">ITER</h4> |
|
132 |
<p>Number of measurement iterations per fork. Same as specifying <code>-i <num></code>.</p> |
|
133 |
<h4 id="time">TIME</h4> |
|
134 |
<p>Amount of time to spend in each measurement iteration, in seconds. Same as specifying <code>-r <num></code></p> |
|
135 |
<h4 id="warmup_iter">WARMUP_ITER</h4> |
|
136 |
<p>Number of warmup iterations to run before the measurement phase in each fork. Same as specifying <code>-wi <num></code>.</p> |
|
137 |
<h4 id="warmup_time">WARMUP_TIME</h4> |
|
138 |
<p>Amount of time to spend in each warmup iteration. Same as specifying <code>-w <num></code>.</p> |
|
139 |
<h4 id="results_format">RESULTS_FORMAT</h4> |
|
140 |
<p>Specify to have the test run save a log of the values. Accepts the same values as <code>-rff</code>, i.e., <code>text</code>, <code>csv</code>, <code>scsv</code>, <code>json</code>, or <code>latex</code>.</p> |
|
141 |
<h4 id="vm_options-1">VM_OPTIONS</h4> |
|
142 |
<p>Additional VM arguments to provide to forked off VMs. Same as <code>-jvmArgs <args></code></p> |
|
143 |
<h4 id="options-2">OPTIONS</h4> |
|
144 |
<p>Additional arguments to send to JMH.</p> |
|
| 44511 | 145 |
</body> |
146 |
</html> |