|
1 % Testing OpenJDK |
|
2 |
|
3 ## Using the run-test framework |
|
4 |
|
5 This new way of running tests is developer-centric. It assumes that you have |
|
6 built a jdk locally and want to test it. Running common test targets is simple, |
|
7 and more complex ad-hoc combination of tests is possible. The user interface is |
|
8 forgiving, and clearly report errors it cannot resolve. |
|
9 |
|
10 Some example command-lines: |
|
11 |
|
12 $ make run-test-tier1 |
|
13 $ make run-test-jdk_lang JTREG="JOBS=8" |
|
14 $ make run-test TEST=jdk_lang |
|
15 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
|
16 $ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug" |
|
17 $ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" |
|
18 |
|
19 ## Test selection |
|
20 |
|
21 All functionality is available using the run-test make target. In this use |
|
22 case, the test or tests to be executed is controlled using the `TEST` variable. |
|
23 To speed up subsequent test runs with no source code changes, run-test-only can |
|
24 be used instead, which do not depend on the source and test image build. |
|
25 |
|
26 For some common top-level tests, direct make targets have been generated. This |
|
27 includes all JTreg test groups, the hotspot gtest, and custom tests (if |
|
28 present). This means that `make run-test-tier1` is equivalent to `make run-test |
|
29 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex |
|
30 test runs, the `run-test TEST="x"` solution needs to be used. |
|
31 |
|
32 The test specifications given in `TEST` is parsed into fully qualified test |
|
33 descriptors, which clearly and unambigously show which tests will be run. As an |
|
34 example, `:tier1` will expand to `jtreg:jdk/test:tier1 |
|
35 jtreg:langtools/test:tier1 jtreg:nashorn/test:tier1 jtreg:jaxp/test:tier1`. You |
|
36 can always submit a list of fully qualified test descriptors in the `TEST` |
|
37 variable if you want to shortcut the parser. |
|
38 |
|
39 ### JTreg |
|
40 |
|
41 JTreg test groups can be specified either without a test root, e.g. `:tier1` |
|
42 (or `tier1`, the initial colon is optional), or with, e.g. |
|
43 `hotspot/test:tier1`, `jdk/test:jdk_util`. |
|
44 |
|
45 When specified without a test root, all matching groups from all tests roots |
|
46 will be added. Otherwise, only the group from the specified test root will be |
|
47 added. |
|
48 |
|
49 Individual JTreg tests or directories containing JTreg tests can also be |
|
50 specified, like `hotspot/test/native_sanity/JniVersion.java` or |
|
51 `hotspot/test/native_sanity`. You can also specify an absolute path, to point |
|
52 to a JTreg test outside the source tree. |
|
53 |
|
54 As long as the test groups or test paths can be uniquely resolved, you do not |
|
55 need to enter the `jtreg:` prefix. If this is not possible, or if you want to |
|
56 use a fully qualified test descriptor, add `jtreg:`, e.g. |
|
57 `jtreg:hotspot/test/native_sanity`. |
|
58 |
|
59 ### Gtest |
|
60 |
|
61 Since the Hotspot Gtest suite is so quick, the default is to run all tests. |
|
62 This is specified by just `gtest`, or as a fully qualified test descriptor |
|
63 `gtest:all`. |
|
64 |
|
65 If you want, you can single out an individual test or a group of tests, for |
|
66 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This |
|
67 can be particularly useful if you want to run a shaky test repeatedly. |
|
68 |
|
69 ## Test results and summary |
|
70 |
|
71 At the end of the test run, a summary of all tests run will be presented. This |
|
72 will have a consistent look, regardless of what test suites were used. This is |
|
73 a sample summary: |
|
74 |
|
75 ============================== |
|
76 Test summary |
|
77 ============================== |
|
78 TEST TOTAL PASS FAIL ERROR |
|
79 >> jtreg:jdk/test:tier1 1867 1865 2 0 << |
|
80 jtreg:langtools/test:tier1 4711 4711 0 0 |
|
81 jtreg:nashorn/test:tier1 133 133 0 0 |
|
82 ============================== |
|
83 TEST FAILURE |
|
84 |
|
85 Tests where the number of TOTAL tests does not equal the number of PASSed tests |
|
86 will be considered a test failure. These are marked with the `>> ... <<` marker |
|
87 for easy identification. |
|
88 |
|
89 The classification of non-passed tests differs a bit between test suites. In |
|
90 the summary, ERROR is used as a catch-all for tests that neither passed nor are |
|
91 classified as failed by the framework. This might indicate test framework |
|
92 error, timeout or other problems. |
|
93 |
|
94 In case of test failures, `make run-test` will exit with a non-zero exit value. |
|
95 |
|
96 All tests have their result stored in `build/$BUILD/test-result/$TEST_ID`, |
|
97 where TEST_ID is a path-safe conversion from the fully qualified test |
|
98 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is |
|
99 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the |
|
100 test run. |
|
101 |
|
102 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For |
|
103 some frameworks, this directory might contain information that is useful in |
|
104 determining the cause of a failed test. |
|
105 |
|
106 ## Test suite control |
|
107 |
|
108 It is possible to control various aspects of the test suites using make control |
|
109 variables. |
|
110 |
|
111 These variables use a keyword=value approach to allow multiple values to be |
|
112 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTreg |
|
113 concurrency level to 1 and the timeout factor to 8. This is equivalent to |
|
114 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that |
|
115 the `JTREG` variable is parsed and verified for correctness, so |
|
116 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just |
|
117 pass unnoticed. |
|
118 |
|
119 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell |
|
120 normally eats `;`, the recommended usage is to write the assignment inside |
|
121 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, |
|
122 as in `JTREG="VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug"`. |
|
123 |
|
124 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`. |
|
125 Also, as a special technique, the string `%20` will be replaced with space for |
|
126 certain options, e.g. `JTREG=VM_OTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. |
|
127 This can be useful if you have layers of scripts and have trouble getting |
|
128 proper quoting of command line arguments through.) |
|
129 |
|
130 As far as possible, the names of the keywords have been standardized between |
|
131 test suites. |
|
132 |
|
133 ### JTreg keywords |
|
134 |
|
135 #### JOBS |
|
136 The test concurrency (`-concurrency`). |
|
137 |
|
138 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to |
|
139 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but |
|
140 never more than 12. |
|
141 |
|
142 #### TIMEOUT |
|
143 The timeout factor (`-timeoutFactor`). |
|
144 |
|
145 Defaults to 4. |
|
146 |
|
147 #### TEST_MODE |
|
148 The test mode (`-agentvm`, `-samevm` or `-othervm`). |
|
149 |
|
150 Defaults to `-agentvm`. |
|
151 |
|
152 #### ASSERT |
|
153 Enable asserts (`-ea -esa`, or none). |
|
154 |
|
155 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except |
|
156 for hotspot. |
|
157 |
|
158 #### VERBOSE |
|
159 The verbosity level (`-verbose`). |
|
160 |
|
161 Defaults to `fail,error,summary`. |
|
162 |
|
163 #### RETAIN |
|
164 What test data to retain (`-retain`). |
|
165 |
|
166 Defaults to `fail,error`. |
|
167 |
|
168 #### MAX_MEM |
|
169 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). |
|
170 |
|
171 Limit memory consumption for JTreg test framework and VM under test. Set to 0 |
|
172 to disable the limits. |
|
173 |
|
174 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). |
|
175 |
|
176 #### OPTIONS |
|
177 Additional options to the JTreg test framework. |
|
178 |
|
179 Use `JTREG="OPTIONS=--help all"` to see all available JTreg options. |
|
180 |
|
181 #### JAVA_OPTIONS |
|
182 Additional Java options to JTreg (`-javaoption`). |
|
183 |
|
184 #### VM_OPTIONS |
|
185 Additional VM options to JTreg (`-vmoption`). |
|
186 |
|
187 ### Gtest keywords |
|
188 |
|
189 #### REPEAT |
|
190 The number of times to repeat the tests (`--gtest_repeat`). |
|
191 |
|
192 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful |
|
193 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent |
|
194 problem. |
|
195 |
|
196 #### OPTIONS |
|
197 Additional options to the Gtest test framework. |
|
198 |
|
199 Use `GTEST="OPTIONS=--help"` to see all available Gtest options. |
|
200 |
|
201 --- |
|
202 # Override some definitions in http://openjdk.java.net/page.css that are |
|
203 # unsuitable for this document. |
|
204 header-includes: |
|
205 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' |
|
206 - '<style type="text/css">pre { font-size: 10pt; }</style>' |
|
207 --- |