author | joehw |
Thu, 05 Sep 2019 17:26:38 +0000 | |
changeset 58022 | 12885822f0c5 |
parent 57881 | c414c554b38b |
child 58296 | 7b534a5088d9 |
permissions | -rw-r--r-- |
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
1 |
% Testing the JDK |
44511 | 2 |
|
52342 | 3 |
## Using "make test" (the run-test framework) |
44511 | 4 |
|
5 |
This new way of running tests is developer-centric. It assumes that you have |
|
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
6 |
built a JDK locally and want to test it. Running common test targets is simple, |
44511 | 7 |
and more complex ad-hoc combination of tests is possible. The user interface is |
8 |
forgiving, and clearly report errors it cannot resolve. |
|
9 |
||
52342 | 10 |
The main target `test` uses the jdk-image as the tested product. There is |
11 |
also an alternate target `exploded-test` that uses the exploded image |
|
45266
29347d0c7122
8180651: Make target to run tests on exploded image
erikj
parents:
44983
diff
changeset
|
12 |
instead. Not all tests will run successfully on the exploded image, but using |
29347d0c7122
8180651: Make target to run tests on exploded image
erikj
parents:
44983
diff
changeset
|
13 |
this target can greatly improve rebuild times for certain workflows. |
29347d0c7122
8180651: Make target to run tests on exploded image
erikj
parents:
44983
diff
changeset
|
14 |
|
54268
0cf2c87ce79a
8221357: Update test documentation by deleting "cd test && make"
dholmes
parents:
53996
diff
changeset
|
15 |
Previously, `make test` was used to invoke an old system for running tests, and |
52342 | 16 |
`make run-test` was used for the new test framework. For backward compatibility |
17 |
with scripts and muscle memory, `run-test` (and variants like |
|
54268
0cf2c87ce79a
8221357: Update test documentation by deleting "cd test && make"
dholmes
parents:
53996
diff
changeset
|
18 |
`exploded-run-test` or `run-test-tier1`) are kept as aliases. |
52342 | 19 |
|
44511 | 20 |
Some example command-lines: |
21 |
||
52342 | 22 |
$ make test-tier1 |
23 |
$ make test-jdk_lang JTREG="JOBS=8" |
|
24 |
$ make test TEST=jdk_lang |
|
25 |
$ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
|
57881
c414c554b38b
8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents:
57493
diff
changeset
|
26 |
$ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT_FACTOR=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
52342 | 27 |
$ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" |
52595
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
28 |
$ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2" |
52342 | 29 |
$ make exploded-test TEST=tier2 |
44511 | 30 |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
31 |
### Configuration |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
32 |
|
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
33 |
To be able to run JTReg tests, `configure` needs to know where to find the |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
34 |
JTReg test framework. If it is not picked up automatically by configure, use |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
35 |
the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework. |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
36 |
Note that this option should point to the JTReg home, i.e. the top directory, |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
37 |
containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME` |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
38 |
environment variable to point to the JTReg home before running `configure`.) |
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
39 |
|
52595
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
40 |
To be able to run microbenchmarks, `configure` needs to know where to find |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
41 |
the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory |
53464 | 42 |
containing the core JMH and transitive dependencies. The recommended dependencies |
43 |
can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which |
|
52595
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
44 |
`--with-jmh=build/jmh/jars` should work. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
45 |
|
44511 | 46 |
## Test selection |
47 |
||
52342 | 48 |
All functionality is available using the `test` make target. In this use case, |
49 |
the test or tests to be executed is controlled using the `TEST` variable. To |
|
50 |
speed up subsequent test runs with no source code changes, `test-only` can be |
|
51 |
used instead, which do not depend on the source and test image build. |
|
44511 | 52 |
|
53 |
For some common top-level tests, direct make targets have been generated. This |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
54 |
includes all JTReg test groups, the hotspot gtest, and custom tests (if |
52342 | 55 |
present). This means that `make test-tier1` is equivalent to `make test |
44511 | 56 |
TEST="tier1"`, but the latter is more tab-completion friendly. For more complex |
52342 | 57 |
test runs, the `test TEST="x"` solution needs to be used. |
44511 | 58 |
|
59 |
The test specifications given in `TEST` is parsed into fully qualified test |
|
60 |
descriptors, which clearly and unambigously show which tests will be run. As an |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
61 |
example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
62 |
jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
63 |
jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
64 |
always submit a list of fully qualified test descriptors in the `TEST` variable |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
65 |
if you want to shortcut the parser. |
44511 | 66 |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
67 |
### JTReg |
44511 | 68 |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
69 |
JTReg tests can be selected either by picking a JTReg test group, or a selection |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
70 |
of files or directories containing JTReg tests. |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
71 |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
72 |
JTReg test groups can be specified either without a test root, e.g. `:tier1` |
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
73 |
(or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`, |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
74 |
`test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
75 |
root can be specified either as an absolute path, or a path relative to the |
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
76 |
JDK top directory, or the `test` directory. For simplicity, the hotspot |
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
77 |
JTReg test root, which really is `hotspot/jtreg` can be abbreviated as |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
78 |
just `hotspot`. |
44511 | 79 |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
80 |
When specified without a test root, all matching groups from all test roots |
44511 | 81 |
will be added. Otherwise, only the group from the specified test root will be |
82 |
added. |
|
83 |
||
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
84 |
Individual JTReg tests or directories containing JTReg tests can also be |
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
85 |
specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
86 |
`hotspot/jtreg/native_sanity`. Just like for test root selection, you can |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
87 |
either specify an absolute path (which can even point to JTReg tests outside |
50885
7c728fa9d1af
8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents:
50267
diff
changeset
|
88 |
the source tree), or a path relative to either the JDK top directory or the |
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
89 |
`test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as |
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
90 |
well. |
44511 | 91 |
|
92 |
As long as the test groups or test paths can be uniquely resolved, you do not |
|
93 |
need to enter the `jtreg:` prefix. If this is not possible, or if you want to |
|
94 |
use a fully qualified test descriptor, add `jtreg:`, e.g. |
|
50267
1582de22e3a1
8198323: testing.md not updated for repository layout change
ihse
parents:
49293
diff
changeset
|
95 |
`jtreg:test/hotspot/jtreg/native_sanity`. |
44511 | 96 |
|
97 |
### Gtest |
|
98 |
||
99 |
Since the Hotspot Gtest suite is so quick, the default is to run all tests. |
|
100 |
This is specified by just `gtest`, or as a fully qualified test descriptor |
|
101 |
`gtest:all`. |
|
102 |
||
103 |
If you want, you can single out an individual test or a group of tests, for |
|
104 |
instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This |
|
105 |
can be particularly useful if you want to run a shaky test repeatedly. |
|
106 |
||
48058
c5eefa465e37
8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents:
47216
diff
changeset
|
107 |
For Gtest, there is a separate test suite for each JVM variant. The JVM variant |
c5eefa465e37
8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents:
47216
diff
changeset
|
108 |
is defined by adding `/<variant>` to the test descriptor, e.g. |
c5eefa465e37
8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents:
47216
diff
changeset
|
109 |
`gtest:Log/client`. If you specify no variant, gtest will run once for each JVM |
c5eefa465e37
8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents:
47216
diff
changeset
|
110 |
variant present (e.g. server, client). So if you only have the server JVM |
c5eefa465e37
8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents:
47216
diff
changeset
|
111 |
present, then `gtest:all` will be equivalent to `gtest:all/server`. |
c5eefa465e37
8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents:
47216
diff
changeset
|
112 |
|
52595
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
113 |
### Microbenchmarks |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
114 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
115 |
Which microbenchmarks to run is selected using a regular expression |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
116 |
following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
117 |
delegates the test selection to JMH, meaning package name, class name and even |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
118 |
benchmark method names can be used to select tests. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
119 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
120 |
Using special characters like `|` in the regular expression is possible, but |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
121 |
needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
122 |
|
52342 | 123 |
### Special tests |
124 |
||
125 |
A handful of odd tests that are not covered by any other testing framework are |
|
126 |
accessible using the `special:` test descriptor. Currently, this includes |
|
52396
e292e94b448a
8213058: remove ExecuteInternalVMTests and VerboseInternalVMTests flags
iignatyev
parents:
52342
diff
changeset
|
127 |
`failure-handler` and `make`. |
52342 | 128 |
|
129 |
* Failure handler testing is run using `special:failure-handler` or just |
|
130 |
`failure-handler` as test descriptor. |
|
131 |
||
132 |
* Tests for the build system, including both makefiles and related |
|
133 |
functionality, is run using `special:make` or just `make` as test |
|
134 |
descriptor. This is equivalent to `special:make:all`. |
|
135 |
||
136 |
A specific make test can be run by supplying it as argument, e.g. |
|
137 |
`special:make:idea`. As a special syntax, this can also be expressed as |
|
138 |
`make-idea`, which allows for command lines as `make test-make-idea`. |
|
139 |
||
44511 | 140 |
## Test results and summary |
141 |
||
142 |
At the end of the test run, a summary of all tests run will be presented. This |
|
143 |
will have a consistent look, regardless of what test suites were used. This is |
|
144 |
a sample summary: |
|
145 |
||
146 |
============================== |
|
147 |
Test summary |
|
148 |
============================== |
|
149 |
TEST TOTAL PASS FAIL ERROR |
|
150 |
>> jtreg:jdk/test:tier1 1867 1865 2 0 << |
|
151 |
jtreg:langtools/test:tier1 4711 4711 0 0 |
|
152 |
jtreg:nashorn/test:tier1 133 133 0 0 |
|
153 |
============================== |
|
154 |
TEST FAILURE |
|
155 |
||
156 |
Tests where the number of TOTAL tests does not equal the number of PASSed tests |
|
157 |
will be considered a test failure. These are marked with the `>> ... <<` marker |
|
158 |
for easy identification. |
|
159 |
||
160 |
The classification of non-passed tests differs a bit between test suites. In |
|
161 |
the summary, ERROR is used as a catch-all for tests that neither passed nor are |
|
162 |
classified as failed by the framework. This might indicate test framework |
|
163 |
error, timeout or other problems. |
|
164 |
||
52342 | 165 |
In case of test failures, `make test` will exit with a non-zero exit value. |
44511 | 166 |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
167 |
All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, |
44511 | 168 |
where TEST_ID is a path-safe conversion from the fully qualified test |
169 |
descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is |
|
170 |
`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the |
|
171 |
test run. |
|
172 |
||
173 |
Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For |
|
174 |
some frameworks, this directory might contain information that is useful in |
|
175 |
determining the cause of a failed test. |
|
176 |
||
177 |
## Test suite control |
|
178 |
||
179 |
It is possible to control various aspects of the test suites using make control |
|
180 |
variables. |
|
181 |
||
182 |
These variables use a keyword=value approach to allow multiple values to be |
|
57881
c414c554b38b
8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents:
57493
diff
changeset
|
183 |
set. So, for instance, `JTREG="JOBS=1;TIMEOUT_FACTOR=8"` will set the JTReg |
44511 | 184 |
concurrency level to 1 and the timeout factor to 8. This is equivalent to |
57881
c414c554b38b
8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents:
57493
diff
changeset
|
185 |
setting `JTREG_JOBS=1 JTREG_TIMEOUT_FACTOR=8`, but using the keyword format means that |
44511 | 186 |
the `JTREG` variable is parsed and verified for correctness, so |
57881
c414c554b38b
8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents:
57493
diff
changeset
|
187 |
`JTREG="TMIEOUT_FACTOR=8"` would give an error, while `JTREG_TMIEOUT_FACTOR=8` would just |
44511 | 188 |
pass unnoticed. |
189 |
||
190 |
To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell |
|
191 |
normally eats `;`, the recommended usage is to write the assignment inside |
|
192 |
qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, |
|
49293
9e1c1889136e
8200180: fix a typo in run-test framework documentation
iignatyev
parents:
48058
diff
changeset
|
193 |
as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`. |
44511 | 194 |
|
57881
c414c554b38b
8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents:
57493
diff
changeset
|
195 |
(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT_FACTOR=8`. |
44511 | 196 |
Also, as a special technique, the string `%20` will be replaced with space for |
49293
9e1c1889136e
8200180: fix a typo in run-test framework documentation
iignatyev
parents:
48058
diff
changeset
|
197 |
certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. |
44511 | 198 |
This can be useful if you have layers of scripts and have trouble getting |
199 |
proper quoting of command line arguments through.) |
|
200 |
||
201 |
As far as possible, the names of the keywords have been standardized between |
|
202 |
test suites. |
|
203 |
||
53464 | 204 |
### General keywords (TEST_OPTS) |
205 |
||
206 |
Some keywords are valid across different test suites. If you want to run |
|
207 |
tests from multiple test suites, or just don't want to care which test suite specific |
|
208 |
control variable to use, then you can use the general TEST_OPTS control variable. |
|
209 |
||
210 |
There are also some keywords that applies globally to the test runner system, |
|
211 |
not to any specific test suites. These are also available as TEST_OPTS keywords. |
|
212 |
||
213 |
#### JOBS |
|
214 |
||
215 |
Currently only applies to JTReg. |
|
216 |
||
217 |
#### TIMEOUT_FACTOR |
|
218 |
||
219 |
Currently only applies to JTReg. |
|
220 |
||
221 |
#### VM_OPTIONS |
|
222 |
||
223 |
Applies to JTReg, GTest and Micro. |
|
224 |
||
225 |
#### JAVA_OPTIONS |
|
226 |
||
227 |
Applies to JTReg, GTest and Micro. |
|
228 |
||
229 |
#### AOT_MODULES |
|
230 |
||
231 |
Applies to JTReg and GTest. |
|
232 |
||
233 |
#### JCOV |
|
234 |
||
235 |
This keywords applies globally to the test runner system. If set to `true`, it |
|
236 |
enables JCov coverage reporting for all tests run. To be useful, the JDK under |
|
237 |
test must be run with a JDK built with JCov instrumentation (`configure |
|
238 |
--with-jcov=<path to directory containing lib/jcov.jar>`, `make jcov-image`). |
|
239 |
||
240 |
The simplest way to run tests with JCov coverage report is to use the special |
|
241 |
target `jcov-test` instead of `test`, e.g. `make jcov-test TEST=jdk_lang`. This |
|
242 |
will make sure the JCov image is built, and that JCov reporting is enabled. |
|
243 |
||
244 |
The JCov report is stored in `build/$BUILD/test-results/jcov-output`. |
|
245 |
||
246 |
Please note that running with JCov reporting can be very memory intensive. |
|
247 |
||
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
248 |
### JTReg keywords |
44511 | 249 |
|
250 |
#### JOBS |
|
251 |
The test concurrency (`-concurrency`). |
|
252 |
||
253 |
Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to |
|
53996
35a30b432396
8219906: Update test documentation with default test jobs settings
aoqi
parents:
53464
diff
changeset
|
254 |
JOBS, except for Hotspot, where the default is *number of CPU cores/2* (for |
35a30b432396
8219906: Update test documentation with default test jobs settings
aoqi
parents:
53464
diff
changeset
|
255 |
sparc, if more than 16 cpus, then *number of CPU cores/5*, otherwise *number of |
35a30b432396
8219906: Update test documentation with default test jobs settings
aoqi
parents:
53464
diff
changeset
|
256 |
CPU cores/4*), but never more than *memory size in GB/2*. |
44511 | 257 |
|
53464 | 258 |
#### TIMEOUT_FACTOR |
44511 | 259 |
The timeout factor (`-timeoutFactor`). |
260 |
||
261 |
Defaults to 4. |
|
262 |
||
263 |
#### TEST_MODE |
|
264 |
The test mode (`-agentvm`, `-samevm` or `-othervm`). |
|
265 |
||
266 |
Defaults to `-agentvm`. |
|
267 |
||
268 |
#### ASSERT |
|
269 |
Enable asserts (`-ea -esa`, or none). |
|
270 |
||
271 |
Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except |
|
272 |
for hotspot. |
|
273 |
||
274 |
#### VERBOSE |
|
275 |
The verbosity level (`-verbose`). |
|
276 |
||
277 |
Defaults to `fail,error,summary`. |
|
278 |
||
279 |
#### RETAIN |
|
280 |
What test data to retain (`-retain`). |
|
281 |
||
282 |
Defaults to `fail,error`. |
|
283 |
||
284 |
#### MAX_MEM |
|
285 |
Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). |
|
286 |
||
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
287 |
Limit memory consumption for JTReg test framework and VM under test. Set to 0 |
44511 | 288 |
to disable the limits. |
289 |
||
290 |
Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). |
|
291 |
||
53464 | 292 |
#### KEYWORDS |
293 |
||
294 |
JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that |
|
295 |
spaces and special characters (like `!`) are properly quoted. To avoid some |
|
296 |
issues, the special value `%20` can be used instead of space. |
|
297 |
||
298 |
#### EXTRA_PROBLEM_LISTS |
|
299 |
||
300 |
Use additional problem lists file or files, in addition to the default |
|
301 |
ProblemList.txt located at the JTReg test roots. |
|
302 |
||
303 |
If multiple file names are specified, they should be separated by space (or, to |
|
304 |
help avoid quoting issues, the special value `%20`). |
|
305 |
||
306 |
The file names should be either absolute, or relative to the JTReg test root of |
|
307 |
the tests to be run. |
|
308 |
||
57493
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
309 |
#### RUN_PROBLEM_LISTS |
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
310 |
|
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
311 |
Use the problem lists to select tests instead of excluding them. |
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
312 |
|
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
313 |
Set to `true` or `false`. |
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
314 |
If `true`, JTReg will use `-match:` option, otherwise `-exclude:` will be used. |
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
315 |
Default is `false`. |
b95ebdbf68ca
8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents:
54570
diff
changeset
|
316 |
|
53464 | 317 |
|
44511 | 318 |
#### OPTIONS |
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
319 |
Additional options to the JTReg test framework. |
44511 | 320 |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
321 |
Use `JTREG="OPTIONS=--help all"` to see all available JTReg options. |
44511 | 322 |
|
323 |
#### JAVA_OPTIONS |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
324 |
Additional Java options to JTReg (`-javaoption`). |
44511 | 325 |
|
326 |
#### VM_OPTIONS |
|
45618
74f490ba783a
8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents:
44983
diff
changeset
|
327 |
Additional VM options to JTReg (`-vmoption`). |
44511 | 328 |
|
53464 | 329 |
#### AOT_MODULES |
330 |
||
331 |
Generate AOT modules before testing for the specified module, or set of |
|
332 |
modules. If multiple modules are specified, they should be separated by space |
|
333 |
(or, to help avoid quoting issues, the special value `%20`). |
|
334 |
||
44511 | 335 |
### Gtest keywords |
336 |
||
337 |
#### REPEAT |
|
338 |
The number of times to repeat the tests (`--gtest_repeat`). |
|
339 |
||
340 |
Default is 1. Set to -1 to repeat indefinitely. This can be especially useful |
|
341 |
combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent |
|
342 |
problem. |
|
343 |
||
344 |
#### OPTIONS |
|
345 |
Additional options to the Gtest test framework. |
|
346 |
||
347 |
Use `GTEST="OPTIONS=--help"` to see all available Gtest options. |
|
348 |
||
53464 | 349 |
#### AOT_MODULES |
350 |
||
351 |
Generate AOT modules before testing for the specified module, or set of |
|
352 |
modules. If multiple modules are specified, they should be separated by space |
|
353 |
(or, to help avoid quoting issues, the special value `%20`). |
|
354 |
||
52595
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
355 |
### Microbenchmark keywords |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
356 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
357 |
#### FORK |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
358 |
Override the number of benchmark forks to spawn. Same as specifying `-f <num>`. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
359 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
360 |
#### ITER |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
361 |
Number of measurement iterations per fork. Same as specifying `-i <num>`. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
362 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
363 |
#### TIME |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
364 |
Amount of time to spend in each measurement iteration, in seconds. Same as |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
365 |
specifying `-r <num>` |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
366 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
367 |
#### WARMUP_ITER |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
368 |
Number of warmup iterations to run before the measurement phase in each fork. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
369 |
Same as specifying `-wi <num>`. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
370 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
371 |
#### WARMUP_TIME |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
372 |
Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
373 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
374 |
#### RESULTS_FORMAT |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
375 |
Specify to have the test run save a log of the values. Accepts the same values |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
376 |
as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
377 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
378 |
#### VM_OPTIONS |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
379 |
Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>` |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
380 |
|
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
381 |
#### OPTIONS |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
382 |
Additional arguments to send to JMH. |
16609197022c
8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents:
52396
diff
changeset
|
383 |
|
54473
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
384 |
## Notes for Specific Tests |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
385 |
|
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
386 |
### Docker Tests |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
387 |
|
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
388 |
Docker tests with default parameters may fail on systems with glibc versions not |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
389 |
compatible with the one used in the default docker image (e.g., Oracle Linux 7.6 for x86). |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
390 |
For example, they pass on Ubuntu 16.04 but fail on Ubuntu 18.04 if run like this on x86: |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
391 |
|
54525
5ae4d3f46537
8222299: [TESTBUG] move hotspot container tests to hotspot/containers
mseledtsov
parents:
54473
diff
changeset
|
392 |
$ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" |
54473
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
393 |
|
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
394 |
To run these tests correctly, additional parameters for the correct docker image are |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
395 |
required on Ubuntu 18.04 by using `JAVA_OPTIONS`. |
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
396 |
|
54525
5ae4d3f46537
8222299: [TESTBUG] move hotspot container tests to hotspot/containers
mseledtsov
parents:
54473
diff
changeset
|
397 |
$ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu -Djdk.test.docker.image.version=latest" |
54473
e437ad5643d6
8221894: Add comments for docker tests in the test doc
jiefu
parents:
54268
diff
changeset
|
398 |
|
54570
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
399 |
### Non-US locale |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
400 |
|
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
401 |
If your locale is non-US, some tests are likely to fail. To work around this you can |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
402 |
set the locale to US. On Unix platforms simply setting `LANG="en_US"` in the |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
403 |
environment before running tests should work. On Windows, setting |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
404 |
`JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US"` helps for most, but not all test cases. |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
405 |
For example: |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
406 |
|
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
407 |
$ export LANG="en_US" && make test TEST=... |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
408 |
$ make test JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US" TEST=... |
93b702d2a0cb
8222444: Add a suggestion for non-US locale in the test doc
erikj
parents:
54525
diff
changeset
|
409 |
|
44511 | 410 |
--- |
44983 | 411 |
# Override some definitions in the global css file that are not optimal for |
412 |
# this document. |
|
44511 | 413 |
header-includes: |
414 |
- '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' |
|
415 |
--- |