1 % Testing the JDK |
1 % Testing the JDK |
2 |
2 |
3 ## Using the run-test framework |
3 ## Using "make test" (the run-test framework) |
4 |
4 |
5 This new way of running tests is developer-centric. It assumes that you have |
5 This new way of running tests is developer-centric. It assumes that you have |
6 built a JDK locally and want to test it. Running common test targets is simple, |
6 built a JDK locally and want to test it. Running common test targets is simple, |
7 and more complex ad-hoc combination of tests is possible. The user interface is |
7 and more complex ad-hoc combination of tests is possible. The user interface is |
8 forgiving, and clearly report errors it cannot resolve. |
8 forgiving, and clearly report errors it cannot resolve. |
9 |
9 |
10 The main target "run-test" uses the jdk-image as the tested product. There is |
10 The main target `test` uses the jdk-image as the tested product. There is |
11 also an alternate target "exploded-run-test" that uses the exploded image |
11 also an alternate target `exploded-test` that uses the exploded image |
12 instead. Not all tests will run successfully on the exploded image, but using |
12 instead. Not all tests will run successfully on the exploded image, but using |
13 this target can greatly improve rebuild times for certain workflows. |
13 this target can greatly improve rebuild times for certain workflows. |
14 |
14 |
|
15 Previously, `make test` was used invoke an old system for running test, and |
|
16 `make run-test` was used for the new test framework. For backward compatibility |
|
17 with scripts and muscle memory, `run-test` (and variants like |
|
18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system |
|
19 can still be accessed for some time using `cd test && make`. |
|
20 |
15 Some example command-lines: |
21 Some example command-lines: |
16 |
22 |
17 $ make run-test-tier1 |
23 $ make test-tier1 |
18 $ make run-test-jdk_lang JTREG="JOBS=8" |
24 $ make test-jdk_lang JTREG="JOBS=8" |
19 $ make run-test TEST=jdk_lang |
25 $ make test TEST=jdk_lang |
20 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
26 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
21 $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
27 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
22 $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" |
28 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" |
23 $ make exploded-run-test TEST=tier2 |
29 $ make exploded-test TEST=tier2 |
24 |
30 |
25 ### Configuration |
31 ### Configuration |
26 |
32 |
27 To be able to run JTReg tests, `configure` needs to know where to find the |
33 To be able to run JTReg tests, `configure` needs to know where to find the |
28 JTReg test framework. If it is not picked up automatically by configure, use |
34 JTReg test framework. If it is not picked up automatically by configure, use |
31 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME` |
37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME` |
32 environment variable to point to the JTReg home before running `configure`.) |
38 environment variable to point to the JTReg home before running `configure`.) |
33 |
39 |
34 ## Test selection |
40 ## Test selection |
35 |
41 |
36 All functionality is available using the run-test make target. In this use |
42 All functionality is available using the `test` make target. In this use case, |
37 case, the test or tests to be executed is controlled using the `TEST` variable. |
43 the test or tests to be executed is controlled using the `TEST` variable. To |
38 To speed up subsequent test runs with no source code changes, run-test-only can |
44 speed up subsequent test runs with no source code changes, `test-only` can be |
39 be used instead, which do not depend on the source and test image build. |
45 used instead, which do not depend on the source and test image build. |
40 |
46 |
41 For some common top-level tests, direct make targets have been generated. This |
47 For some common top-level tests, direct make targets have been generated. This |
42 includes all JTReg test groups, the hotspot gtest, and custom tests (if |
48 includes all JTReg test groups, the hotspot gtest, and custom tests (if |
43 present). This means that `make run-test-tier1` is equivalent to `make run-test |
49 present). This means that `make test-tier1` is equivalent to `make test |
44 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex |
50 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex |
45 test runs, the `run-test TEST="x"` solution needs to be used. |
51 test runs, the `test TEST="x"` solution needs to be used. |
46 |
52 |
47 The test specifications given in `TEST` is parsed into fully qualified test |
53 The test specifications given in `TEST` is parsed into fully qualified test |
48 descriptors, which clearly and unambigously show which tests will be run. As an |
54 descriptors, which clearly and unambigously show which tests will be run. As an |
49 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 |
55 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 |
50 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 |
56 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 |
95 For Gtest, there is a separate test suite for each JVM variant. The JVM variant |
101 For Gtest, there is a separate test suite for each JVM variant. The JVM variant |
96 is defined by adding `/<variant>` to the test descriptor, e.g. |
102 is defined by adding `/<variant>` to the test descriptor, e.g. |
97 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM |
103 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM |
98 variant present (e.g. server, client). So if you only have the server JVM |
104 variant present (e.g. server, client). So if you only have the server JVM |
99 present, then `gtest:all` will be equivalent to `gtest:all/server`. |
105 present, then `gtest:all` will be equivalent to `gtest:all/server`. |
|
106 |
|
107 ### Special tests |
|
108 |
|
109 A handful of odd tests that are not covered by any other testing framework are |
|
110 accessible using the `special:` test descriptor. Currently, this includes |
|
111 `hotspot-internal`, `failure-handler` and `make`. |
|
112 |
|
113 * Hotspot legacy internal testing (run using `-XX:+ExecuteInternalVMTests`) |
|
114 is run using `special:hotspot-internal` or just `hotspot-internal` as test |
|
115 descriptor, and will only work on a debug JVM. |
|
116 |
|
117 * Failure handler testing is run using `special:failure-handler` or just |
|
118 `failure-handler` as test descriptor. |
|
119 |
|
120 * Tests for the build system, including both makefiles and related |
|
121 functionality, is run using `special:make` or just `make` as test |
|
122 descriptor. This is equivalent to `special:make:all`. |
|
123 |
|
124 A specific make test can be run by supplying it as argument, e.g. |
|
125 `special:make:idea`. As a special syntax, this can also be expressed as |
|
126 `make-idea`, which allows for command lines as `make test-make-idea`. |
100 |
127 |
101 ## Test results and summary |
128 ## Test results and summary |
102 |
129 |
103 At the end of the test run, a summary of all tests run will be presented. This |
130 At the end of the test run, a summary of all tests run will be presented. This |
104 will have a consistent look, regardless of what test suites were used. This is |
131 will have a consistent look, regardless of what test suites were used. This is |
121 The classification of non-passed tests differs a bit between test suites. In |
148 The classification of non-passed tests differs a bit between test suites. In |
122 the summary, ERROR is used as a catch-all for tests that neither passed nor are |
149 the summary, ERROR is used as a catch-all for tests that neither passed nor are |
123 classified as failed by the framework. This might indicate test framework |
150 classified as failed by the framework. This might indicate test framework |
124 error, timeout or other problems. |
151 error, timeout or other problems. |
125 |
152 |
126 In case of test failures, `make run-test` will exit with a non-zero exit value. |
153 In case of test failures, `make test` will exit with a non-zero exit value. |
127 |
154 |
128 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, |
155 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, |
129 where TEST_ID is a path-safe conversion from the fully qualified test |
156 where TEST_ID is a path-safe conversion from the fully qualified test |
130 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is |
157 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is |
131 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the |
158 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the |