To be able to run JTReg tests, configure needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the --with-jtreg=<path to jtreg home> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing lib/jtreg.jar etc. (An alternative is to set the JT_HOME environment variable to point to the JTReg home before running configure.)
To be able to run microbenchmarks, configure needs to know where to find the JMH dependency. Use --with-jmh=<path to JMH jars> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running sh make/devkit/createJMHBundle.sh, after which --with-jmh=build/jmh/jars should work.
All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the TEST variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.
For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that make run-test-tier1 is equivalent to make run-test TEST="tier1", but the latter is more tab-completion friendly. For more complex test runs, the run-test TEST="x" solution needs to be used.
Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just gtest, or as a fully qualified test descriptor gtest:all.
If you want, you can single out an individual test or a group of tests, for instance gtest:LogDecorations or gtest:LogDecorations.level_test_vm. This can be particularly useful if you want to run a shaky test repeatedly.
For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding /<variant> to the test descriptor, e.g. gtest:Log/client. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then gtest:all will be equivalent to gtest:all/server.
Which microbenchmarks to run is selected using a regular expression following the micro: test descriptor, e.g., micro:java.lang.reflect. This delegates the test selection to JMH, meaning package name, class name and even benchmark method names can be used to select tests.
Using special characters like | in the regular expression is possible, but needs to be escaped multiple times: micro:ArrayCopy\\\\\|reflect.
At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:
==============================
@@ -118,5 +125,22 @@
OPTIONS
Additional options to the Gtest test framework.
Use GTEST="OPTIONS=--help" to see all available Gtest options.
+Microbenchmark keywords
+FORK
+Override the number of benchmark forks to spawn. Same as specifying -f <num>.
+ITER
+Number of measurement iterations per fork. Same as specifying -i <num>.
+TIME
+Amount of time to spend in each measurement iteration, in seconds. Same as specifying -r <num>
+WARMUP_ITER
+Number of warmup iterations to run before the measurement phase in each fork. Same as specifying -wi <num>.
+WARMUP_TIME
+Amount of time to spend in each warmup iteration. Same as specifying -w <num>.
+RESULTS_FORMAT
+Specify to have the test run save a log of the values. Accepts the same values as -rff, i.e., text, csv, scsv, json, or latex.
+VM_OPTIONS
+Additional VM arguments to provide to forked off VMs. Same as -jvmArgs <args>
+OPTIONS
+Additional arguments to send to JMH.