doc/testing.md
author redestad
Thu, 14 Nov 2019 15:24:35 +0100
changeset 59081 95a99e617f28
parent 58930 a4ddd1667c72
permissions -rw-r--r--
8234003: Improve IndexSet iteration Reviewed-by: neliasso, thartmann
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
     1
% Testing the JDK
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     2
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
     3
## Using "make test" (the run-test framework)
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     4
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     5
This new way of running tests is developer-centric. It assumes that you have
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
     6
built a JDK locally and want to test it. Running common test targets is simple,
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     7
and more complex ad-hoc combination of tests is possible. The user interface is
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     8
forgiving, and clearly report errors it cannot resolve.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     9
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    10
The main target `test` uses the jdk-image as the tested product. There is
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    11
also an alternate target `exploded-test` that uses the exploded image
45266
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    12
instead. Not all tests will run successfully on the exploded image, but using
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    13
this target can greatly improve rebuild times for certain workflows.
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    14
54268
0cf2c87ce79a 8221357: Update test documentation by deleting "cd test && make"
dholmes
parents: 53996
diff changeset
    15
Previously, `make test` was used to invoke an old system for running tests, and
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    16
`make run-test` was used for the new test framework. For backward compatibility
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    17
with scripts and muscle memory, `run-test` (and variants like
54268
0cf2c87ce79a 8221357: Update test documentation by deleting "cd test && make"
dholmes
parents: 53996
diff changeset
    18
`exploded-run-test` or `run-test-tier1`) are kept as aliases.
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    19
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    20
Some example command-lines:
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    21
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    22
    $ make test-tier1
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    23
    $ make test-jdk_lang JTREG="JOBS=8"
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    24
    $ make test TEST=jdk_lang
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    25
    $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
57881
c414c554b38b 8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents: 57493
diff changeset
    26
    $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT_FACTOR=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    27
    $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
52595
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
    28
    $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    29
    $ make exploded-test TEST=tier2
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    30
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    31
### Configuration
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    32
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    33
To be able to run JTReg tests, `configure` needs to know where to find the
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    34
JTReg test framework. If it is not picked up automatically by configure, use
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    35
the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    36
Note that this option should point to the JTReg home, i.e. the top directory,
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    37
containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    38
environment variable to point to the JTReg home before running `configure`.)
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    39
52595
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
    40
To be able to run microbenchmarks, `configure` needs to know where to find
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
    41
the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
    42
containing the core JMH and transitive dependencies. The recommended dependencies
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
    43
can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which
52595
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
    44
`--with-jmh=build/jmh/jars` should work.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
    45
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    46
## Test selection
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    47
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    48
All functionality is available using the `test` make target. In this use case,
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    49
the test or tests to be executed is controlled using the `TEST` variable. To
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    50
speed up subsequent test runs with no source code changes, `test-only` can be
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    51
used instead, which do not depend on the source and test image build.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    52
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    53
For some common top-level tests, direct make targets have been generated. This
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    54
includes all JTReg test groups, the hotspot gtest, and custom tests (if
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    55
present). This means that `make test-tier1` is equivalent to `make test
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    56
TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
    57
test runs, the `test TEST="x"` solution needs to be used.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    58
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    59
The test specifications given in `TEST` is parsed into fully qualified test
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    60
descriptors, which clearly and unambigously show which tests will be run. As an
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    61
example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    62
jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    63
jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    64
always submit a list of fully qualified test descriptors in the `TEST` variable
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    65
if you want to shortcut the parser.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    66
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    67
### JTReg
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    68
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    69
JTReg tests can be selected either by picking a JTReg test group, or a selection
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    70
of files or directories containing JTReg tests.
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    71
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    72
JTReg test groups can be specified either without a test root, e.g. `:tier1`
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    73
(or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    74
`test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    75
root can be specified either as an absolute path, or a path relative to the
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
    76
JDK top directory, or the `test` directory. For simplicity, the hotspot
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    77
JTReg test root, which really is `hotspot/jtreg` can be abbreviated as
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    78
just `hotspot`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    79
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    80
When specified without a test root, all matching groups from all test roots
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    81
will be added. Otherwise, only the group from the specified test root will be
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    82
added.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    83
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    84
Individual JTReg tests or directories containing JTReg tests can also be
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    85
specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    86
`hotspot/jtreg/native_sanity`. Just like for test root selection, you can
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    87
either specify an absolute path (which can even point to JTReg tests outside
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
    88
the source tree), or a path relative to either the JDK top directory or the
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    89
`test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    90
well.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    91
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    92
As long as the test groups or test paths can be uniquely resolved, you do not
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    93
need to enter the `jtreg:` prefix. If this is not possible, or if you want to
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    94
use a fully qualified test descriptor, add `jtreg:`, e.g.
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    95
`jtreg:test/hotspot/jtreg/native_sanity`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    96
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    97
### Gtest
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    98
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    99
Since the Hotspot Gtest suite is so quick, the default is to run all tests.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   100
This is specified by just `gtest`, or as a fully qualified test descriptor
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   101
`gtest:all`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   102
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   103
If you want, you can single out an individual test or a group of tests, for
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   104
instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   105
can be particularly useful if you want to run a shaky test repeatedly.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   106
48058
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   107
For Gtest, there is a separate test suite for each JVM variant. The JVM variant
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   108
is defined by adding `/<variant>` to the test descriptor, e.g.
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   109
`gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   110
variant present (e.g. server, client). So if you only have the server JVM
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   111
present, then `gtest:all` will be equivalent to `gtest:all/server`.
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   112
52595
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   113
### Microbenchmarks
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   114
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   115
Which microbenchmarks to run is selected using a regular expression
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   116
following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   117
delegates the test selection to JMH, meaning package name, class name and even
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   118
benchmark method names can be used to select tests.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   119
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   120
Using special characters like `|` in the regular expression is possible, but
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   121
needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   122
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   123
### Special tests
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   124
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   125
A handful of odd tests that are not covered by any other testing framework are
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   126
accessible using the `special:` test descriptor. Currently, this includes
52396
e292e94b448a 8213058: remove ExecuteInternalVMTests and VerboseInternalVMTests flags
iignatyev
parents: 52342
diff changeset
   127
`failure-handler` and `make`.
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   128
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   129
  * Failure handler testing is run using `special:failure-handler` or just
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   130
    `failure-handler` as test descriptor.
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   131
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   132
  * Tests for the build system, including both makefiles and related
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   133
    functionality, is run using `special:make` or just `make` as test
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   134
    descriptor. This is equivalent to `special:make:all`.
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   135
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   136
    A specific make test can be run by supplying it as argument, e.g.
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   137
    `special:make:idea`. As a special syntax, this can also be expressed as
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   138
    `make-idea`, which allows for command lines as `make test-make-idea`.
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   139
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   140
## Test results and summary
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   141
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   142
At the end of the test run, a summary of all tests run will be presented. This
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   143
will have a consistent look, regardless of what test suites were used. This is
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   144
a sample summary:
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   145
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   146
    ==============================
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   147
    Test summary
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   148
    ==============================
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   149
       TEST                                          TOTAL  PASS  FAIL ERROR
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   150
    >> jtreg:jdk/test:tier1                           1867  1865     2     0 <<
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   151
       jtreg:langtools/test:tier1                     4711  4711     0     0
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   152
       jtreg:nashorn/test:tier1                        133   133     0     0
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   153
    ==============================
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   154
    TEST FAILURE
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   155
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   156
Tests where the number of TOTAL tests does not equal the number of PASSed tests
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   157
will be considered a test failure. These are marked with the `>> ... <<` marker
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   158
for easy identification.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   159
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   160
The classification of non-passed tests differs a bit between test suites. In
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   161
the summary, ERROR is used as a catch-all for tests that neither passed nor are
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   162
classified as failed by the framework. This might indicate test framework
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   163
error, timeout or other problems.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   164
52342
9341b077bd55 8210958: Rename "make run-test" to "make test"
ihse
parents: 50885
diff changeset
   165
In case of test failures, `make test` will exit with a non-zero exit value.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   166
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   167
All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   168
where TEST_ID is a path-safe conversion from the fully qualified test
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   169
descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   170
`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   171
test run.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   172
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   173
Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   174
some frameworks, this directory might contain information that is useful in
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   175
determining the cause of a failed test.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   176
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   177
## Test suite control
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   178
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   179
It is possible to control various aspects of the test suites using make control
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   180
variables.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   181
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   182
These variables use a keyword=value approach to allow multiple values to be
57881
c414c554b38b 8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents: 57493
diff changeset
   183
set. So, for instance, `JTREG="JOBS=1;TIMEOUT_FACTOR=8"` will set the JTReg
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   184
concurrency level to 1 and the timeout factor to 8. This is equivalent to
57881
c414c554b38b 8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents: 57493
diff changeset
   185
setting `JTREG_JOBS=1 JTREG_TIMEOUT_FACTOR=8`, but using the keyword format means that
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   186
the `JTREG` variable is parsed and verified for correctness, so
57881
c414c554b38b 8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents: 57493
diff changeset
   187
`JTREG="TMIEOUT_FACTOR=8"` would give an error, while `JTREG_TMIEOUT_FACTOR=8` would just
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   188
pass unnoticed.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   189
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   190
To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   191
normally eats `;`, the recommended usage is to write the assignment inside
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   192
qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
49293
9e1c1889136e 8200180: fix a typo in run-test framework documentation
iignatyev
parents: 48058
diff changeset
   193
as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   194
57881
c414c554b38b 8230079: Update test document by changing "TIMEOUT" to "TIMEOUT_FACTOR"
erikj
parents: 57493
diff changeset
   195
(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT_FACTOR=8`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   196
Also, as a special technique, the string `%20` will be replaced with space for
49293
9e1c1889136e 8200180: fix a typo in run-test framework documentation
iignatyev
parents: 48058
diff changeset
   197
certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   198
This can be useful if you have layers of scripts and have trouble getting
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   199
proper quoting of command line arguments through.)
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   200
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   201
As far as possible, the names of the keywords have been standardized between
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   202
test suites.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   203
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   204
### General keywords (TEST_OPTS)
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   205
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   206
Some keywords are valid across different test suites. If you want to run
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   207
tests from multiple test suites, or just don't want to care which test suite specific
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   208
control variable to use, then you can use the general TEST_OPTS control variable.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   209
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   210
There are also some keywords that applies globally to the test runner system,
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   211
not to any specific test suites. These are also available as TEST_OPTS keywords.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   212
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   213
#### JOBS
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   214
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   215
Currently only applies to JTReg.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   216
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   217
#### TIMEOUT_FACTOR
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   218
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   219
Currently only applies to JTReg.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   220
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   221
#### VM_OPTIONS
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   222
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   223
Applies to JTReg, GTest and Micro.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   224
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   225
#### JAVA_OPTIONS
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   226
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   227
Applies to JTReg, GTest and Micro.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   228
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   229
#### AOT_MODULES
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   230
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   231
Applies to JTReg and GTest.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   232
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   233
#### JCOV
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   234
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   235
This keywords applies globally to the test runner system. If set to `true`, it
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   236
enables JCov coverage reporting for all tests run. To be useful, the JDK under
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   237
test must be run with a JDK built with JCov instrumentation (`configure
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   238
--with-jcov=<path to directory containing lib/jcov.jar>`, `make jcov-image`).
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   239
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   240
The simplest way to run tests with JCov coverage report is to use the special
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   241
target `jcov-test` instead of `test`, e.g. `make jcov-test TEST=jdk_lang`. This
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   242
will make sure the JCov image is built, and that JCov reporting is enabled.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   243
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   244
The JCov report is stored in `build/$BUILD/test-results/jcov-output`.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   245
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   246
Please note that running with JCov reporting can be very memory intensive.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   247
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   248
### JTReg keywords
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   249
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   250
#### JOBS
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   251
The test concurrency (`-concurrency`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   252
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   253
Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
53996
35a30b432396 8219906: Update test documentation with default test jobs settings
aoqi
parents: 53464
diff changeset
   254
JOBS, except for Hotspot, where the default is *number of CPU cores/2* (for
35a30b432396 8219906: Update test documentation with default test jobs settings
aoqi
parents: 53464
diff changeset
   255
sparc, if more than 16 cpus, then *number of CPU cores/5*, otherwise *number of
35a30b432396 8219906: Update test documentation with default test jobs settings
aoqi
parents: 53464
diff changeset
   256
CPU cores/4*), but never more than *memory size in GB/2*.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   257
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   258
#### TIMEOUT_FACTOR
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   259
The timeout factor (`-timeoutFactor`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   260
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   261
Defaults to 4.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   262
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   263
#### TEST_MODE
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   264
The test mode (`-agentvm`, `-samevm` or `-othervm`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   265
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   266
Defaults to `-agentvm`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   267
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   268
#### ASSERT
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   269
Enable asserts (`-ea -esa`, or none).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   270
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   271
Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   272
for hotspot.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   273
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   274
#### VERBOSE
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   275
The verbosity level (`-verbose`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   276
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   277
Defaults to `fail,error,summary`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   278
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   279
#### RETAIN
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   280
What test data to retain (`-retain`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   281
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   282
Defaults to `fail,error`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   283
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   284
#### MAX_MEM
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   285
Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   286
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   287
Limit memory consumption for JTReg test framework and VM under test. Set to 0
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   288
to disable the limits.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   289
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   290
Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   291
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   292
#### KEYWORDS
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   293
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   294
JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   295
spaces and special characters (like `!`) are properly quoted. To avoid some
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   296
issues, the special value `%20` can be used instead of space.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   297
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   298
#### EXTRA_PROBLEM_LISTS
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   299
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   300
Use additional problem lists file or files, in addition to the default
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   301
ProblemList.txt located at the JTReg test roots.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   302
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   303
If multiple file names are specified, they should be separated by space (or, to
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   304
help avoid quoting issues, the special value `%20`).
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   305
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   306
The file names should be either absolute, or relative to the JTReg test root of
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   307
the tests to be run.
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   308
57493
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   309
#### RUN_PROBLEM_LISTS
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   310
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   311
Use the problem lists to select tests instead of excluding them.
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   312
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   313
Set to `true` or `false`.
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   314
If `true`, JTReg will use `-match:` option, otherwise `-exclude:` will be used.
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   315
Default is `false`.
b95ebdbf68ca 8226910: make it possible to use jtreg's -match via run-test framework
iignatyev
parents: 54570
diff changeset
   316
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   317
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   318
#### OPTIONS
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   319
Additional options to the JTReg test framework.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   320
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   321
Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   322
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   323
#### JAVA_OPTIONS
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   324
Additional Java options to JTReg (`-javaoption`).
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   325
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   326
#### VM_OPTIONS
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   327
Additional VM options to JTReg (`-vmoption`).
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   328
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   329
#### AOT_MODULES
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   330
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   331
Generate AOT modules before testing for the specified module, or set of
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   332
modules. If multiple modules are specified, they should be separated by space
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   333
(or, to help avoid quoting issues, the special value `%20`).
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   334
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   335
### Gtest keywords
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   336
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   337
#### REPEAT
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   338
The number of times to repeat the tests (`--gtest_repeat`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   339
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   340
Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   341
combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   342
problem.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   343
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   344
#### OPTIONS
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   345
Additional options to the Gtest test framework.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   346
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   347
Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   348
53464
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   349
#### AOT_MODULES
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   350
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   351
Generate AOT modules before testing for the specified module, or set of
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   352
modules. If multiple modules are specified, they should be separated by space
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   353
(or, to help avoid quoting issues, the special value `%20`).
650527b39f00 8217634: RunTest documentation and usability update
ihse
parents: 52595
diff changeset
   354
52595
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   355
### Microbenchmark keywords
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   356
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   357
#### FORK
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   358
Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   359
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   360
#### ITER
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   361
Number of measurement iterations per fork. Same as specifying `-i <num>`.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   362
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   363
#### TIME
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   364
Amount of time to spend in each measurement iteration, in seconds. Same as
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   365
specifying `-r <num>`
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   366
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   367
#### WARMUP_ITER
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   368
Number of warmup iterations to run before the measurement phase in each fork.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   369
Same as specifying `-wi <num>`.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   370
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   371
#### WARMUP_TIME
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   372
Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   373
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   374
#### RESULTS_FORMAT
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   375
Specify to have the test run save a log of the values. Accepts the same values
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   376
as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   377
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   378
#### VM_OPTIONS
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   379
Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   380
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   381
#### OPTIONS
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   382
Additional arguments to send to JMH.
16609197022c 8061281: Microbenchmark suite build support, directory layout and sample benchmarks
redestad
parents: 52396
diff changeset
   383
54473
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   384
## Notes for Specific Tests
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   385
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   386
### Docker Tests
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   387
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   388
Docker tests with default parameters may fail on systems with glibc versions not
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   389
compatible with the one used in the default docker image (e.g., Oracle Linux 7.6 for x86).
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   390
For example, they pass on Ubuntu 16.04 but fail on Ubuntu 18.04 if run like this on x86:
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   391
54525
5ae4d3f46537 8222299: [TESTBUG] move hotspot container tests to hotspot/containers
mseledtsov
parents: 54473
diff changeset
   392
    $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker"
54473
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   393
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   394
To run these tests correctly, additional parameters for the correct docker image are
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   395
required on Ubuntu 18.04 by using `JAVA_OPTIONS`.
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   396
54525
5ae4d3f46537 8222299: [TESTBUG] move hotspot container tests to hotspot/containers
mseledtsov
parents: 54473
diff changeset
   397
    $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu -Djdk.test.docker.image.version=latest"
54473
e437ad5643d6 8221894: Add comments for docker tests in the test doc
jiefu
parents: 54268
diff changeset
   398
54570
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   399
### Non-US locale
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   400
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   401
If your locale is non-US, some tests are likely to fail. To work around this you can
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   402
set the locale to US. On Unix platforms simply setting `LANG="en_US"` in the
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   403
environment before running tests should work. On Windows, setting
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   404
`JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US"` helps for most, but not all test cases.
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   405
For example:
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   406
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   407
    $ export LANG="en_US" && make test TEST=...
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   408
    $ make test JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US" TEST=...
93b702d2a0cb 8222444: Add a suggestion for non-US locale in the test doc
erikj
parents: 54525
diff changeset
   409
58296
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   410
### PKCS11 Tests
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   411
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   412
It is highly recommended to use the latest NSS version when running PKCS11 tests.
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   413
Improper NSS version may lead to unexpected failures which are hard to diagnose.
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   414
For example, sun/security/pkcs11/Secmod/AddTrustedCert.java may fail on Ubuntu
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   415
18.04 with the default NSS version in the system.
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   416
To run these tests correctly, the system property `test.nss.lib.paths` is required
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   417
on Ubuntu 18.04 to specify the alternative NSS lib directories.
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   418
For example:
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   419
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   420
    $ make test TEST="jtreg:sun/security/pkcs11/Secmod/AddTrustedCert.java" JTREG="JAVA_OPTIONS=-Dtest.nss.lib.paths=/path/to/your/latest/NSS-libs"
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   421
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   422
For more notes about the PKCS11 tests, please refer to test/jdk/sun/security/pkcs11/README.
7b534a5088d9 8231351: Add notes for PKCS11 tests in the test doc
erikj
parents: 57881
diff changeset
   423
58930
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   424
### Client UI Tests
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   425
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   426
Some Client UI tests use key sequences which may be reserved by the operating
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   427
system. Usually that causes the test failure. So it is highly recommended to disable
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   428
system key shortcuts prior testing. The steps to access and disable system key shortcuts
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   429
for various platforms are provided below.
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   430
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   431
#### MacOS
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   432
Choose Apple menu; System Preferences, click Keyboard, then click Shortcuts;
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   433
select or deselect desired shortcut.
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   434
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   435
For example, test/jdk/javax/swing/TooltipManager/JMenuItemToolTipKeyBindingsTest/JMenuItemToolTipKeyBindingsTest.java fails
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   436
on MacOS because it uses `CTRL + F1` key sequence to show or hide tooltip message
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   437
but the key combination is reserved by the operating system. To run the test correctly
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   438
the default global key shortcut should be disabled using the steps described above, and then deselect
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   439
"Turn keyboard access on or off" option which is responsible for `CTRL + F1` combination.
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   440
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   441
#### Linux
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   442
Open the Activities overview and start typing Settings; Choose Settings, click Devices,
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   443
then click Keyboard; set or override desired shortcut.
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   444
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   445
#### Windows
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   446
Type `gpedit` in the Search and then click Edit group policy; navigate to
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   447
User Configuration -> Administrative Templates -> Windows Components -> File Explorer;
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   448
in the right-side pane look for "Turn off Windows key hotkeys" and double click on it;
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   449
enable or disable hotkeys.
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   450
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   451
Note: restart is required to make the settings take effect.
a4ddd1667c72 8232880: Update test documentation with additional settings for client UI tooltip tests
dmarkov
parents: 58296
diff changeset
   452
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   453
---
44983
4e216c0475ee 8179453: Add a proper SetupProcessMarkdown
ihse
parents: 44511
diff changeset
   454
# Override some definitions in the global css file that are not optimal for
4e216c0475ee 8179453: Add a proper SetupProcessMarkdown
ihse
parents: 44511
diff changeset
   455
# this document.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   456
header-includes:
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   457
 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   458
---