doc/testing.md
author alitvinov
Mon, 27 Aug 2018 18:29:07 +0100
changeset 51913 451ec22b1a46
parent 50885 7c728fa9d1af
child 52342 9341b077bd55
child 56953 2ea07b19b446
permissions -rw-r--r--
8201818: [macosx] Printing attributes break page size set via "java.awt.print.Book" object Reviewed-by: prr, psadhukhan
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
     1
% Testing the JDK
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     2
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     3
## Using the run-test framework
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     4
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     5
This new way of running tests is developer-centric. It assumes that you have
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
     6
built a JDK locally and want to test it. Running common test targets is simple,
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     7
and more complex ad-hoc combination of tests is possible. The user interface is
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     8
forgiving, and clearly report errors it cannot resolve.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
     9
45266
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    10
The main target "run-test" uses the jdk-image as the tested product. There is
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    11
also an alternate target "exploded-run-test" that uses the exploded image
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    12
instead. Not all tests will run successfully on the exploded image, but using
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    13
this target can greatly improve rebuild times for certain workflows.
29347d0c7122 8180651: Make target to run tests on exploded image
erikj
parents: 44983
diff changeset
    14
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    15
Some example command-lines:
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    16
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    17
    $ make run-test-tier1
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    18
    $ make run-test-jdk_lang JTREG="JOBS=8"
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    19
    $ make run-test TEST=jdk_lang
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    20
    $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    21
    $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    22
    $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    23
    $ make exploded-run-test TEST=tier2
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    24
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    25
### Configuration
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    26
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    27
To be able to run JTReg tests, `configure` needs to know where to find the
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    28
JTReg test framework. If it is not picked up automatically by configure, use
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    29
the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    30
Note that this option should point to the JTReg home, i.e. the top directory,
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    31
containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    32
environment variable to point to the JTReg home before running `configure`.)
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    33
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    34
## Test selection
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    35
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    36
All functionality is available using the run-test make target. In this use
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    37
case, the test or tests to be executed is controlled using the `TEST` variable.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    38
To speed up subsequent test runs with no source code changes, run-test-only can
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    39
be used instead, which do not depend on the source and test image build.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    40
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    41
For some common top-level tests, direct make targets have been generated. This
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    42
includes all JTReg test groups, the hotspot gtest, and custom tests (if
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    43
present). This means that `make run-test-tier1` is equivalent to `make run-test
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    44
TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    45
test runs, the `run-test TEST="x"` solution needs to be used.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    46
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    47
The test specifications given in `TEST` is parsed into fully qualified test
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    48
descriptors, which clearly and unambigously show which tests will be run. As an
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    49
example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    50
jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    51
jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    52
always submit a list of fully qualified test descriptors in the `TEST` variable
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    53
if you want to shortcut the parser.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    54
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    55
### JTReg
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    56
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    57
JTReg tests can be selected either by picking a JTReg test group, or a selection
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    58
of files or directories containing JTReg tests.
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    59
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    60
JTReg test groups can be specified either without a test root, e.g. `:tier1`
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    61
(or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    62
`test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    63
root can be specified either as an absolute path, or a path relative to the
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
    64
JDK top directory, or the `test` directory. For simplicity, the hotspot
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    65
JTReg test root, which really is `hotspot/jtreg` can be abbreviated as
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    66
just `hotspot`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    67
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    68
When specified without a test root, all matching groups from all test roots
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    69
will be added. Otherwise, only the group from the specified test root will be
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    70
added.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    71
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
    72
Individual JTReg tests or directories containing JTReg tests can also be
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    73
specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    74
`hotspot/jtreg/native_sanity`. Just like for test root selection, you can
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    75
either specify an absolute path (which can even point to JTReg tests outside
50885
7c728fa9d1af 8205956: Fix usage of "OpenJDK" in build and test instructions
mr
parents: 50267
diff changeset
    76
the source tree), or a path relative to either the JDK top directory or the
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    77
`test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    78
well.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    79
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    80
As long as the test groups or test paths can be uniquely resolved, you do not
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    81
need to enter the `jtreg:` prefix. If this is not possible, or if you want to
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    82
use a fully qualified test descriptor, add `jtreg:`, e.g.
50267
1582de22e3a1 8198323: testing.md not updated for repository layout change
ihse
parents: 49293
diff changeset
    83
`jtreg:test/hotspot/jtreg/native_sanity`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    84
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    85
### Gtest
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    86
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    87
Since the Hotspot Gtest suite is so quick, the default is to run all tests.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    88
This is specified by just `gtest`, or as a fully qualified test descriptor
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    89
`gtest:all`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    90
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    91
If you want, you can single out an individual test or a group of tests, for
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    92
instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    93
can be particularly useful if you want to run a shaky test repeatedly.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
    94
48058
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
    95
For Gtest, there is a separate test suite for each JVM variant. The JVM variant
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
    96
is defined by adding `/<variant>` to the test descriptor, e.g.
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
    97
`gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
    98
variant present (e.g. server, client). So if you only have the server JVM
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
    99
present, then `gtest:all` will be equivalent to `gtest:all/server`.
c5eefa465e37 8192995: run-test gtest should use all jvm variants, not just "server"
ihse
parents: 47216
diff changeset
   100
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   101
## Test results and summary
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   102
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   103
At the end of the test run, a summary of all tests run will be presented. This
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   104
will have a consistent look, regardless of what test suites were used. This is
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   105
a sample summary:
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   106
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   107
    ==============================
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   108
    Test summary
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   109
    ==============================
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   110
       TEST                                          TOTAL  PASS  FAIL ERROR
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   111
    >> jtreg:jdk/test:tier1                           1867  1865     2     0 <<
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   112
       jtreg:langtools/test:tier1                     4711  4711     0     0
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   113
       jtreg:nashorn/test:tier1                        133   133     0     0
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   114
    ==============================
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   115
    TEST FAILURE
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   116
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   117
Tests where the number of TOTAL tests does not equal the number of PASSed tests
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   118
will be considered a test failure. These are marked with the `>> ... <<` marker
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   119
for easy identification.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   120
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   121
The classification of non-passed tests differs a bit between test suites. In
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   122
the summary, ERROR is used as a catch-all for tests that neither passed nor are
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   123
classified as failed by the framework. This might indicate test framework
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   124
error, timeout or other problems.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   125
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   126
In case of test failures, `make run-test` will exit with a non-zero exit value.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   127
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   128
All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   129
where TEST_ID is a path-safe conversion from the fully qualified test
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   130
descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   131
`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   132
test run.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   133
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   134
Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   135
some frameworks, this directory might contain information that is useful in
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   136
determining the cause of a failed test.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   137
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   138
## Test suite control
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   139
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   140
It is possible to control various aspects of the test suites using make control
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   141
variables.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   142
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   143
These variables use a keyword=value approach to allow multiple values to be
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   144
set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   145
concurrency level to 1 and the timeout factor to 8. This is equivalent to
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   146
setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   147
the `JTREG` variable is parsed and verified for correctness, so
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   148
`JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   149
pass unnoticed.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   150
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   151
To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   152
normally eats `;`, the recommended usage is to write the assignment inside
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   153
qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
49293
9e1c1889136e 8200180: fix a typo in run-test framework documentation
iignatyev
parents: 48058
diff changeset
   154
as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   155
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   156
(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   157
Also, as a special technique, the string `%20` will be replaced with space for
49293
9e1c1889136e 8200180: fix a typo in run-test framework documentation
iignatyev
parents: 48058
diff changeset
   158
certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   159
This can be useful if you have layers of scripts and have trouble getting
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   160
proper quoting of command line arguments through.)
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   161
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   162
As far as possible, the names of the keywords have been standardized between
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   163
test suites.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   164
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   165
### JTReg keywords
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   166
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   167
#### JOBS
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   168
The test concurrency (`-concurrency`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   169
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   170
Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   171
JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   172
never more than 12.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   173
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   174
#### TIMEOUT
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   175
The timeout factor (`-timeoutFactor`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   176
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   177
Defaults to 4.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   178
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   179
#### TEST_MODE
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   180
The test mode (`-agentvm`, `-samevm` or `-othervm`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   181
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   182
Defaults to `-agentvm`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   183
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   184
#### ASSERT
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   185
Enable asserts (`-ea -esa`, or none).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   186
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   187
Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   188
for hotspot.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   189
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   190
#### VERBOSE
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   191
The verbosity level (`-verbose`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   192
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   193
Defaults to `fail,error,summary`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   194
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   195
#### RETAIN
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   196
What test data to retain (`-retain`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   197
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   198
Defaults to `fail,error`.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   199
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   200
#### MAX_MEM
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   201
Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   202
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   203
Limit memory consumption for JTReg test framework and VM under test. Set to 0
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   204
to disable the limits.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   205
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   206
Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   207
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   208
#### OPTIONS
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   209
Additional options to the JTReg test framework.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   210
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   211
Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   212
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   213
#### JAVA_OPTIONS
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   214
Additional Java options to JTReg (`-javaoption`).
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   215
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   216
#### VM_OPTIONS
45618
74f490ba783a 8179537: Update testing.md for more clarity regarding JTReg configuration
ihse
parents: 44983
diff changeset
   217
Additional VM options to JTReg (`-vmoption`).
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   218
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   219
### Gtest keywords
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   220
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   221
#### REPEAT
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   222
The number of times to repeat the tests (`--gtest_repeat`).
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   223
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   224
Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   225
combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   226
problem.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   227
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   228
#### OPTIONS
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   229
Additional options to the Gtest test framework.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   230
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   231
Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   232
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   233
---
44983
4e216c0475ee 8179453: Add a proper SetupProcessMarkdown
ihse
parents: 44511
diff changeset
   234
# Override some definitions in the global css file that are not optimal for
4e216c0475ee 8179453: Add a proper SetupProcessMarkdown
ihse
parents: 44511
diff changeset
   235
# this document.
44511
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   236
header-includes:
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   237
 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
f3c27401a860 8177955: Add testing documentation
ihse
parents:
diff changeset
   238
---