52 <p>Some example command-lines:</p> |
52 <p>Some example command-lines:</p> |
53 <pre><code>$ make test-tier1 |
53 <pre><code>$ make test-tier1 |
54 $ make test-jdk_lang JTREG="JOBS=8" |
54 $ make test-jdk_lang JTREG="JOBS=8" |
55 $ make test TEST=jdk_lang |
55 $ make test TEST=jdk_lang |
56 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
56 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
57 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
57 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT_FACTOR=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
58 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" |
58 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" |
59 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2" |
59 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2" |
60 $ make exploded-test TEST=tier2</code></pre> |
60 $ make exploded-test TEST=tier2</code></pre> |
61 <h3 id="configuration">Configuration</h3> |
61 <h3 id="configuration">Configuration</h3> |
62 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> |
62 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> |
101 <p>In case of test failures, <code>make test</code> will exit with a non-zero exit value.</p> |
101 <p>In case of test failures, <code>make test</code> will exit with a non-zero exit value.</p> |
102 <p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> |
102 <p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> |
103 <p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> |
103 <p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> |
104 <h2 id="test-suite-control">Test suite control</h2> |
104 <h2 id="test-suite-control">Test suite control</h2> |
105 <p>It is possible to control various aspects of the test suites using make control variables.</p> |
105 <p>It is possible to control various aspects of the test suites using make control variables.</p> |
106 <p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> |
106 <p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT_FACTOR=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT_FACTOR=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT_FACTOR=8"</code> would give an error, while <code>JTREG_TMIEOUT_FACTOR=8</code> would just pass unnoticed.</p> |
107 <p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
107 <p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
108 <p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
108 <p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT_FACTOR=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
109 <p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
109 <p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
110 <h3 id="general-keywords-test_opts">General keywords (TEST_OPTS)</h3> |
110 <h3 id="general-keywords-test_opts">General keywords (TEST_OPTS)</h3> |
111 <p>Some keywords are valid across different test suites. If you want to run tests from multiple test suites, or just don't want to care which test suite specific control variable to use, then you can use the general TEST_OPTS control variable.</p> |
111 <p>Some keywords are valid across different test suites. If you want to run tests from multiple test suites, or just don't want to care which test suite specific control variable to use, then you can use the general TEST_OPTS control variable.</p> |
112 <p>There are also some keywords that applies globally to the test runner system, not to any specific test suites. These are also available as TEST_OPTS keywords.</p> |
112 <p>There are also some keywords that applies globally to the test runner system, not to any specific test suites. These are also available as TEST_OPTS keywords.</p> |
113 <h4 id="jobs">JOBS</h4> |
113 <h4 id="jobs">JOBS</h4> |