38 <p>Some example command-lines:</p> |
38 <p>Some example command-lines:</p> |
39 <pre><code>$ make run-test-tier1 |
39 <pre><code>$ make run-test-tier1 |
40 $ make run-test-jdk_lang JTREG="JOBS=8" |
40 $ make run-test-jdk_lang JTREG="JOBS=8" |
41 $ make run-test TEST=jdk_lang |
41 $ make run-test TEST=jdk_lang |
42 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
42 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" |
43 $ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug" |
43 $ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" |
44 $ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" |
44 $ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" |
45 $ make exploded-run-test TEST=hotspot_tier1</code></pre> |
45 $ make exploded-run-test TEST=hotspot_tier1</code></pre> |
46 <h3 id="configuration">Configuration</h3> |
46 <h3 id="configuration">Configuration</h3> |
47 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> |
47 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> |
48 <h2 id="test-selection">Test selection</h2> |
48 <h2 id="test-selection">Test selection</h2> |
75 <p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> |
75 <p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> |
76 <p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> |
76 <p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> |
77 <h2 id="test-suite-control">Test suite control</h2> |
77 <h2 id="test-suite-control">Test suite control</h2> |
78 <p>It is possible to control various aspects of the test suites using make control variables.</p> |
78 <p>It is possible to control various aspects of the test suites using make control variables.</p> |
79 <p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> |
79 <p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> |
80 <p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
80 <p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
81 <p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
81 <p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
82 <p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
82 <p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
83 <h3 id="jtreg-keywords">JTReg keywords</h3> |
83 <h3 id="jtreg-keywords">JTReg keywords</h3> |
84 <h4 id="jobs">JOBS</h4> |
84 <h4 id="jobs">JOBS</h4> |
85 <p>The test concurrency (<code>-concurrency</code>).</p> |
85 <p>The test concurrency (<code>-concurrency</code>).</p> |
86 <p>Defaults to TEST_JOBS (if set by <code>--with-test-jobs=</code>), otherwise it defaults to JOBS, except for Hotspot, where the default is <em>number of CPU cores/2</em>, but never more than 12.</p> |
86 <p>Defaults to TEST_JOBS (if set by <code>--with-test-jobs=</code>), otherwise it defaults to JOBS, except for Hotspot, where the default is <em>number of CPU cores/2</em>, but never more than 12.</p> |