100 <p>It is possible to control various aspects of the test suites using make control variables.</p> |
101 <p>It is possible to control various aspects of the test suites using make control variables.</p> |
101 <p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> |
102 <p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTReg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> |
102 <p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
103 <p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> |
103 <p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
104 <p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> |
104 <p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
105 <p>As far as possible, the names of the keywords have been standardized between test suites.</p> |
|
106 <h3 id="general-keywords-test_opts">General keywords (TEST_OPTS)</h3> |
|
107 <p>Some keywords are valid across different test suites. If you want to run tests from multiple test suites, or just don't want to care which test suite specific control variable to use, then you can use the general TEST_OPTS control variable.</p> |
|
108 <p>There are also some keywords that applies globally to the test runner system, not to any specific test suites. These are also available as TEST_OPTS keywords.</p> |
|
109 <h4 id="jobs">JOBS</h4> |
|
110 <p>Currently only applies to JTReg.</p> |
|
111 <h4 id="timeout_factor">TIMEOUT_FACTOR</h4> |
|
112 <p>Currently only applies to JTReg.</p> |
|
113 <h4 id="vm_options">VM_OPTIONS</h4> |
|
114 <p>Applies to JTReg, GTest and Micro.</p> |
|
115 <h4 id="java_options">JAVA_OPTIONS</h4> |
|
116 <p>Applies to JTReg, GTest and Micro.</p> |
|
117 <h4 id="aot_modules">AOT_MODULES</h4> |
|
118 <p>Applies to JTReg and GTest.</p> |
|
119 <h4 id="jcov">JCOV</h4> |
|
120 <p>This keywords applies globally to the test runner system. If set to <code>true</code>, it enables JCov coverage reporting for all tests run. To be useful, the JDK under test must be run with a JDK built with JCov instrumentation (<code>configure --with-jcov=<path to directory containing lib/jcov.jar></code>, <code>make jcov-image</code>).</p> |
|
121 <p>The simplest way to run tests with JCov coverage report is to use the special target <code>jcov-test</code> instead of <code>test</code>, e.g. <code>make jcov-test TEST=jdk_lang</code>. This will make sure the JCov image is built, and that JCov reporting is enabled.</p> |
|
122 <p>The JCov report is stored in <code>build/$BUILD/test-results/jcov-output</code>.</p> |
|
123 <p>Please note that running with JCov reporting can be very memory intensive.</p> |
105 <h3 id="jtreg-keywords">JTReg keywords</h3> |
124 <h3 id="jtreg-keywords">JTReg keywords</h3> |
106 <h4 id="jobs">JOBS</h4> |
125 <h4 id="jobs-1">JOBS</h4> |
107 <p>The test concurrency (<code>-concurrency</code>).</p> |
126 <p>The test concurrency (<code>-concurrency</code>).</p> |
108 <p>Defaults to TEST_JOBS (if set by <code>--with-test-jobs=</code>), otherwise it defaults to JOBS, except for Hotspot, where the default is <em>number of CPU cores/2</em>, but never more than 12.</p> |
127 <p>Defaults to TEST_JOBS (if set by <code>--with-test-jobs=</code>), otherwise it defaults to JOBS, except for Hotspot, where the default is <em>number of CPU cores/2</em>, but never more than 12.</p> |
109 <h4 id="timeout">TIMEOUT</h4> |
128 <h4 id="timeout_factor-1">TIMEOUT_FACTOR</h4> |
110 <p>The timeout factor (<code>-timeoutFactor</code>).</p> |
129 <p>The timeout factor (<code>-timeoutFactor</code>).</p> |
111 <p>Defaults to 4.</p> |
130 <p>Defaults to 4.</p> |
112 <h4 id="test_mode">TEST_MODE</h4> |
131 <h4 id="test_mode">TEST_MODE</h4> |
113 <p>The test mode (<code>-agentvm</code>, <code>-samevm</code> or <code>-othervm</code>).</p> |
132 <p>The test mode (<code>-agentvm</code>, <code>-samevm</code> or <code>-othervm</code>).</p> |
114 <p>Defaults to <code>-agentvm</code>.</p> |
133 <p>Defaults to <code>-agentvm</code>.</p> |
123 <p>Defaults to <code>fail,error</code>.</p> |
142 <p>Defaults to <code>fail,error</code>.</p> |
124 <h4 id="max_mem">MAX_MEM</h4> |
143 <h4 id="max_mem">MAX_MEM</h4> |
125 <p>Limit memory consumption (<code>-Xmx</code> and <code>-vmoption:-Xmx</code>, or none).</p> |
144 <p>Limit memory consumption (<code>-Xmx</code> and <code>-vmoption:-Xmx</code>, or none).</p> |
126 <p>Limit memory consumption for JTReg test framework and VM under test. Set to 0 to disable the limits.</p> |
145 <p>Limit memory consumption for JTReg test framework and VM under test. Set to 0 to disable the limits.</p> |
127 <p>Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).</p> |
146 <p>Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).</p> |
|
147 <h4 id="keywords">KEYWORDS</h4> |
|
148 <p>JTReg kewords sent to JTReg using <code>-k</code>. Please be careful in making sure that spaces and special characters (like <code>!</code>) are properly quoted. To avoid some issues, the special value <code>%20</code> can be used instead of space.</p> |
|
149 <h4 id="extra_problem_lists">EXTRA_PROBLEM_LISTS</h4> |
|
150 <p>Use additional problem lists file or files, in addition to the default ProblemList.txt located at the JTReg test roots.</p> |
|
151 <p>If multiple file names are specified, they should be separated by space (or, to help avoid quoting issues, the special value <code>%20</code>).</p> |
|
152 <p>The file names should be either absolute, or relative to the JTReg test root of the tests to be run.</p> |
128 <h4 id="options">OPTIONS</h4> |
153 <h4 id="options">OPTIONS</h4> |
129 <p>Additional options to the JTReg test framework.</p> |
154 <p>Additional options to the JTReg test framework.</p> |
130 <p>Use <code>JTREG="OPTIONS=--help all"</code> to see all available JTReg options.</p> |
155 <p>Use <code>JTREG="OPTIONS=--help all"</code> to see all available JTReg options.</p> |
131 <h4 id="java_options">JAVA_OPTIONS</h4> |
156 <h4 id="java_options-1">JAVA_OPTIONS</h4> |
132 <p>Additional Java options to JTReg (<code>-javaoption</code>).</p> |
157 <p>Additional Java options to JTReg (<code>-javaoption</code>).</p> |
133 <h4 id="vm_options">VM_OPTIONS</h4> |
158 <h4 id="vm_options-1">VM_OPTIONS</h4> |
134 <p>Additional VM options to JTReg (<code>-vmoption</code>).</p> |
159 <p>Additional VM options to JTReg (<code>-vmoption</code>).</p> |
|
160 <h4 id="aot_modules-1">AOT_MODULES</h4> |
|
161 <p>Generate AOT modules before testing for the specified module, or set of modules. If multiple modules are specified, they should be separated by space (or, to help avoid quoting issues, the special value <code>%20</code>).</p> |
135 <h3 id="gtest-keywords">Gtest keywords</h3> |
162 <h3 id="gtest-keywords">Gtest keywords</h3> |
136 <h4 id="repeat">REPEAT</h4> |
163 <h4 id="repeat">REPEAT</h4> |
137 <p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p> |
164 <p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p> |
138 <p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p> |
165 <p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p> |
139 <h4 id="options-1">OPTIONS</h4> |
166 <h4 id="options-1">OPTIONS</h4> |
140 <p>Additional options to the Gtest test framework.</p> |
167 <p>Additional options to the Gtest test framework.</p> |
141 <p>Use <code>GTEST="OPTIONS=--help"</code> to see all available Gtest options.</p> |
168 <p>Use <code>GTEST="OPTIONS=--help"</code> to see all available Gtest options.</p> |
|
169 <h4 id="aot_modules-2">AOT_MODULES</h4> |
|
170 <p>Generate AOT modules before testing for the specified module, or set of modules. If multiple modules are specified, they should be separated by space (or, to help avoid quoting issues, the special value <code>%20</code>).</p> |
142 <h3 id="microbenchmark-keywords">Microbenchmark keywords</h3> |
171 <h3 id="microbenchmark-keywords">Microbenchmark keywords</h3> |
143 <h4 id="fork">FORK</h4> |
172 <h4 id="fork">FORK</h4> |
144 <p>Override the number of benchmark forks to spawn. Same as specifying <code>-f <num></code>.</p> |
173 <p>Override the number of benchmark forks to spawn. Same as specifying <code>-f <num></code>.</p> |
145 <h4 id="iter">ITER</h4> |
174 <h4 id="iter">ITER</h4> |
146 <p>Number of measurement iterations per fork. Same as specifying <code>-i <num></code>.</p> |
175 <p>Number of measurement iterations per fork. Same as specifying <code>-i <num></code>.</p> |
150 <p>Number of warmup iterations to run before the measurement phase in each fork. Same as specifying <code>-wi <num></code>.</p> |
179 <p>Number of warmup iterations to run before the measurement phase in each fork. Same as specifying <code>-wi <num></code>.</p> |
151 <h4 id="warmup_time">WARMUP_TIME</h4> |
180 <h4 id="warmup_time">WARMUP_TIME</h4> |
152 <p>Amount of time to spend in each warmup iteration. Same as specifying <code>-w <num></code>.</p> |
181 <p>Amount of time to spend in each warmup iteration. Same as specifying <code>-w <num></code>.</p> |
153 <h4 id="results_format">RESULTS_FORMAT</h4> |
182 <h4 id="results_format">RESULTS_FORMAT</h4> |
154 <p>Specify to have the test run save a log of the values. Accepts the same values as <code>-rff</code>, i.e., <code>text</code>, <code>csv</code>, <code>scsv</code>, <code>json</code>, or <code>latex</code>.</p> |
183 <p>Specify to have the test run save a log of the values. Accepts the same values as <code>-rff</code>, i.e., <code>text</code>, <code>csv</code>, <code>scsv</code>, <code>json</code>, or <code>latex</code>.</p> |
155 <h4 id="vm_options-1">VM_OPTIONS</h4> |
184 <h4 id="vm_options-2">VM_OPTIONS</h4> |
156 <p>Additional VM arguments to provide to forked off VMs. Same as <code>-jvmArgs <args></code></p> |
185 <p>Additional VM arguments to provide to forked off VMs. Same as <code>-jvmArgs <args></code></p> |
157 <h4 id="options-2">OPTIONS</h4> |
186 <h4 id="options-2">OPTIONS</h4> |
158 <p>Additional arguments to send to JMH.</p> |
187 <p>Additional arguments to send to JMH.</p> |
159 </body> |
188 </body> |
160 </html> |
189 </html> |