jdk-sandbox: comparison doc/testing.html

equal deleted inserted replaced

-:f62a71a762af
+:26e61f9ceec4
 <li><a href="#configuration">Configuration</a></li>
 </ul></li>
 <li><a href="#test-selection">Test selection</a><ul>
 <li><a href="#jtreg">JTReg</a></li>
 <li><a href="#gtest">Gtest</a></li>
+<li><a href="#microbenchmarks">Microbenchmarks</a></li>
 </ul></li>
 <li><a href="#test-results-and-summary">Test results and summary</a></li>
 <li><a href="#test-suite-control">Test suite control</a><ul>
 <li><a href="#jtreg-keywords">JTReg keywords</a></li>
 <li><a href="#gtest-keywords">Gtest keywords</a></li>
+<li><a href="#microbenchmark-keywords">Microbenchmark keywords</a></li>
 </ul></li>
 </ul>
 </nav>
 <h2 id="using-the-run-test-framework">Using the run-test framework</h2>
 <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
 $ make run-test-jdk_lang JTREG=&quot;JOBS=8&quot;
 $ make run-test TEST=jdk_lang
 $ make run-test-only TEST=&quot;gtest:LogTagSet gtest:LogTagSetDescriptions&quot; GTEST=&quot;REPEAT=-1&quot;
 $ make run-test TEST=&quot;hotspot:hotspot_gc&quot; JTREG=&quot;JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug&quot;
 $ make run-test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot;
+$ make run-test TEST=&quot;micro:java.lang.reflect&quot; MICRO=&quot;FORK=1;WARMUP_ITER=2&quot;
 $ make exploded-run-test TEST=tier2</code></pre>
 <h3 id="configuration">Configuration</h3>
 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
+<p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=&lt;path to JMH jars&gt;</code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
 <h2 id="test-selection">Test selection</h2>
 <p>All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.</p>
 <p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make run-test-tier1</code> is equivalent to <code>make run-test TEST=&quot;tier1&quot;</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>run-test TEST=&quot;x&quot;</code> solution needs to be used.</p>
 <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
 <h3 id="jtreg">JTReg</h3>
 <p>As long as the test groups or test paths can be uniquely resolved, you do not need to enter the <code>jtreg:</code> prefix. If this is not possible, or if you want to use a fully qualified test descriptor, add <code>jtreg:</code>, e.g. <code>jtreg:test/hotspot/jtreg/native_sanity</code>.</p>
 <h3 id="gtest">Gtest</h3>
 <p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p>
 <p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p>
 <p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/&lt;variant&gt;</code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p>
+<h3 id="microbenchmarks">Microbenchmarks</h3>
+<p>Which microbenchmarks to run is selected using a regular expression following the <code>micro:</code> test descriptor, e.g., <code>micro:java.lang.reflect</code>. This delegates the test selection to JMH, meaning package name, class name and even benchmark method names can be used to select tests.</p>
+<p>Using special characters like <code>|</code> in the regular expression is possible, but needs to be escaped multiple times: <code>micro:ArrayCopy\\\\\|reflect</code>.</p>
 <h2 id="test-results-and-summary">Test results and summary</h2>
 <p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p>
 <pre><code>==============================
 Test summary
 ==============================
 <p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p>
 <p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p>
 <h4 id="options-1">OPTIONS</h4>
 <p>Additional options to the Gtest test framework.</p>
 <p>Use <code>GTEST=&quot;OPTIONS=--help&quot;</code> to see all available Gtest options.</p>
+<h3 id="microbenchmark-keywords">Microbenchmark keywords</h3>
+<h4 id="fork">FORK</h4>
+<p>Override the number of benchmark forks to spawn. Same as specifying <code>-f &lt;num&gt;</code>.</p>
+<h4 id="iter">ITER</h4>
+<p>Number of measurement iterations per fork. Same as specifying <code>-i &lt;num&gt;</code>.</p>
+<h4 id="time">TIME</h4>
+<p>Amount of time to spend in each measurement iteration, in seconds. Same as specifying <code>-r &lt;num&gt;</code></p>
+<h4 id="warmup_iter">WARMUP_ITER</h4>
+<p>Number of warmup iterations to run before the measurement phase in each fork. Same as specifying <code>-wi &lt;num&gt;</code>.</p>
+<h4 id="warmup_time">WARMUP_TIME</h4>
+<p>Amount of time to spend in each warmup iteration. Same as specifying <code>-w &lt;num&gt;</code>.</p>
+<h4 id="results_format">RESULTS_FORMAT</h4>
+<p>Specify to have the test run save a log of the values. Accepts the same values as <code>-rff</code>, i.e., <code>text</code>, <code>csv</code>, <code>scsv</code>, <code>json</code>, or <code>latex</code>.</p>
+<h4 id="vm_options-1">VM_OPTIONS</h4>
+<p>Additional VM arguments to provide to forked off VMs. Same as <code>-jvmArgs &lt;args&gt;</code></p>
+<h4 id="options-2">OPTIONS</h4>
+<p>Additional arguments to send to JMH.</p>
 </body>
 </html>

branch	JEP-230-microbenchmarks-branch
changeset 56980	26e61f9ceec4
parent 51644	0df09dadd445