doc/testing.md
branchJEP-230-microbenchmarks-branch
changeset 56953 2ea07b19b446
parent 50885 7c728fa9d1af
equal deleted inserted replaced
56944:96ecae6d8049 56953:2ea07b19b446
    18     $ make run-test-jdk_lang JTREG="JOBS=8"
    18     $ make run-test-jdk_lang JTREG="JOBS=8"
    19     $ make run-test TEST=jdk_lang
    19     $ make run-test TEST=jdk_lang
    20     $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
    20     $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
    21     $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
    21     $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
    22     $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
    22     $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
       
    23     $ make run-test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
    23     $ make exploded-run-test TEST=tier2
    24     $ make exploded-run-test TEST=tier2
    24 
    25 
    25 ### Configuration
    26 ### Configuration
    26 
    27 
    27 To be able to run JTReg tests, `configure` needs to know where to find the
    28 To be able to run JTReg tests, `configure` needs to know where to find the
    28 JTReg test framework. If it is not picked up automatically by configure, use
    29 JTReg test framework. If it is not picked up automatically by configure, use
    29 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
    30 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
    30 Note that this option should point to the JTReg home, i.e. the top directory,
    31 Note that this option should point to the JTReg home, i.e. the top directory,
    31 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
    32 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
    32 environment variable to point to the JTReg home before running `configure`.)
    33 environment variable to point to the JTReg home before running `configure`.)
       
    34 
       
    35 To be able to run microbenchmarks, `configure` needs to know where to find
       
    36 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
       
    37 containing the core JMH and transitive dependencies. The recommended dependencies 
       
    38 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which 
       
    39 `--with-jmh=build/jmh/jars` should work.
    33 
    40 
    34 ## Test selection
    41 ## Test selection
    35 
    42 
    36 All functionality is available using the run-test make target. In this use
    43 All functionality is available using the run-test make target. In this use
    37 case, the test or tests to be executed is controlled using the `TEST` variable.
    44 case, the test or tests to be executed is controlled using the `TEST` variable.
    95 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
   102 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
    96 is defined by adding `/<variant>` to the test descriptor, e.g.
   103 is defined by adding `/<variant>` to the test descriptor, e.g.
    97 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
   104 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
    98 variant present (e.g. server, client). So if you only have the server JVM
   105 variant present (e.g. server, client). So if you only have the server JVM
    99 present, then `gtest:all` will be equivalent to `gtest:all/server`.
   106 present, then `gtest:all` will be equivalent to `gtest:all/server`.
       
   107 
       
   108 ### Microbenchmarks
       
   109 
       
   110 Which microbenchmarks to run is selected using a regular expression
       
   111 following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This
       
   112 delegates the test selection to JMH, meaning package name, class name and even
       
   113 benchmark method names can be used to select tests.
       
   114 
       
   115 Using special characters like `|` in the regular expression is possible, but
       
   116 needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
   100 
   117 
   101 ## Test results and summary
   118 ## Test results and summary
   102 
   119 
   103 At the end of the test run, a summary of all tests run will be presented. This
   120 At the end of the test run, a summary of all tests run will be presented. This
   104 will have a consistent look, regardless of what test suites were used. This is
   121 will have a consistent look, regardless of what test suites were used. This is
   228 #### OPTIONS
   245 #### OPTIONS
   229 Additional options to the Gtest test framework.
   246 Additional options to the Gtest test framework.
   230 
   247 
   231 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
   248 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
   232 
   249 
       
   250 ### Microbenchmark keywords
       
   251 
       
   252 #### FORK
       
   253 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
       
   254 
       
   255 #### ITER
       
   256 Number of measurement iterations per fork. Same as specifying `-i <num>`.
       
   257 
       
   258 #### TIME
       
   259 Amount of time to spend in each measurement iteration, in seconds. Same as
       
   260 specifying `-r <num>`
       
   261 
       
   262 #### WARMUP_ITER
       
   263 Number of warmup iterations to run before the measurement phase in each fork.
       
   264 Same as specifying `-wi <num>`.
       
   265 
       
   266 #### WARMUP_TIME
       
   267 Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
       
   268 
       
   269 #### RESULTS_FORMAT
       
   270 Specify to have the test run save a log of the values. Accepts the same values
       
   271 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
       
   272 
       
   273 #### VM_OPTIONS
       
   274 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
       
   275 
       
   276 #### OPTIONS
       
   277 Additional arguments to send to JMH.
       
   278 
   233 ---
   279 ---
   234 # Override some definitions in the global css file that are not optimal for
   280 # Override some definitions in the global css file that are not optimal for
   235 # this document.
   281 # this document.
   236 header-includes:
   282 header-includes:
   237  - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
   283  - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'