--- a/doc/testing.html Wed Oct 31 08:09:45 2018 +0100
+++ b/doc/testing.html Wed Oct 31 09:30:24 2018 +0100
@@ -18,12 +18,13 @@
</header>
<nav id="TOC">
<ul>
-<li><a href="#using-the-run-test-framework">Using the run-test framework</a><ul>
+<li><a href="#using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</a><ul>
<li><a href="#configuration">Configuration</a></li>
</ul></li>
<li><a href="#test-selection">Test selection</a><ul>
<li><a href="#jtreg">JTReg</a></li>
<li><a href="#gtest">Gtest</a></li>
+<li><a href="#special-tests">Special tests</a></li>
</ul></li>
<li><a href="#test-results-and-summary">Test results and summary</a></li>
<li><a href="#test-suite-control">Test suite control</a><ul>
@@ -32,22 +33,23 @@
</ul></li>
</ul>
</nav>
-<h2 id="using-the-run-test-framework">Using the run-test framework</h2>
+<h2 id="using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</h2>
<p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
-<p>The main target "run-test" uses the jdk-image as the tested product. There is also an alternate target "exploded-run-test" that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
+<p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
+<p>Previously, <code>make test</code> was used invoke an old system for running test, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases. The old system can still be accessed for some time using <code>cd test && make</code>.</p>
<p>Some example command-lines:</p>
-<pre><code>$ make run-test-tier1
-$ make run-test-jdk_lang JTREG="JOBS=8"
-$ make run-test TEST=jdk_lang
-$ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
-$ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
-$ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
-$ make exploded-run-test TEST=tier2</code></pre>
+<pre><code>$ make test-tier1
+$ make test-jdk_lang JTREG="JOBS=8"
+$ make test TEST=jdk_lang
+$ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
+$ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
+$ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
+$ make exploded-test TEST=tier2</code></pre>
<h3 id="configuration">Configuration</h3>
<p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
<h2 id="test-selection">Test selection</h2>
-<p>All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.</p>
-<p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make run-test-tier1</code> is equivalent to <code>make run-test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>run-test TEST="x"</code> solution needs to be used.</p>
+<p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
+<p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST="x"</code> solution needs to be used.</p>
<p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
<h3 id="jtreg">JTReg</h3>
<p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
@@ -59,6 +61,14 @@
<p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p>
<p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p>
<p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/<variant></code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p>
+<h3 id="special-tests">Special tests</h3>
+<p>A handful of odd tests that are not covered by any other testing framework are accessible using the <code>special:</code> test descriptor. Currently, this includes <code>hotspot-internal</code>, <code>failure-handler</code> and <code>make</code>.</p>
+<ul>
+<li><p>Hotspot legacy internal testing (run using <code>-XX:+ExecuteInternalVMTests</code>) is run using <code>special:hotspot-internal</code> or just <code>hotspot-internal</code> as test descriptor, and will only work on a debug JVM.</p></li>
+<li><p>Failure handler testing is run using <code>special:failure-handler</code> or just <code>failure-handler</code> as test descriptor.</p></li>
+<li><p>Tests for the build system, including both makefiles and related functionality, is run using <code>special:make</code> or just <code>make</code> as test descriptor. This is equivalent to <code>special:make:all</code>.</p>
+<p>A specific make test can be run by supplying it as argument, e.g. <code>special:make:idea</code>. As a special syntax, this can also be expressed as <code>make-idea</code>, which allows for command lines as <code>make test-make-idea</code>.</p></li>
+</ul>
<h2 id="test-results-and-summary">Test results and summary</h2>
<p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p>
<pre><code>==============================
@@ -72,7 +82,7 @@
TEST FAILURE</code></pre>
<p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>>> ... <<</code> marker for easy identification.</p>
<p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p>
-<p>In case of test failures, <code>make run-test</code> will exit with a non-zero exit value.</p>
+<p>In case of test failures, <code>make test</code> will exit with a non-zero exit value.</p>
<p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p>
<p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p>
<h2 id="test-suite-control">Test suite control</h2>
--- a/doc/testing.md Wed Oct 31 08:09:45 2018 +0100
+++ b/doc/testing.md Wed Oct 31 09:30:24 2018 +0100
@@ -1,26 +1,32 @@
% Testing the JDK
-## Using the run-test framework
+## Using "make test" (the run-test framework)
This new way of running tests is developer-centric. It assumes that you have
built a JDK locally and want to test it. Running common test targets is simple,
and more complex ad-hoc combination of tests is possible. The user interface is
forgiving, and clearly report errors it cannot resolve.
-The main target "run-test" uses the jdk-image as the tested product. There is
-also an alternate target "exploded-run-test" that uses the exploded image
+The main target `test` uses the jdk-image as the tested product. There is
+also an alternate target `exploded-test` that uses the exploded image
instead. Not all tests will run successfully on the exploded image, but using
this target can greatly improve rebuild times for certain workflows.
+Previously, `make test` was used invoke an old system for running test, and
+`make run-test` was used for the new test framework. For backward compatibility
+with scripts and muscle memory, `run-test` (and variants like
+`exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
+can still be accessed for some time using `cd test && make`.
+
Some example command-lines:
- $ make run-test-tier1
- $ make run-test-jdk_lang JTREG="JOBS=8"
- $ make run-test TEST=jdk_lang
- $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
- $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
- $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
- $ make exploded-run-test TEST=tier2
+ $ make test-tier1
+ $ make test-jdk_lang JTREG="JOBS=8"
+ $ make test TEST=jdk_lang
+ $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
+ $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
+ $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
+ $ make exploded-test TEST=tier2
### Configuration
@@ -33,16 +39,16 @@
## Test selection
-All functionality is available using the run-test make target. In this use
-case, the test or tests to be executed is controlled using the `TEST` variable.
-To speed up subsequent test runs with no source code changes, run-test-only can
-be used instead, which do not depend on the source and test image build.
+All functionality is available using the `test` make target. In this use case,
+the test or tests to be executed is controlled using the `TEST` variable. To
+speed up subsequent test runs with no source code changes, `test-only` can be
+used instead, which do not depend on the source and test image build.
For some common top-level tests, direct make targets have been generated. This
includes all JTReg test groups, the hotspot gtest, and custom tests (if
-present). This means that `make run-test-tier1` is equivalent to `make run-test
+present). This means that `make test-tier1` is equivalent to `make test
TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
-test runs, the `run-test TEST="x"` solution needs to be used.
+test runs, the `test TEST="x"` solution needs to be used.
The test specifications given in `TEST` is parsed into fully qualified test
descriptors, which clearly and unambigously show which tests will be run. As an
@@ -98,6 +104,27 @@
variant present (e.g. server, client). So if you only have the server JVM
present, then `gtest:all` will be equivalent to `gtest:all/server`.
+### Special tests
+
+A handful of odd tests that are not covered by any other testing framework are
+accessible using the `special:` test descriptor. Currently, this includes
+`hotspot-internal`, `failure-handler` and `make`.
+
+ * Hotspot legacy internal testing (run using `-XX:+ExecuteInternalVMTests`)
+ is run using `special:hotspot-internal` or just `hotspot-internal` as test
+ descriptor, and will only work on a debug JVM.
+
+ * Failure handler testing is run using `special:failure-handler` or just
+ `failure-handler` as test descriptor.
+
+ * Tests for the build system, including both makefiles and related
+ functionality, is run using `special:make` or just `make` as test
+ descriptor. This is equivalent to `special:make:all`.
+
+ A specific make test can be run by supplying it as argument, e.g.
+ `special:make:idea`. As a special syntax, this can also be expressed as
+ `make-idea`, which allows for command lines as `make test-make-idea`.
+
## Test results and summary
At the end of the test run, a summary of all tests run will be presented. This
@@ -123,7 +150,7 @@
classified as failed by the framework. This might indicate test framework
error, timeout or other problems.
-In case of test failures, `make run-test` will exit with a non-zero exit value.
+In case of test failures, `make test` will exit with a non-zero exit value.
All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
where TEST_ID is a path-safe conversion from the fully qualified test
--- a/make/Help.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/make/Help.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2012, 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2012, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -55,10 +55,10 @@
$(info $(_) make install # Install the generated images locally)
$(info $(_) make reconfigure # Rerun configure with the same arguments as last time)
$(info $(_) make help # Give some help on using make)
- $(info $(_) make test # Run tests, default is all tests (see TEST below))
- $(info $(_) make run-test-<test> # Run test, e.g. run-test-tier1)
- $(info $(_) make run-test TEST=<t> # Run test(s) given by TEST specification)
- $(info $(_) make exploded-run-test TEST=<t> # Run test(s) on the exploded image instead of)
+ $(info $(_) make check # Run basic testing (currently tier1))
+ $(info $(_) make test-<test> # Run test, e.g. test-tier1)
+ $(info $(_) make test TEST=<t> # Run test(s) given by TEST specification)
+ $(info $(_) make exploded-test TEST=<t> # Run test(s) on the exploded image instead of)
$(info $(_) # the full jdk image)
$(info )
$(info Targets for cleaning)
@@ -99,10 +99,12 @@
$(info $(_) TEST_JOBS=<n> # Run <n> parallel test jobs)
$(info $(_) CONF_CHECK=<method> # What to do if spec file is out of date)
$(info $(_) # method is 'auto', 'ignore' or 'fail' (default))
- $(info $(_) make test TEST=<test> # Only run the given test or tests, e.g.)
- $(info $(_) # make test TEST="jdk_lang jdk_net")
- $(info $(_) JTREG="OPT1=x;OPT2=y" # Control the JTREG test harness for run-test)
- $(info $(_) GTEST="OPT1=x;OPT2=y" # Control the GTEST test harness for run-test)
+ $(info $(_) TEST="test1 ..." # Use the given test descriptor(s) for testing, e.g.)
+ $(info $(_) # make test TEST="jdk_lang gtest:all")
+ $(info $(_) JTREG="OPT1=x;OPT2=y" # Control the JTREG test harness)
+ $(info $(_) GTEST="OPT1=x;OPT2=y" # Control the GTEST test harness)
+ $(info $(_) TEST_OPTS="OPT1=x;..." # Generic control of all test harnesses)
+ $(info $(_) TEST_VM_OPTS="ARG ..." # Same as setting TEST_OPTS to VM_OPTIONS="ARG ...")
$(info )
$(if $(all_confs), $(info Available configurations in $(build_dir):) $(foreach var,$(all_confs),$(info * $(var))),\
$(info No configurations were found in $(build_dir).) $(info Run 'bash configure' to create a configuration.))
--- a/make/Main.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/make/Main.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -473,10 +473,10 @@
#
define DeclareRunTestRecipe
- run-test-$1:
+ test-$1:
+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$1")
- exploded-run-test-$1:
+ exploded-test-$1:
+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
TEST="$1" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
@@ -484,8 +484,8 @@
# ALL_NAMED_TESTS is defined in FindTests.gmk
$(foreach t, $(ALL_NAMED_TESTS), $(eval $(call DeclareRunTestRecipe,$t)))
-ALL_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS))
-ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS))
+ALL_TEST_TARGETS := $(addprefix test-, $(ALL_NAMED_TESTS))
+ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-test-, $(ALL_NAMED_TESTS))
ALL_TARGETS += $(ALL_TEST_TARGETS) $(ALL_EXPLODED_TEST_TARGETS)
@@ -520,13 +520,6 @@
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f JtregGraalUnit.gmk \
test-image-hotspot-jtreg-graal)
-run-test:
- +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)")
-
-exploded-run-test:
- +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
- TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
-
ifeq ($(BUILD_GTEST), true)
test-image-hotspot-gtest:
+($(CD) $(TOPDIR)/make/hotspot/test && $(MAKE) $(MAKE_ARGS) -f GtestImage.gmk)
@@ -541,11 +534,6 @@
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
-f BuildFailureHandler.gmk build)
- # Runs the tests for the failure handler jtreg extension
- test-failure-handler:
- +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
- -f BuildFailureHandler.gmk test)
-
# Copies the failure handler jtreg extension into the test image
test-image-failure-handler:
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
@@ -556,40 +544,19 @@
test-image-hotspot-jtreg-native build-test-jdk-jtreg-native \
test-image-jdk-jtreg-native build-test-lib build-test-failure-handler \
test-failure-handler test-image-failure-handler test-image-hotspot-gtest \
- test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal \
- run-test exploded-run-test
+ test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal
################################################################################
# Run tests
-# Run tests specified by $(TEST), or the default test set.
test:
- $(call RunTests, $(TEST), $(JDK_IMAGE_DIR))
-
-test-hotspot-jtreg:
- $(call RunTests, "hotspot_all", $(JDK_IMAGE_DIR))
-
-test-hotspot-jtreg-native:
- $(call RunTests, "hotspot_native_sanity", $(JDK_IMAGE_DIR))
-
-test-hotspot-internal:
- $(call RunTests, "hotspot_internal", $(JDK_OUTPUTDIR))
+ +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)")
-test-hotspot-gtest:
- $(call RunTests, "hotspot_gtest", $(JDK_OUTPUTDIR))
-
-test-jdk-jtreg-native:
- $(call RunTests, "jdk_native_sanity", $(JDK_IMAGE_DIR))
+exploded-test:
+ +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
+ TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
-test-make:
- ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk $(TEST_TARGET))
-
-test-compile-commands:
- ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk test-compile-commands)
-
-ALL_TARGETS += test test-hotspot-jtreg test-hotspot-jtreg-native \
- test-hotspot-internal test-hotspot-gtest test-jdk-jtreg-native test-make \
- test-compile-commands
+ALL_TARGETS += test exploded-test
################################################################################
# Bundles
@@ -874,14 +841,18 @@
docs-zip: docs-jdk
+ # Tests
test: jdk-image test-image
- run-test: jdk-image test-image
- exploded-run-test: exploded-image test-image
+ exploded-test: exploded-image test-image
+
+ test-make: clean-test-make compile-commands
+
+ test-make-compile-commands: compile-commands
# Declare dependency for all generated test targets
- $(foreach t, $(ALL_TEST_TARGETS), $(eval $t: jdk-image test-image))
- $(foreach t, $(ALL_EXPLODED_TEST_TARGETS), $(eval $t: exploded-image test-image))
+ $(foreach t, $(filter-out test-make%, $(ALL_TEST_TARGETS)), $(eval $t: jdk-image test-image))
+ $(foreach t, $(filter-out exploded-test-make%, $(ALL_EXPLODED_TEST_TARGETS)), $(eval $t: exploded-image test-image))
create-buildjdk-copy: jdk.jlink-java java.base-gendata \
$(addsuffix -java, $(INTERIM_IMAGE_MODULES))
@@ -890,16 +861,10 @@
interim-image: $(INTERIM_JMOD_TARGETS)
- test-make: clean-test-make
-
- test-compile-commands: compile-commands
-
build-test-lib: exploded-image-optimize
build-test-failure-handler: interim-langtools
- test-failure-handler: build-test-failure-handler
-
test-image-failure-handler: build-test-failure-handler
build-test-hotspot-jtreg-native: buildtools-jdk \
@@ -917,12 +882,6 @@
test-image-hotspot-gtest: hotspot
- test-hotspot-internal: exploded-image
-
- test-hotspot-jtreg: jdk-image test-image
-
- test-hotspot-gtest: exploded-image test-image-hotspot-gtest
-
install: product-images
product-bundles: product-images
@@ -1091,6 +1050,30 @@
ALL_TARGETS += default jdk images docs bundles all
+# Aliases used for running tests.
+
+# Let "run-test" be an alias for "test"
+$(foreach t, $(ALL_NAMED_TESTS), $(eval run-test-$t: test-$t))
+$(foreach t, $(ALL_NAMED_TESTS), $(eval exploded-run-test-$t: exploded-test-$t))
+RUN_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS)) \
+ $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS))
+
+run-test: test
+exploded-run-test: exploded-test
+
+# "make check" is a common idiom for running basic testing
+check: test-tier1
+
+# Keep some old names as aliases
+test-hotspot-jtreg: test-hotspot_all
+test-hotspot-jtreg-native: test-hotspot_native_sanity
+test-hotspot-gtest: exploded-test-gtest
+test-jdk-jtreg-native: test-jdk_native_sanity
+
+ALL_TARGETS += $(RUN_TEST_TARGETS) run-test exploded-run-test check \
+ test-hotspot-jtreg test-hotspot-jtreg-native test-hotspot-gtest \
+ test-jdk-jtreg-native
+
################################################################################
################################################################################
#
--- a/make/MainSupport.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/make/MainSupport.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2011, 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -30,17 +30,6 @@
ifndef _MAINSUPPORT_GMK
_MAINSUPPORT_GMK := 1
-# Run the tests specified by $1, with PRODUCT_HOME specified by $2
-# JT_JAVA is picked up by the jtreg launcher and used to run Jtreg itself.
-define RunTests
- ($(CD) $(TOPDIR)/test && $(MAKE) $(MAKE_ARGS) -j1 -k MAKEFLAGS= \
- JT_HOME=$(JT_HOME) PRODUCT_HOME=$(strip $2) \
- TEST_IMAGE_DIR=$(TEST_IMAGE_DIR) \
- ALT_OUTPUTDIR=$(OUTPUTDIR) TEST_JOBS=$(TEST_JOBS) \
- JT_JAVA=$(BOOT_JDK) JIB_JAR=$(JIB_JAR) \
- JOBS=$(JOBS) $1) || true
-endef
-
define CleanDocs
@$(PRINTF) "Cleaning docs ..."
@$(PRINTF) "\n" $(LOG_DEBUG)
--- a/make/RunTests.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/make/RunTests.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2016, 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2016, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -300,16 +300,32 @@
# Helper function to determine if a test specification is a special test
#
-# It is a special test if it is "special:" followed by a test name.
+# It is a special test if it is "special:" followed by a test name,
+# if it is "make:" or "make-" followed by a make test, or any of the special test names
+# as a single word.
define ParseSpecialTestSelection
$(if $(filter special:%, $1), \
$1 \
+ ) \
+ $(if $(filter make%, $1), \
+ $(if $(filter make:%, $1), \
+ special:$(strip $1) \
+ ) \
+ $(if $(filter make-%, $1), \
+ special:$(patsubst make-%,make:%, $1) \
+ ) \
+ $(if $(filter make, $1), \
+ special:make:all \
+ )
+ ) \
+ $(if $(filter hotspot-internal failure-handler, $1), \
+ special:$(strip $1) \
)
endef
ifeq ($(TEST), )
$(info No test selection given in TEST!)
- $(info Please use e.g. 'run-test TEST=tier1' or 'run-test-tier1')
+ $(info Please use e.g. 'make test TEST=tier1' or 'make test-tier1')
$(info See doc/testing.[md|html] for help)
$(error Cannot continue)
endif
@@ -663,9 +679,13 @@
$$(JDK_IMAGE_DIR)/bin/java -XX:+ExecuteInternalVMTests \
-XX:+ShowMessageBoxOnError -version
else ifeq ($$($1_TEST_NAME), failure-handler)
- $1_TEST_COMMAND_LINE := \
- ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \
- BuildFailureHandler.gmk test)
+ ifeq ($(BUILD_FAILURE_HANDLER), true)
+ $1_TEST_COMMAND_LINE := \
+ ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \
+ BuildFailureHandler.gmk test)
+ else
+ $$(error Cannot test failure handler if it is not built)
+ endif
else ifeq ($$($1_TEST_NAME), make)
$1_TEST_COMMAND_LINE := \
($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f \
--- a/make/common/FindTests.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/make/common/FindTests.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2017, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -72,6 +72,15 @@
# Add Gtest
ALL_NAMED_TESTS += gtest
+# Find make test targets
+MAKE_TEST_TARGETS := $(shell $(MAKE) -s --no-print-directory $(MAKE_ARGS) \
+ SPEC=$(SPEC) -f $(TOPDIR)/test/make/TestMake.gmk print-targets)
+
+ALL_NAMED_TESTS += $(addprefix make-, $(MAKE_TEST_TARGETS))
+
+# Add special tests
+ALL_NAMED_TESTS += hotspot-internal failure-handler make
+
################################################################################
endif # _FIND_TESTS_GMK
--- a/make/conf/jib-profiles.js Wed Oct 31 08:09:45 2018 +0100
+++ b/make/conf/jib-profiles.js Wed Oct 31 09:30:24 2018 +0100
@@ -759,7 +759,7 @@
testedProfile + ".test"
],
src: "src.conf",
- make_args: [ "run-test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ],
+ make_args: [ "test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ],
environment: {
"BOOT_JDK": common.boot_jdk_home,
"JDK_IMAGE_DIR": input.get(testedProfile + ".jdk", "home_path"),
--- a/test/Makefile Wed Oct 31 08:09:45 2018 +0100
+++ b/test/Makefile Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2010, 2015, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2010, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -27,11 +27,16 @@
# Makefile to run tests from multiple sibling directories
#
+$(info WARNING: This way of running tests ("cd test && make") is deprecated)
+$(info Please use "make test TEST=..." instead. See doc/testing.md for details)
+
# Macro to run a test target in a subdir
define SUBDIR_TEST # subdirectory target
if [ -d $1 ] ; then \
if [ -r $1/Makefile ] ; then \
$(MAKE) --no-print-directory -k -C $1 $2 ; \
+ echo 'WARNING: This way of running tests ("cd test && make") is deprecated' ; \
+ echo 'Please use "make test TEST=..." instead. See doc/testing.md for details' ; \
else \
echo "ERROR: File does not exist: $1/Makefile"; \
exit 1; \
--- a/test/make/TestMake.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/test/make/TestMake.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2014, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -36,13 +36,17 @@
copy-files:
+$(MAKE) -f TestCopyFiles.gmk $(TEST_SUBTARGET)
-test-idea:
+idea:
+$(MAKE) -f TestIdea.gmk $(TEST_SUBTARGET)
-test-compile-commands:
+compile-commands:
+$(MAKE) -f TestCompileCommands.gmk $(TEST_SUBTARGET)
+TARGETS += make-base java-compilation copy-files idea compile-commands
-all: make-base java-compilation copy-files test-idea
+all: $(TARGETS)
-.PHONY: default all make-base java-compilation copy-files test-idea test-compile-commands
+print-targets:
+ $(ECHO) "$(TARGETS)"
+
+.PHONY: default all $(TARGETS)
--- a/test/make/TestMakeBase.gmk Wed Oct 31 08:09:45 2018 +0100
+++ b/test/make/TestMakeBase.gmk Wed Oct 31 09:30:24 2018 +0100
@@ -344,7 +344,7 @@
KWBASE := APA=banan;GURKA=tomat;COUNT=1%202%203%204%205;SUM=1+2+3+4+5;MANY_WORDS=I have the best words.
$(eval $(call ParseKeywordVariable, KWBASE, \
- KEYWORDS := APA GURKA SUM, \
+ SINGLE_KEYWORDS := APA GURKA SUM, \
STRING_KEYWORDS := COUNT MANY_WORDS, \
))
@@ -377,7 +377,7 @@
KWBASE_WEIRD := ;;APA=banan;;;GURKA=apelsin;APA=skansen;;
$(eval $(call ParseKeywordVariable, KWBASE_WEIRD, \
- KEYWORDS := APA GURKA SUM, \
+ SINGLE_KEYWORDS := APA GURKA SUM, \
STRING_KEYWORDS := COUNT, \
))