8213698: Improve devkit creation and add support for linux/ppc64/ppc64le/s390x
Reviewed-by: erikj, ihse
--- a/doc/building.html Thu Nov 22 15:38:20 2018 +0100
+++ b/doc/building.html Thu Nov 22 17:22:40 2018 +0100
@@ -69,6 +69,7 @@
</ul></li>
<li><a href="#running-tests">Running Tests</a></li>
<li><a href="#cross-compiling">Cross-compiling</a><ul>
+<li><a href="#cross-compiling-the-easy-way-with-openjdk-devkits">Cross compiling the easy way with OpenJDK devkits</a></li>
<li><a href="#boot-jdk-and-build-jdk">Boot JDK and Build JDK</a></li>
<li><a href="#specifying-the-target-platform">Specifying the Target Platform</a></li>
<li><a href="#toolchain-considerations">Toolchain Considerations</a></li>
@@ -570,6 +571,47 @@
<p>This requires a more complex setup and build procedure. This section assumes you are familiar with cross-compiling in general, and will only deal with the particularities of cross-compiling the JDK. If you are new to cross-compiling, please see the <a href="https://en.wikipedia.org/wiki/Cross_compiler#External_links">external links at Wikipedia</a> for a good start on reading materials.</p>
<p>Cross-compiling the JDK requires you to be able to build both for the build platform and for the target platform. The reason for the former is that we need to build and execute tools during the build process, both native tools and Java tools.</p>
<p>If all you want to do is to compile a 32-bit version, for the same OS, on a 64-bit machine, consider using <code>--with-target-bits=32</code> instead of doing a full-blown cross-compilation. (While this surely is possible, it's a lot more work and will take much longer to build.)</p>
+<h3 id="cross-compiling-the-easy-way-with-openjdk-devkits">Cross compiling the easy way with OpenJDK devkits</h3>
+<p>The OpenJDK build system provides out-of-the box support for creating and using so called devkits. A <code>devkit</code> is basically a collection of a cross-compiling toolchain and a sysroot environment which can easily be used together with the <code>--with-devkit</code> configure option to cross compile the OpenJDK. On Linux/x86_64, the following command:</p>
+<pre><code>bash configure --with-devkit=<devkit-path> --openjdk-target=ppc64-linux-gnu && make</code></pre>
+<p>will configure and build OpenJDK for Linux/ppc64 assuming that <code><devkit-path></code> points to a Linux/x86_64 to Linux/ppc64 devkit.</p>
+<p>Devkits can be created from the <code>make/devkit</code> directory by executing:</p>
+<pre><code>make [ TARGETS="<TARGET_TRIPLET>+" ] [ BASE_OS=<OS> ] [ BASE_OS_VERSION=<VER> ]</code></pre>
+<p>where <code>TARGETS</code> contains one or more <code>TARGET_TRIPLET</code>s of the form described in <a href="https://sourceware.org/autobook/autobook/autobook_17.html">section 3.4 of the GNU Autobook</a>. If no targets are given, a native toolchain for the current platform will be created. Currently, at least the following targets are known to work:</p>
+<table>
+<thead>
+<tr class="header">
+<th style="text-align: left;">Supported devkit targets</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td style="text-align: left;">x86_64-linux-gnu</td>
+</tr>
+<tr class="even">
+<td style="text-align: left;">aarch64-linux-gnu</td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;">arm-linux-gnueabihf</td>
+</tr>
+<tr class="even">
+<td style="text-align: left;">ppc64-linux-gnu</td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;">ppc64le-linux-gnu</td>
+</tr>
+<tr class="even">
+<td style="text-align: left;">s390x-linux-gnu</td>
+</tr>
+</tbody>
+</table>
+<p><code>BASE_OS</code> must be one of "OEL6" for Oracle Enterprise Linux 6 or "Fedora" (if not specified "OEL6" will be the default). If the base OS is "Fedora" the corresponding Fedora release can be specified with the help of the <code>BASE_OS_VERSION</code> option (with "27" as default version). If the build is successful, the new devkits can be found in the <code>build/devkit/result</code> subdirectory:</p>
+<pre><code>cd make/devkit
+make TARGETS="ppc64le-linux-gnu aarch64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=21
+ls -1 ../../build/devkit/result/
+x86_64-linux-gnu-to-aarch64-linux-gnu
+x86_64-linux-gnu-to-ppc64le-linux-gnu</code></pre>
+<p>Notice that devkits are not only useful for targeting different build platforms. Because they contain the full build dependencies for a system (i.e. compiler and root file system), they can easily be used to build well-known, reliable and reproducible build environments. You can for example create and use a devkit with GCC 7.3 and a Fedora 12 sysroot environment (with glibc 2.11) on Ubuntu 14.04 (which doesn't have GCC 7.3 by default) to produce OpenJDK binaries which will run on all Linux systems with runtime libraries newer than the ones from Fedora 12 (e.g. Ubuntu 16.04, SLES 11 or RHEL 6).</p>
<h3 id="boot-jdk-and-build-jdk">Boot JDK and Build JDK</h3>
<p>When cross-compiling, make sure you use a boot JDK that runs on the <em>build</em> system, and not on the <em>target</em> system.</p>
<p>To be able to build, we need a "Build JDK", which is a JDK built from the current sources (that is, the same as the end result of the entire build process), but able to run on the <em>build</em> system, and not the <em>target</em> system. (In contrast, the Boot JDK should be from an older release, e.g. JDK 8 when building JDK 9.)</p>
@@ -662,8 +704,8 @@
<th style="text-align: left;">Target</th>
<th style="text-align: left;"><code>CC</code></th>
<th style="text-align: left;"><code>CXX</code></th>
-<th><code>--arch=...</code></th>
-<th><code>--openjdk-target=...</code></th>
+<th style="text-align: left;"><code>--arch=...</code></th>
+<th style="text-align: left;"><code>--openjdk-target=...</code></th>
</tr>
</thead>
<tbody>
@@ -671,36 +713,36 @@
<td style="text-align: left;">x86</td>
<td style="text-align: left;">default</td>
<td style="text-align: left;">default</td>
-<td>i386</td>
-<td>i386-linux-gnu</td>
+<td style="text-align: left;">i386</td>
+<td style="text-align: left;">i386-linux-gnu</td>
</tr>
<tr class="even">
<td style="text-align: left;">armhf</td>
<td style="text-align: left;">gcc-arm-linux-gnueabihf</td>
<td style="text-align: left;">g++-arm-linux-gnueabihf</td>
-<td>armhf</td>
-<td>arm-linux-gnueabihf</td>
+<td style="text-align: left;">armhf</td>
+<td style="text-align: left;">arm-linux-gnueabihf</td>
</tr>
<tr class="odd">
<td style="text-align: left;">aarch64</td>
<td style="text-align: left;">gcc-aarch64-linux-gnu</td>
<td style="text-align: left;">g++-aarch64-linux-gnu</td>
-<td>arm64</td>
-<td>aarch64-linux-gnu</td>
+<td style="text-align: left;">arm64</td>
+<td style="text-align: left;">aarch64-linux-gnu</td>
</tr>
<tr class="even">
<td style="text-align: left;">ppc64el</td>
<td style="text-align: left;">gcc-powerpc64le-linux-gnu</td>
<td style="text-align: left;">g++-powerpc64le-linux-gnu</td>
-<td>ppc64el</td>
-<td>powerpc64le-linux-gnu</td>
+<td style="text-align: left;">ppc64el</td>
+<td style="text-align: left;">powerpc64le-linux-gnu</td>
</tr>
<tr class="odd">
<td style="text-align: left;">s390x</td>
<td style="text-align: left;">gcc-s390x-linux-gnu</td>
<td style="text-align: left;">g++-s390x-linux-gnu</td>
-<td>s390x</td>
-<td>s390x-linux-gnu</td>
+<td style="text-align: left;">s390x</td>
+<td style="text-align: left;">s390x-linux-gnu</td>
</tr>
</tbody>
</table>
@@ -813,7 +855,12 @@
spawn failed</code></pre>
<p>This can be a sign of a Cygwin problem. See the information about solving problems in the <a href="#cygwin">Cygwin</a> section. Rebooting the computer might help temporarily.</p>
<h3 id="getting-help">Getting Help</h3>
-<p>If none of the suggestions in this document helps you, or if you find what you believe is a bug in the build system, please contact the Build Group by sending a mail to <a href="mailto:build-dev@openjdk.java.net">build-dev@openjdk.java.net</a>. Please include the relevant parts of the configure and/or build log.</p>
+<p>If none of the suggestions in this document helps you, or if you find what you believe is a bug in the build system, please contact the Build Group by sending a mail to <script type="text/javascript">
+<!--
+h='openjdk.java.net';a='@';n='build-dev';e=n+a+h;
+document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'" clas'+'s="em' + 'ail">'+e+'<\/'+'a'+'>');
+// -->
+</script><noscript>build-dev at openjdk dot java dot net</noscript>. Please include the relevant parts of the configure and/or build log.</p>
<p>If you need general help or advice about developing for the JDK, you can also contact the Adoption Group. See the section on <a href="#contributing-to-openjdk">Contributing to OpenJDK</a> for more information.</p>
<h2 id="hints-and-suggestions-for-advanced-users">Hints and Suggestions for Advanced Users</h2>
<h3 id="setting-up-a-repository-for-pushing-changes-defpath">Setting Up a Repository for Pushing Changes (defpath)</h3>
--- a/doc/building.md Thu Nov 22 15:38:20 2018 +0100
+++ b/doc/building.md Thu Nov 22 17:22:40 2018 +0100
@@ -884,6 +884,64 @@
full-blown cross-compilation. (While this surely is possible, it's a lot more
work and will take much longer to build.)
+### Cross compiling the easy way with OpenJDK devkits
+
+The OpenJDK build system provides out-of-the box support for creating and using
+so called devkits. A `devkit` is basically a collection of a cross-compiling
+toolchain and a sysroot environment which can easily be used together with the
+`--with-devkit` configure option to cross compile the OpenJDK. On Linux/x86_64,
+the following command:
+```
+bash configure --with-devkit=<devkit-path> --openjdk-target=ppc64-linux-gnu && make
+```
+
+will configure and build OpenJDK for Linux/ppc64 assuming that `<devkit-path>`
+points to a Linux/x86_64 to Linux/ppc64 devkit.
+
+Devkits can be created from the `make/devkit` directory by executing:
+```
+make [ TARGETS="<TARGET_TRIPLET>+" ] [ BASE_OS=<OS> ] [ BASE_OS_VERSION=<VER> ]
+```
+
+where `TARGETS` contains one or more `TARGET_TRIPLET`s of the form
+described in [section 3.4 of the GNU Autobook](
+https://sourceware.org/autobook/autobook/autobook_17.html). If no
+targets are given, a native toolchain for the current platform will be
+created. Currently, at least the following targets are known to work:
+
+ Supported devkit targets
+ ------------------------
+ x86_64-linux-gnu
+ aarch64-linux-gnu
+ arm-linux-gnueabihf
+ ppc64-linux-gnu
+ ppc64le-linux-gnu
+ s390x-linux-gnu
+
+`BASE_OS` must be one of "OEL6" for Oracle Enterprise Linux 6 or
+"Fedora" (if not specified "OEL6" will be the default). If the base OS
+is "Fedora" the corresponding Fedora release can be specified with the
+help of the `BASE_OS_VERSION` option (with "27" as default version).
+If the build is successful, the new devkits can be found in the
+`build/devkit/result` subdirectory:
+```
+cd make/devkit
+make TARGETS="ppc64le-linux-gnu aarch64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=21
+ls -1 ../../build/devkit/result/
+x86_64-linux-gnu-to-aarch64-linux-gnu
+x86_64-linux-gnu-to-ppc64le-linux-gnu
+```
+
+Notice that devkits are not only useful for targeting different build
+platforms. Because they contain the full build dependencies for a
+system (i.e. compiler and root file system), they can easily be used
+to build well-known, reliable and reproducible build environments. You
+can for example create and use a devkit with GCC 7.3 and a Fedora 12
+sysroot environment (with glibc 2.11) on Ubuntu 14.04 (which doesn't
+have GCC 7.3 by default) to produce OpenJDK binaries which will run on
+all Linux systems with runtime libraries newer than the ones from
+Fedora 12 (e.g. Ubuntu 16.04, SLES 11 or RHEL 6).
+
### Boot JDK and Build JDK
When cross-compiling, make sure you use a boot JDK that runs on the *build*
--- a/make/devkit/Makefile Thu Nov 22 15:38:20 2018 +0100
+++ b/make/devkit/Makefile Thu Nov 22 17:22:40 2018 +0100
@@ -41,9 +41,12 @@
# To build the full set of crosstools for additional platforms, use a command
# line looking like this:
#
-# make cross_compile_target="aarch64-linux-gnu" BASE_OS=Fedora27
+# make TARGETS="aarch64-linux-gnu" BASE_OS=Fedora
# or
-# make cross_compile_target="arm-linux-gnueabihf" BASE_OS=Fedora27
+# make TARGETS="arm-linux-gnueabihf ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
+#
+# to build several devkits for a specific OS version at once.
+# You can find the final results under ../../build/devkit/result/<host>-to-<target>
#
# This is the makefile which iterates over all host and target platforms.
#
@@ -52,18 +55,18 @@
cpu := $(shell uname -p)
# Figure out what platform this is building on.
-me := $(cpu)-$(if $(findstring Linux,$(os)),unknown-linux-gnu)
+me := $(cpu)-$(if $(findstring Linux,$(os)),linux-gnu)
$(info Building on platform $(me))
#
# By default just build for the current platform, which is assumed to be Linux
#
-ifeq ($(cross_compile_target), )
+ifeq ($(TARGETS), )
platforms := $(me)
host_platforms := $(platforms)
else
- platforms := $(cross_compile_target)
+ platforms := $(TARGETS)
host_platforms := $(me)
endif
target_platforms := $(platforms)
@@ -79,37 +82,37 @@
OUTPUT_ROOT = $(abspath ../../build/devkit)
RESULT = $(OUTPUT_ROOT)/result
-submakevars = HOST=$@ BUILD=$(me) \
- RESULT=$(RESULT) PREFIX=$(RESULT)/$@ \
- OUTPUT_ROOT=$(OUTPUT_ROOT)
+submakevars = HOST=$@ BUILD=$(me) RESULT=$(RESULT) OUTPUT_ROOT=$(OUTPUT_ROOT)
+
$(host_platforms) :
@echo 'Building compilers for $@'
@echo 'Targets: $(target_platforms)'
for p in $(filter $@, $(target_platforms)) $(filter-out $@, $(target_platforms)); do \
- $(MAKE) -f Tools.gmk download-rpms $(submakevars) TARGET=$$p && \
+ $(MAKE) -f Tools.gmk download-rpms $(submakevars) \
+ TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
$(MAKE) -f Tools.gmk all $(submakevars) \
- TARGET=$$p || exit 1 ; \
+ TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
+ $(MAKE) -f Tools.gmk ccache $(submakevars) \
+ TARGET=$@ PREFIX=$(RESULT)/$@-to-$$p BUILDDIR=$(OUTPUT_ROOT)/$@/$$p || exit 1 ; \
done
- @echo 'Building ccache program for $@'
- $(MAKE) -f Tools.gmk ccache $(submakevars) TARGET=$@
@echo 'All done"'
today := $(shell date +%Y%m%d)
define Mktar
- $(1)_tar = $$(RESULT)/sdk-$(1)-$$(today).tar.gz
- $$($(1)_tar) : PLATFORM = $(1)
- TARFILES += $$($(1)_tar)
- $$($(1)_tar) : $(1) $$(shell find $$(RESULT)/$(1))
+ $(1)-to-$(2)_tar = $$(RESULT)/sdk-$(1)-to-$(2)-$$(today).tar.gz
+ $$($(1)-to-$(2)_tar) : PLATFORM = $(1)-to-$(2)
+ TARFILES += $$($(1)-to-$(2)_tar)
+ $$($(1)-to-$(2)_tar) : $$(shell find $$(RESULT)/$(1)-to-$(2) -type f)
endef
-$(foreach p,$(host_platforms),$(eval $(call Mktar,$(p))))
+$(foreach p,$(host_platforms),$(foreach t,$(target_platforms),$(eval $(call Mktar,$(p),$(t)))))
tars : all $(TARFILES)
onlytars : $(TARFILES)
%.tar.gz :
@echo 'Creating compiler package $@'
- cd $(RESULT)/$(PLATFORM) && tar -czf $@ *
+ cd $(RESULT) && tar -czf $@ $(PLATFORM)/*
touch $@
clean :
--- a/make/devkit/Tools.gmk Thu Nov 22 15:38:20 2018 +0100
+++ b/make/devkit/Tools.gmk Thu Nov 22 17:22:40 2018 +0100
@@ -52,16 +52,25 @@
$(info ARCH=$(ARCH))
ifeq ($(BASE_OS), OEL6)
- OEL_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
+ BASE_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
LINUX_VERSION := OEL6.4
-else ifeq ($(BASE_OS), Fedora27)
- ifeq ($(ARCH), aarch64)
- FEDORA_TYPE=fedora-secondary
+else ifeq ($(BASE_OS), Fedora)
+ DEFAULT_OS_VERSION := 27
+ ifeq ($(BASE_OS_VERSION), )
+ BASE_OS_VERSION := $(DEFAULT_OS_VERSION)
+ endif
+ ifeq ($(filter x86_64 armhfp, $(ARCH)), )
+ FEDORA_TYPE := fedora-secondary
else
- FEDORA_TYPE=fedora/linux
+ FEDORA_TYPE := fedora/linux
endif
- OEL_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/27/Everything/$(ARCH)/os/Packages/
- LINUX_VERSION := Fedora 27
+ ARCHIVED := $(shell [ $(BASE_OS_VERSION) -lt $(DEFAULT_OS_VERSION) ] && echo true)
+ ifeq ($(ARCHIVED),true)
+ BASE_URL := https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
+ else
+ BASE_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
+ endif
+ LINUX_VERSION := Fedora_$(BASE_OS_VERSION)
else
$(error Unknown base OS $(BASE_OS))
endif
@@ -137,13 +146,11 @@
endif
# Define directories
-RESULT := $(OUTPUT_ROOT)/result
BUILDDIR := $(OUTPUT_ROOT)/$(HOST)/$(TARGET)
-PREFIX := $(RESULT)/$(HOST)
TARGETDIR := $(PREFIX)/$(TARGET)
SYSROOT := $(TARGETDIR)/sysroot
DOWNLOAD := $(OUTPUT_ROOT)/download
-DOWNLOAD_RPMS := $(DOWNLOAD)/rpms
+DOWNLOAD_RPMS := $(DOWNLOAD)/rpms/$(TARGET)-$(LINUX_VERSION)
SRCDIR := $(OUTPUT_ROOT)/src
# Marker file for unpacking rpms
@@ -159,7 +166,7 @@
# Only run this if rpm dir is empty.
ifeq ($(wildcard $(DOWNLOAD_RPMS)/*.rpm), )
cd $(DOWNLOAD_RPMS) && \
- wget -r -np -nd $(patsubst %, -A "*%*.rpm", $(RPM_LIST)) $(OEL_URL)
+ wget -r -np -nd $(patsubst %, -A "*%*.rpm", $(RPM_LIST)) $(BASE_URL)
endif
##########################################################################################
@@ -190,8 +197,8 @@
##########################################################################################
# Unpack RPMS
+RPM_ARCHS := $(ARCH) noarch
ifeq ($(ARCH),x86_64)
- RPM_ARCHS := x86_64 noarch
ifeq ($(BUILD),$(HOST))
ifeq ($(TARGET),$(HOST))
# When building the native compiler for x86_64, enable mixed mode.
@@ -199,11 +206,9 @@
endif
endif
else ifeq ($(ARCH),i686)
- RPM_ARCHS := i386 i686 noarch
+ RPM_ARCHS += i386
else ifeq ($(ARCH), armhfp)
- RPM_ARCHS := $(ARCH) armv7hl noarch
-else
- RPM_ARCHS := $(ARCH) noarch
+ RPM_ARCHS += armv7hl
endif
RPM_FILE_LIST := $(sort $(foreach a, $(RPM_ARCHS), \
@@ -277,7 +282,7 @@
--host=$(HOST) --build=$(BUILD) \
--prefix=$(PREFIX)
-PATHEXT = $(RESULT)/$(BUILD)/bin:
+PATHEXT = $(PREFIX)/bin:
PATHPRE = PATH=$(PATHEXT)$(PATH)
NUM_CORES := $(shell cat /proc/cpuinfo | grep -c processor)
@@ -427,6 +432,11 @@
$(BUILDDIR)/$(gcc_ver)/Makefile : CONFIG += --with-float=hard
endif
+ifneq ($(filter ppc64 ppc64le s390x, $(ARCH)), )
+ # We only support 64-bit on these platforms anyway
+ CONFIG += --disable-multilib
+endif
+
# Want:
# c,c++
# shared libs
@@ -552,7 +562,7 @@
##########################################################################################
-$(PREFIX)/devkit.info: FRC
+$(PREFIX)/devkit.info:
@echo 'Creating devkit.info in the root of the kit'
rm -f $@
touch $@
@@ -611,7 +621,4 @@
# this is only built for host. so separate.
ccache : $(ccache)
-# Force target
-FRC:
-
.PHONY : gcc all binutils bfdlib link_libs rpms libs sysroot