Temurin-build: Prototype "Reproducible Build" using current available Adopt jenkins job & infra framework and "build info"

Created on 21 Apr 2021  ·  26Comments  ·  Source: adoptium/temurin-build

This is a work item for EPIC https://github.com/AdoptOpenJDK/openjdk-build/issues/2522.
The aim being to "prototype" an initial attempt at being able to "recreate" an Adopt build based on specifying the basic "build info" currently available for an existing "release", and assuming the current available "infra" setup.

Design:

Assumptions for 1st prototype:

  1. Hotspot only
  2. no installer
  3. no signing

Spec:
Extend the current openjdk-pipeline & build jobs to take as extra inputs basic "build info":

  • Openjdk "mirror" source git "commit SHA" or "tag"
  • openjdk-build repo SHA
  • ci-jenkins-pipelines repo SHA
    • -
compatibility enhancement infra installer jenkins
Source
picture andrew-m-leonard

All 26 comments

The Jenkins build jobs are currently "generated" whenever openjdk-build or ci-jenkins-pipelines changes occur. The contents of these jobs also contain an extended amount of JSON configuration/settings, both the job parameters and the JSON settings are specific to the build setup at the time of the release/build.
It thus makes sense to architect the prototype as a two step process:

  1. "Generate" required level of openjdk-pipeline & build jobs based on "build info"
  2. Execute the generated jobs..
andrew-m-leonard picture andrew-m-leonard on 21 Apr 2021

So utilizing the Adopt Build job USER build scripts settings to point at a SHA of a level of the openjdk-build scripts you want to use (eg.January 19th CPU Update), is quite easy to do. Here is a build of January jdk-11.0.10+9 Hotspot Release: https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-linux-x64-hotspot/960/parameters/
So this is a re-build using the january build scripts of the january source tag, using the current infrastructure and tooling.
Q: Is this "reproduced"? A: Maybe!
In reality all this has done has re-built that source SHA using the current infrastructure and tooling levels. So as such, it's probably using different versions of numerous dependencies, especially for non-container build platforms.
The build above was on xLinux, so the build container it was using was from the current adoptopenjdk/centos6_build_image image. Improvement could be achieved if we cached along side each release we mark as reproducible the docker build images used to build them, so that would improve reproducibility of platforms:

  • xLinux
  • pLinux
  • aarch64Linux
    but not with:
  • Mac
  • Windows
  • zLinux
  • AIX
    Docker build containers could be created with some work for Mac, Windows and zLinux, but AIX would be a problem.
andrew-m-leonard picture andrew-m-leonard on 30 Apr 2021

Docker does have its limitations however, the use of dockerhub is one potential security exposure, with the reliance of storing/pulling images to build a secure JDK over the internet, not ideal.
Also Docker is not available on AIX.
An alternative approach maybe "cross-compiling" type setup, whereby the system dependency set is fixed through a defined location.

andrew-m-leonard picture andrew-m-leonard on 30 Apr 2021

Rebuilding Jan CPU Update with the current Jenkins job:

{
    "ARCHITECTURE": "x64",
    "TARGET_OS": "linux",
    "VARIANT": "hotspot",
    "JAVA_TO_BUILD": "jdk11u",
    "TEST_LIST": [
        "sanity.openjdk",
        "sanity.system",
        "extended.system",
        "sanity.perf",
        "sanity.external"
    ],
    "SCM_REF": "jdk-11.0.10+9_adopt",
    "BUILD_ARGS": "",
    "NODE_LABEL": "build&&linux&&x64",
    "ADDITIONAL_TEST_LABEL": "",
    "KEEP_TEST_REPORTDIR": false,
    "ACTIVE_NODE_TIMEOUT": "",
    "CODEBUILD": false,
    "DOCKER_IMAGE": "adoptopenjdk/centos6_build_image@sha256:5655c9a66f4528210e307d033024bd65f5f736eac401fdeaed9dbc903b629a0c",
    "DOCKER_NODE": "",
    "DOCKER_REGISTRY": "",
    "DOCKER_CREDENTIAL": "",
    "PLATFORM_CONFIG_LOCATION": "AdoptOpenJDK/openjdk-build/master/build-farm/platform-specific-configurations",
    "CONFIGURE_ARGS": "--enable-dtrace=auto",
    "OVERRIDE_FILE_NAME_VERSION": "",
    "USE_ADOPT_SHELL_SCRIPTS": false,
    "RELEASE": true,
    "PUBLISH_NAME": "jdk-11.0.10+9",
    "ADOPT_BUILD_NUMBER": "",
    "ENABLE_TESTS": false,
    "ENABLE_INSTALLERS": true,
    "ENABLE_SIGNER": true,
    "CLEAN_WORKSPACE": true,
    "CLEAN_WORKSPACE_AFTER": true,
    "CLEAN_WORKSPACE_BUILD_OUTPUT_ONLY_AFTER": false
}

DEFAULTS_JSON:

  "defaultsUrl": "https://raw.githubusercontent.com/andrew-m-leonard/ci-jenkins-pipelines/myrepo/pipelines/defaults.json"

In the "defaultsUrl" location:

    "repository"             : {
        "build_url"          : "https://github.com/AdoptOpenJDK/openjdk-build.git",
        "build_branch"       : "ea53ac3d1623de5ecefdcd9aee3379a776be84b2",

build_branch SHA is of the openjdk-build scripts commit used to build Jan CPU update.

Note, DOCKER_IMAGE uses @sha256:56... SHA digest of docker image used to build. Didn't have the exact SHA digest, but this could be used if we did store it.

andrew-m-leonard picture andrew-m-leonard on 4 May 2021

diffoscope of the original Release jdk.tar.gz and the Reproduced one, shows many differences, which is not surprising really, as things like timestamps are different everywhere, also any filelists or zip contents(eg.jmods) have files in different orders.
However, doing a high level look at the folders and libs, it seems identical,eg:
Original "lib":

-rw-r--r--  1 andrewleonard  staff      38933 20 Jan 12:17 classlist
-rw-r--r--  1 andrewleonard  staff    4496469 20 Jan 12:17 ct.sym
-rwxr-xr-x  1 andrewleonard  staff      13040 20 Jan 12:17 jexec
drwxr-xr-x  4 andrewleonard  staff        128 20 Jan 12:17 jfr
drwxr-xr-x  3 andrewleonard  staff         96 20 Jan 12:17 jli
-rw-r--r--  1 andrewleonard  staff     106676 20 Jan 12:17 jrt-fs.jar
-rwxr-xr-x  1 andrewleonard  staff      18408 20 Jan 12:17 jspawnhelper
-rw-r--r--  1 andrewleonard  staff         29 20 Jan 12:17 jvm.cfg
-rw-r--r--  1 andrewleonard  staff      13344 20 Jan 12:17 libattach.so
-rw-r--r--  1 andrewleonard  staff     752736 20 Jan 12:17 libawt.so
-rw-r--r--  1 andrewleonard  staff      44728 20 Jan 12:17 libawt_headless.so
-rw-r--r--  1 andrewleonard  staff     484632 20 Jan 12:17 libawt_xawt.so
-rw-r--r--  1 andrewleonard  staff      28216 20 Jan 12:17 libdt_socket.so
-rw-r--r--  1 andrewleonard  staff      12992 20 Jan 12:17 libextnet.so
-rw-r--r--  1 andrewleonard  staff      67984 20 Jan 12:17 libfontmanager.so
-rw-r--r--  1 andrewleonard  staff    1888936 20 Jan 12:17 libharfbuzz.so
-rw-r--r--  1 andrewleonard  staff      51512 20 Jan 12:17 libinstrument.so
-rw-r--r--  1 andrewleonard  staff      47496 20 Jan 12:17 libj2gss.so
-rw-r--r--  1 andrewleonard  staff      18024 20 Jan 12:17 libj2pcsc.so
-rw-r--r--  1 andrewleonard  staff      88992 20 Jan 12:17 libj2pkcs11.so
-rw-r--r--  1 andrewleonard  staff       8064 20 Jan 12:17 libjaas.so
-rw-r--r--  1 andrewleonard  staff     214128 20 Jan 12:17 libjava.so
-rw-r--r--  1 andrewleonard  staff     262728 20 Jan 12:17 libjavajpeg.so
-rw-r--r--  1 andrewleonard  staff       7984 20 Jan 12:17 libjawt.so
-rw-r--r--  1 andrewleonard  staff     282752 20 Jan 12:17 libjdwp.so
-rw-r--r--  1 andrewleonard  staff     596448 20 Jan 12:17 libjimage.so
-rw-r--r--  1 andrewleonard  staff      13248 20 Jan 12:17 libjsig.so
-rw-r--r--  1 andrewleonard  staff      82936 20 Jan 12:17 libjsound.so
-rw-r--r--  1 andrewleonard  staff     576856 20 Jan 12:17 liblcms.so
-rw-r--r--  1 andrewleonard  staff      26128 20 Jan 12:17 libmanagement.so
-rw-r--r--  1 andrewleonard  staff       8040 20 Jan 12:17 libmanagement_agent.so
-rw-r--r--  1 andrewleonard  staff      33744 20 Jan 12:17 libmanagement_ext.so
-rw-r--r--  1 andrewleonard  staff     585064 20 Jan 12:17 libmlib_image.so
-rw-r--r--  1 andrewleonard  staff     111792 20 Jan 12:17 libnet.so
-rw-r--r--  1 andrewleonard  staff      88120 20 Jan 12:17 libnio.so
-rw-r--r--  1 andrewleonard  staff       8224 20 Jan 12:17 libprefs.so
-rw-r--r--  1 andrewleonard  staff       7744 20 Jan 12:17 librmi.so
-rw-r--r--  1 andrewleonard  staff      55664 20 Jan 12:17 libsaproc.so
-rw-r--r--  1 andrewleonard  staff      28960 20 Jan 12:17 libsctp.so
-rw-r--r--  1 andrewleonard  staff     373816 20 Jan 12:17 libsplashscreen.so
-rw-r--r--  1 andrewleonard  staff     776960 20 Jan 12:17 libsunec.so
-rw-r--r--  1 andrewleonard  staff     672416 20 Jan 12:17 libunpack.so
-rw-r--r--  1 andrewleonard  staff      81432 20 Jan 12:17 libverify.so
-rw-r--r--  1 andrewleonard  staff      37616 20 Jan 12:17 libzip.so
-rw-r--r--  1 andrewleonard  staff  142006963 20 Jan 12:17 modules
-rw-r--r--  1 andrewleonard  staff       3793 20 Jan 12:17 psfont.properties.ja
-rw-r--r--  1 andrewleonard  staff      11390 20 Jan 12:17 psfontj2d.properties
drwxr-xr-x  6 andrewleonard  staff        192 20 Jan 12:17 security
drwxr-xr-x  5 andrewleonard  staff        160 20 Jan 12:17 server
-rw-r--r--  1 andrewleonard  staff   58148661 20 Jan 12:17 src.zip
-rw-r--r--  1 andrewleonard  staff     107853 20 Jan 12:17 tzdb.dat

Reproduced "lib":

-rw-r--r--  1 andrewleonard  staff      38933 30 Apr 11:07 classlist
-rw-r--r--  1 andrewleonard  staff    4496469 30 Apr 11:07 ct.sym
-rwxr-xr-x  1 andrewleonard  staff      13040 30 Apr 11:07 jexec
drwxr-xr-x  4 andrewleonard  staff        128 30 Apr 11:07 jfr
drwxr-xr-x  3 andrewleonard  staff         96 30 Apr 11:07 jli
-rw-r--r--  1 andrewleonard  staff     106676 30 Apr 11:07 jrt-fs.jar
-rwxr-xr-x  1 andrewleonard  staff      18408 30 Apr 11:07 jspawnhelper
-rw-r--r--  1 andrewleonard  staff         29 30 Apr 11:07 jvm.cfg
-rw-r--r--  1 andrewleonard  staff      13344 30 Apr 11:07 libattach.so
-rw-r--r--  1 andrewleonard  staff     752736 30 Apr 11:07 libawt.so
-rw-r--r--  1 andrewleonard  staff      44728 30 Apr 11:07 libawt_headless.so
-rw-r--r--  1 andrewleonard  staff     484632 30 Apr 11:07 libawt_xawt.so
-rw-r--r--  1 andrewleonard  staff      28216 30 Apr 11:07 libdt_socket.so
-rw-r--r--  1 andrewleonard  staff      12992 30 Apr 11:07 libextnet.so
-rw-r--r--  1 andrewleonard  staff      67984 30 Apr 11:07 libfontmanager.so
-rw-r--r--  1 andrewleonard  staff    1888936 30 Apr 11:07 libharfbuzz.so
-rw-r--r--  1 andrewleonard  staff      51512 30 Apr 11:07 libinstrument.so
-rw-r--r--  1 andrewleonard  staff      47496 30 Apr 11:07 libj2gss.so
-rw-r--r--  1 andrewleonard  staff      18024 30 Apr 11:07 libj2pcsc.so
-rw-r--r--  1 andrewleonard  staff      88992 30 Apr 11:07 libj2pkcs11.so
-rw-r--r--  1 andrewleonard  staff       8064 30 Apr 11:07 libjaas.so
-rw-r--r--  1 andrewleonard  staff     214128 30 Apr 11:07 libjava.so
-rw-r--r--  1 andrewleonard  staff     262728 30 Apr 11:07 libjavajpeg.so
-rw-r--r--  1 andrewleonard  staff       7984 30 Apr 11:07 libjawt.so
-rw-r--r--  1 andrewleonard  staff     282752 30 Apr 11:07 libjdwp.so
-rw-r--r--  1 andrewleonard  staff     596448 30 Apr 11:07 libjimage.so
-rw-r--r--  1 andrewleonard  staff      13248 30 Apr 11:07 libjsig.so
-rw-r--r--  1 andrewleonard  staff      82936 30 Apr 11:07 libjsound.so
-rw-r--r--  1 andrewleonard  staff     576856 30 Apr 11:07 liblcms.so
-rw-r--r--  1 andrewleonard  staff      26128 30 Apr 11:07 libmanagement.so
-rw-r--r--  1 andrewleonard  staff       8040 30 Apr 11:07 libmanagement_agent.so
-rw-r--r--  1 andrewleonard  staff      33744 30 Apr 11:07 libmanagement_ext.so
-rw-r--r--  1 andrewleonard  staff     585064 30 Apr 11:07 libmlib_image.so
-rw-r--r--  1 andrewleonard  staff     111792 30 Apr 11:07 libnet.so
-rw-r--r--  1 andrewleonard  staff      88120 30 Apr 11:07 libnio.so
-rw-r--r--  1 andrewleonard  staff       8224 30 Apr 11:07 libprefs.so
-rw-r--r--  1 andrewleonard  staff       7744 30 Apr 11:07 librmi.so
-rw-r--r--  1 andrewleonard  staff      55664 30 Apr 11:07 libsaproc.so
-rw-r--r--  1 andrewleonard  staff      28960 30 Apr 11:07 libsctp.so
-rw-r--r--  1 andrewleonard  staff     373816 30 Apr 11:07 libsplashscreen.so
-rw-r--r--  1 andrewleonard  staff     776960 30 Apr 11:07 libsunec.so
-rw-r--r--  1 andrewleonard  staff     672416 30 Apr 11:07 libunpack.so
-rw-r--r--  1 andrewleonard  staff      81432 30 Apr 11:07 libverify.so
-rw-r--r--  1 andrewleonard  staff      37616 30 Apr 11:07 libzip.so
-rw-r--r--  1 andrewleonard  staff  142006992 30 Apr 11:07 modules
-rw-r--r--  1 andrewleonard  staff       3793 30 Apr 11:07 psfont.properties.ja
-rw-r--r--  1 andrewleonard  staff      11390 30 Apr 11:07 psfontj2d.properties
drwxr-xr-x  6 andrewleonard  staff        192 30 Apr 11:07 security
drwxr-xr-x  5 andrewleonard  staff        160 30 Apr 11:07 server
-rw-r--r--  1 andrewleonard  staff   58148665 30 Apr 11:07 src.zip
-rw-r--r--  1 andrewleonard  staff     107853 30 Apr 11:07 tzdb.dat

The only two files differing in size are "modules" and "src.zip":

-rw-r--r--  1 andrewleonard  staff  142006963 20 Jan 12:17 modules
-rw-r--r--  1 andrewleonard  staff   58148661 20 Jan 12:17 src.zip

-rw-r--r--  1 andrewleonard  staff  142006992 30 Apr 11:07 modules
-rw-r--r--  1 andrewleonard  staff   58148665 30 Apr 11:07 src.zip
andrew-m-leonard picture andrew-m-leonard on 4 May 2021

Some jmods Release vs Reproduce:

-rw-r--r--  1 andrewleonard  staff  21632332 20 Jan 12:17 java.base.jmod
-rw-r--r--  1 andrewleonard  staff    119087 20 Jan 12:17 java.compiler.jmod
-rw-r--r--  1 andrewleonard  staff     58585 20 Jan 12:17 java.datatransfer.jmod
-rw-r--r--  1 andrewleonard  staff  13577286 20 Jan 12:17 java.desktop.jmod
-rw-r--r--  1 andrewleonard  staff    416817 20 Jan 12:17 jdk.internal.vm.ci.jmod
-rw-r--r--  1 andrewleonard  staff   6178633 20 Jan 12:17 jdk.internal.vm.compiler.jmod
-rw-r--r--  1 andrewleonard  staff     20467 20 Jan 12:17 jdk.internal.vm.compiler.management.jmod

-rw-r--r--  1 andrewleonard  staff  21632178 30 Apr 11:07 java.base.jmod
-rw-r--r--  1 andrewleonard  staff    119087 30 Apr 11:07 java.compiler.jmod
-rw-r--r--  1 andrewleonard  staff     58585 30 Apr 11:07 java.datatransfer.jmod
-rw-r--r--  1 andrewleonard  staff  13577286 30 Apr 11:07 java.desktop.jmod
-rw-r--r--  1 andrewleonard  staff    416817 30 Apr 11:07 jdk.internal.vm.ci.jmod
-rw-r--r--  1 andrewleonard  staff   6178634 30 Apr 11:07 jdk.internal.vm.compiler.jmod
-rw-r--r--  1 andrewleonard  staff     20467 30 Apr 11:07 jdk.internal.vm.compiler.management.jmod

Minor size differences.

andrew-m-leonard picture andrew-m-leonard on 4 May 2021

There are several important aspects of "Reproducibility" that all work together:

  • Being able to "re-build" a release, re-producing the "same output"
  • Knowing exactly all the "dependencies" and "infra" to build a given release, and knowing each and every one of them is "secure" and "not vulnerable"
  • Security (access/auth) of the "infra" and all servers used for storing "dependencies"
  • Auditable "change management" on all "infra", "tooling" and "dependency servers"
andrew-m-leonard picture andrew-m-leonard on 5 May 2021

Leading on from the "import aspects", we ought to clarify the "Requirements" again with those aspects in mind, and also reproducible-builds.org "How?":

1. "Re-build" a release, re-producing the "same output"

  • "build system needs to be made entirely deterministic" & "build environment should either be recorded or pre-defined":

    • Know exactly all the "tooling", "dependencies" and "infra" to build a given release

    • Know exactly the version of "builds scripts" used to build a given release

  • "validate that the output matches the original build":

    • A "method" is required to validate that a reproduced build is in fact the same as the original release

      2. Security (access/auth) of the "infra" and all servers used for storing "dependencies"

  • Source of "build scripts", "tooling" and "dependencies" must be "trusted" and "verified"

  • Access to "infra" must be under "access control", and "audited"

    3. Auditable "change management" on all "infra", "tooling" and "dependency servers"

  • All changes to "build-scripts", "tooling" and "dependency" servers must be "change controlled"

  • All changes to build "infra" must be "change controlled", no ad-hoc changes to production infra
andrew-m-leonard picture andrew-m-leonard on 7 May 2021

Prototype setting up a new ubuntu 20.04 jdk17 hotspot build environment within a docker container:

  1. Create base container 
docker pull ubuntu:20.04
  1. Start container:
docker run -i -t <container ID> /bin/bash
  1. Install dependencies:
  2. git
root@06acd4f5d2a9:/# apt install git
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ca-certificates git-man krb5-locales less libasn1-8-heimdal libbrotli1 libbsd0 libcbor0.6 libcurl3-gnutls libedit2 liberror-perl libexpat1
  libfido2-1 libgdbm-compat4 libgdbm6 libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal
  libhx509-5-heimdal libk5crypto3 libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libperl5.30
  libpsl5 libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libsqlite3-0 libssh-4 libssl1.1 libwind0-heimdal libx11-6
  libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 netbase openssh-client openssl patch perl perl-modules-5.30 publicsuffix xauth
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn gdbm-l10n krb5-doc
  krb5-user libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql keychain
  libpam-ssh monkeysphere ssh-askpass ed diffutils-doc perl-doc libterm-readline-gnu-perl | libterm-readline-perl-perl make libb-debug-perl
  liblocale-codes-perl
The following NEW packages will be installed:
  ca-certificates git git-man krb5-locales less libasn1-8-heimdal libbrotli1 libbsd0 libcbor0.6 libcurl3-gnutls libedit2 liberror-perl libexpat1
  libfido2-1 libgdbm-compat4 libgdbm6 libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal
  libhx509-5-heimdal libk5crypto3 libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libperl5.30
  libpsl5 libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libsqlite3-0 libssh-4 libssl1.1 libwind0-heimdal libx11-6
  libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 netbase openssh-client openssl patch perl perl-modules-5.30 publicsuffix xauth
0 upgraded, 56 newly installed, 0 to remove and 0 not upgraded.

As you can see installing "git" alone installs 56 new pkgs, do we need to record all those?

Dependency scanning?

We could install all the required dependencies, then simply "scan"/"list" ALL the pkgs and "record" those as "build info"? What does that achieve? Not sure it's reproducible, as not easy to re-build with that exact set of 100s of specific versions.

andrew-m-leonard picture andrew-m-leonard on 12 May 2021

An excellent project working on and discussing various approaches and issues with reproducible builds: https://wiki.yoctoproject.org/wiki/Reproducible_Builds

andrew-m-leonard picture andrew-m-leonard on 13 May 2021

From the evidence my thinking the difficulty in achieving "exact binary reproducibility" and keeping it there actually don't achieve enough benefit for the cost, the continual effort to fix upstream and other tooling products that might be introducing a single timestamp where you don't want it is simply not a useful use of effort. We ought to concentrate on the benefits that "reproducible builds" is aiming to achieve,eg.trusted supply chain.

andrew-m-leonard picture andrew-m-leonard on 13 May 2021

As you can see installing "git" alone installs 56 new pkgs, do we need to record all those?

Since git is not directly used as part of the build itself (another git version will generally extract the same source, a different version of the build tools will not necessarily produce the same output) I'd tentatively say no (at least as a first pass) although recording all versions used for a build is probably useful.

sxa picture sxa on 13 May 2021

openjdk16 at a minimum is required for "reproducible" Java apps, as prior to that things like jmods/jars had random "hashes"

andrew-m-leonard picture andrew-m-leonard on 13 May 2021

strace -ff -e trace=openat cat /etc/vdpau_wrapper.cfg
openat(AT_FDCWD, “/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, “/lib64/libc.so.6”, O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, “/usr/lib/locale/locale-archive”, O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, “/etc/vdpau_wrapper.cfg”, O_RDONLY) = 3
enable_flash_uv_swap=1
disable_flash_pq_bg_color=1
+++ exited with 0 +++

stap -e ‘probe syscall.open { printf (“UID %d: %s(%d) open (%s)\n”, uid(), execname(), pid(), argstr) }’

andrew-m-leonard picture andrew-m-leonard on 13 May 2021
👍1

Tooling:

  • git
  • build-essential:
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev g++ g++-9 gcc gcc-9 gcc-9-base libasan5
  libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libgcc-9-dev libgomp1
  libisl22 libitm1 liblsan0 libmpc3 libmpfr6 libquadmath0 libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make xz-utils

apt list -a :

binutils-common/focal-updates,focal-security,now 2.34-6ubuntu1.1 amd64 [installed,automatic]
binutils-common/focal 2.34-6ubuntu1 amd64

binutils-x86-64-linux-gnu/focal-updates,focal-security,now 2.34-6ubuntu1.1 amd64 [installed,automatic]
binutils-x86-64-linux-gnu/focal 2.34-6ubuntu1 amd64

binutils/focal-updates,focal-security,now 2.34-6ubuntu1.1 amd64 [installed,automatic]
binutils/focal 2.34-6ubuntu1 amd64

build-essential/focal-updates,now 12.8ubuntu1.1 amd64 [installed]
build-essential/focal 12.8ubuntu1 amd64

cpp-9/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
cpp-9/focal 9.3.0-10ubuntu2 amd64

cpp/focal,now 4:9.3.0-1ubuntu2 amd64 [installed,automatic]

dpkg-dev/focal,now 1.19.7ubuntu3 all [installed,automatic]

g++-9/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
g++-9/focal 9.3.0-10ubuntu2 amd64

g++/focal,now 4:9.3.0-1ubuntu2 amd64 [installed,automatic]

gcc-9-base/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
gcc-9-base/focal 9.3.0-10ubuntu2 amd64

gcc-9/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
gcc-9/focal 9.3.0-10ubuntu2 amd64

gcc/focal,now 4:9.3.0-1ubuntu2 amd64 [installed,automatic]

libasan5/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
libasan5/focal 9.3.0-10ubuntu2 amd64

libatomic1/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libatomic1/focal 10-20200411-0ubuntu1 amd64

libbinutils/focal-updates,focal-security,now 2.34-6ubuntu1.1 amd64 [installed,automatic]
libbinutils/focal 2.34-6ubuntu1 amd64

libc-dev-bin/focal-updates,now 2.31-0ubuntu9.2 amd64 [installed,automatic]
libc-dev-bin/focal 2.31-0ubuntu9 amd64

libc6-dev/focal-updates,now 2.31-0ubuntu9.2 amd64 [installed,automatic]
libc6-dev/focal 2.31-0ubuntu9 amd64

libcc1-0/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libcc1-0/focal 10-20200411-0ubuntu1 amd64

libcrypt-dev/focal,now 1:4.4.10-10ubuntu4 amd64 [installed,automatic]

libctf-nobfd0/focal-updates,focal-security,now 2.34-6ubuntu1.1 amd64 [installed,automatic]
libctf-nobfd0/focal 2.34-6ubuntu1 amd64

libctf0/focal-updates,focal-security,now 2.34-6ubuntu1.1 amd64 [installed,automatic]
libctf0/focal 2.34-6ubuntu1 amd64

libdpkg-perl/focal,now 1.19.7ubuntu3 all [installed,automatic]

libgcc-9-dev/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
libgcc-9-dev/focal 9.3.0-10ubuntu2 amd64

libgomp1/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libgomp1/focal 10-20200411-0ubuntu1 amd64

libisl22/focal,now 0.22.1-1 amd64 [installed,automatic]

libitm1/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libitm1/focal 10-20200411-0ubuntu1 amd64

liblsan0/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
liblsan0/focal 10-20200411-0ubuntu1 amd64

libmpc3/focal,now 1.1.0-1 amd64 [installed,automatic]

libmpfr6/focal,now 4.0.2-1 amd64 [installed,automatic]

libquadmath0/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libquadmath0/focal 10-20200411-0ubuntu1 amd64

libstdc++-9-dev/focal-updates,focal-security,now 9.3.0-17ubuntu1~20.04 amd64 [installed,automatic]
libstdc++-9-dev/focal 9.3.0-10ubuntu2 amd64

libtsan0/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libtsan0/focal 10-20200411-0ubuntu1 amd64

libubsan1/focal-updates,focal-security,now 10.2.0-5ubuntu1~20.04 amd64 [installed,automatic]
libubsan1/focal 10-20200411-0ubuntu1 amd64

linux-libc-dev/focal-updates,focal-security,now 5.4.0-73.82 amd64 [installed,automatic]
linux-libc-dev/focal 5.4.0-26.30 amd64

make/focal,now 4.2.1-1.2 amd64 [installed,automatic]

xz-utils/focal-updates,now 5.2.4-1ubuntu1 amd64 [installed,automatic]
xz-utils/focal 5.2.4-1 amd64
  • curl:
The following NEW packages will be installed:
  curl libcurl4

apt list -a

curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.5 amd64 [installed]
curl/focal 7.68.0-1ubuntu2 amd64

libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.5 amd64 [installed,automatic]
libcurl4/focal 7.68.0-1ubuntu2 amd64
  • BootJDK from a trusted location: AdoptOpenJDK jdk-16.0.1+9
  • gunzip (System)
  • tar (System)
  • autoconf 
The following NEW packages will be installed:
  autoconf libsigsegv2 m4

apt list -a

autoconf/focal,now 2.69-11.1 all [installed]

libsigsegv2/focal,now 2.12-2 amd64 [installed,automatic]

m4/focal,now 1.4.18-4 amd64 [installed,automatic]
  • file
The following NEW packages will be installed:
  file libmagic-mgc libmagic1

apt list -a

file/focal,now 1:5.38-4 amd64 [installed]

libmagic-mgc/focal,now 1:5.38-4 amd64 [installed,automatic]

libmagic1/focal,now 1:5.38-4 amd64 [installed,automatic]
  • unzip
The following NEW packages will be installed:
  unzip

apt list -a

unzip/focal,now 6.0-25ubuntu1 amd64 [installed]
  • zip
The following NEW packages will be installed:
  zip

apt list -a

zip/focal,now 3.0-11build1 amd64 [installed]
  • apt-get install libx11-dev libxext-dev libxrender-dev libxrandr-dev libxtst-dev libxt-dev
The following NEW packages will be installed:
  libice-dev libice6 libpthread-stubs0-dev libsm-dev libsm6 libxau-dev libxcb1-dev libxdmcp-dev libxfixes-dev libxfixes3
  lib xi-dev libxi6 libxrandr2 libxrender1 libxt6 libxtst6 x11-common x11proto-core-dev x11proto-dev x11proto-input-dev
  x11proto-randr-dev x11proto-record-dev x11proto-xext-dev xorg-sgml-doctools xtrans-dev

apt list -a

libice-dev/focal,now 2:1.0.10-0ubuntu1 amd64 [installed,automatic]

libice6/focal,now 2:1.0.10-0ubuntu1 amd64 [installed,automatic]

libpthread-stubs0-dev/focal,now 0.4-1 amd64 [installed,automatic]

libsm-dev/focal,now 2:1.2.3-1 amd64 [installed,automatic]

libsm6/focal,now 2:1.2.3-1 amd64 [installed,automatic]

libxau-dev/focal,now 1:1.0.9-0ubuntu1 amd64 [installed,automatic]

libxcb1-dev/focal,now 1.14-2 amd64 [installed,automatic]

libxdmcp-dev/focal,now 1:1.1.3-0ubuntu1 amd64 [installed,automatic]

libxfixes-dev/focal,now 1:5.0.3-2 amd64 [installed,automatic]

libxfixes3/focal,now 1:5.0.3-2 amd64 [installed,automatic]

libxi-dev/focal,now 2:1.7.10-0ubuntu1 amd64 [installed,automatic]

libxi6/focal,now 2:1.7.10-0ubuntu1 amd64 [installed,automatic]

libxrandr2/focal,now 2:1.5.2-0ubuntu1 amd64 [installed,automatic]

libxrender1/focal,now 1:0.9.10-1 amd64 [installed,automatic]

libxt6/focal,now 1:1.1.5-1 amd64 [installed,automatic]

libxtst6/focal,now 2:1.2.3-1 amd64 [installed,automatic]

x11-common/focal,now 1:7.7+19ubuntu14 all [installed,automatic]

x11proto-core-dev/focal,now 2019.2-1ubuntu1 all [installed,automatic]

x11proto-dev/focal,now 2019.2-1ubuntu1 all [installed,automatic]

x11proto-input-dev/focal,now 2019.2-1ubuntu1 all [installed,automatic]

x11proto-randr-dev/focal,now 2019.2-1ubuntu1 all [installed,automatic]

x11proto-record-dev/focal,now 2019.2-1ubuntu1 all [installed,automatic]

x11proto-xext-dev/focal,now 2019.2-1ubuntu1 all [installed,automatic]

xorg-sgml-doctools/focal,now 1:1.11-1 all [installed,automatic]

xtrans-dev/focal,now 1.4.0-1 all [installed,automatic]
  • cups
The following NEW packages will be installed:
  libavahi-client3 libavahi-common-data libavahi-common3 libcups2 libcups2-dev libcupsimage2 libcupsimage2-dev libdbus-1-3
  libjbig-dev libjbig0 libjpeg-dev libjpeg-turbo8 libjpeg-turbo8-dev libjpeg8 libjpeg8-dev liblzma-dev libpng-dev libpng16-16
  libtiff-dev libtiff5 libtiffxx5 libwebp6 zlib1g-dev

apt list -a

libavahi-client3/focal,now 0.7-4ubuntu7 amd64 [installed,automatic]

libavahi-common-data/focal,now 0.7-4ubuntu7 amd64 [installed,automatic]

libavahi-common3/focal,now 0.7-4ubuntu7 amd64 [installed,automatic]

libcups2-dev/focal-updates,focal-security,now 2.3.1-9ubuntu1.1 amd64 [installed]
libcups2-dev/focal 2.3.1-9ubuntu1 amd64

libcups2/focal-updates,focal-security,now 2.3.1-9ubuntu1.1 amd64 [installed,automatic]
libcups2/focal 2.3.1-9ubuntu1 amd64

libcupsimage2-dev/focal-updates,focal-security,now 2.3.1-9ubuntu1.1 amd64 [installed,automatic]
libcupsimage2-dev/focal 2.3.1-9ubuntu1 amd64

libcupsimage2/focal-updates,focal-security,now 2.3.1-9ubuntu1.1 amd64 [installed,automatic]
libcupsimage2/focal 2.3.1-9ubuntu1 amd64

libdbus-1-3/focal-updates,focal-security,now 1.12.16-2ubuntu2.1 amd64 [installed,automatic]
libdbus-1-3/focal 1.12.16-2ubuntu2 amd64

libjbig-dev/focal,now 2.1-3.1build1 amd64 [installed,automatic]

libjbig0/focal,now 2.1-3.1build1 amd64 [installed,automatic]

libjpeg-dev/focal,now 8c-2ubuntu8 amd64 [installed,automatic]

libjpeg-turbo8-dev/focal-updates,focal-security,now 2.0.3-0ubuntu1.20.04.1 amd64 [installed,automatic]
libjpeg-turbo8-dev/focal 2.0.3-0ubuntu1 amd64

libjpeg-turbo8/focal-updates,focal-security,now 2.0.3-0ubuntu1.20.04.1 amd64 [installed,automatic]
libjpeg-turbo8/focal 2.0.3-0ubuntu1 amd64

libjpeg8-dev/focal,now 8c-2ubuntu8 amd64 [installed,automatic]

libjpeg8/focal,now 8c-2ubuntu8 amd64 [installed,automatic]

liblzma-dev/focal-updates,now 5.2.4-1ubuntu1 amd64 [installed,automatic]
liblzma-dev/focal 5.2.4-1 amd64

libpng-dev/focal,now 1.6.37-2 amd64 [installed,automatic]

libpng16-16/focal,now 1.6.37-2 amd64 [installed,automatic]

libtiff-dev/focal-updates,focal-security,now 4.1.0+git191117-2ubuntu0.20.04.1 amd64 [installed,automatic]
libtiff-dev/focal 4.1.0+git191117-2build1 amd64

libtiff5/focal-updates,focal-security,now 4.1.0+git191117-2ubuntu0.20.04.1 amd64 [installed,automatic]
libtiff5/focal 4.1.0+git191117-2build1 amd64

libtiffxx5/focal-updates,focal-security,now 4.1.0+git191117-2ubuntu0.20.04.1 amd64 [installed,automatic]
libtiffxx5/focal 4.1.0+git191117-2build1 amd64

libwebp6/focal,now 0.6.1-2 amd64 [installed,automatic]

zlib1g-dev/focal-updates,now 1:1.2.11.dfsg-2ubuntu1.2 amd64 [installed,automatic]
zlib1g-dev/focal 1:1.2.11.dfsg-2ubuntu1 amd64
  • fontconfig
The following NEW packages will be installed:
  fontconfig fontconfig-config fonts-dejavu-core libfontconfig1 libfreetype6 ucf

apt list -a

fontconfig-config/focal,now 2.13.1-2ubuntu3 all [installed,automatic]

fontconfig/focal,now 2.13.1-2ubuntu3 amd64 [installed]

fonts-dejavu-core/focal,now 2.37-1 all [installed,automatic]

libfontconfig1/focal,now 2.13.1-2ubuntu3 amd64 [installed,automatic]

libfreetype6/focal-updates,focal-security,now 2.10.1-2ubuntu0.1 amd64 [installed,automatic]
libfreetype6/focal 2.10.1-2 amd64

ucf/focal,now 3.0038+nmu1 all [installed,automatic]
  • libfontconfig1-dev
The following additional packages will be installed:
  libexpat1-dev libfreetype-dev libfreetype6-dev libglib2.0-0 pkg-config uuid-dev

apt list -a

libexpat1-dev/focal,now 2.2.9-1build1 amd64 [installed,automatic]

libfreetype-dev/focal-updates,focal-security,now 2.10.1-2ubuntu0.1 amd64 [installed,automatic]
libfreetype-dev/focal 2.10.1-2 amd64

libfreetype6-dev/focal-updates,focal-security,now 2.10.1-2ubuntu0.1 amd64 [installed,automatic]
libfreetype6-dev/focal 2.10.1-2 amd64

libglib2.0-0/focal-updates,focal-security,now 2.64.6-1~ubuntu20.04.3 amd64 [installed,automatic]
libglib2.0-0/focal 2.64.2-1~fakesync1 amd64

pkg-config/focal,now 0.29.1-0ubuntu4 amd64 [installed,automatic]

uuid-dev/focal-updates,now 2.34-0.1ubuntu9.1 amd64 [installed,automatic]
uuid-dev/focal 2.34-0.1ubuntu9 amd64
  • alsa
The following additional packages will be installed:
  alsa-topology-conf alsa-ucm-conf libasound2 libasound2-data

apt list -a

alsa-topology-conf/focal,now 1.2.2-1 all [installed,automatic]

alsa-ucm-conf/focal-updates,now 1.2.2-1ubuntu0.7 all [installed,automatic]
alsa-ucm-conf/focal 1.2.2-1 all

libasound2-data/focal-updates,now 1.2.2-2.1ubuntu2.4 all [installed,automatic]
libasound2-data/focal 1.2.2-2.1 all

libasound2/focal-updates,now 1.2.2-2.1ubuntu2.4 amd64 [installed,automatic]
libasound2/focal 1.2.2-2.1 amd64
andrew-m-leonard picture andrew-m-leonard on 13 May 2021

Had a go at installing and running SystemTap, unsuccessfully, issues with environment and compiling it, looks like system debug image is required. Seems a bit too low level for something we would use as part of a build.

andrew-m-leonard picture andrew-m-leonard on 14 May 2021

strace is far easier and usable to use, however the output from running a jdk build is huge, as a trace file is created per process, and 1000s of processes are created to perform a JDK build. Here is a tiny section of the "grep'ing" for "include" output within a "delta" re-build of a JDK when changing just a single .c file:

strace.out.4873:openat(AT_FDCWD, "/jdk/src/hotspot/share/include/bits/sigaction.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/src/hotspot/os/posix/include/bits/sigaction.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/lib/gcc/x86_64-linux-gnu/9/include/bits/sigaction.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/local/include/bits/sigaction.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/include/x86_64-linux-gnu/bits/sigaction.h", O_RDONLY|O_NOCTTY) = 6
strace.out.4873:openat(AT_FDCWD, "/jdk/build/linux-x86_64-server-release/support/modules_include/java.base/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/build/linux-x86_64-server-release/support/modules_include/java.base/linux/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/src/hotspot/share/include/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/src/hotspot/os/posix/include/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/lib/gcc/x86_64-linux-gnu/9/include/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/local/include/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/include/x86_64-linux-gnu/bits/sigcontext.h", O_RDONLY|O_NOCTTY) = 6
strace.out.4873:openat(AT_FDCWD, "/usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h", O_RDONLY|O_NOCTTY) = 6
strace.out.4873:openat(AT_FDCWD, "/jdk/build/linux-x86_64-server-release/support/modules_include/java.base/bits/types/stack_t.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/build/linux-x86_64-server-release/support/modules_include/java.base/linux/bits/types/stack_t.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/src/hotspot/share/include/bits/types/stack_t.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/jdk/src/hotspot/os/posix/include/bits/types/stack_t.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/lib/gcc/x86_64-linux-gnu/9/include/bits/types/stack_t.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
strace.out.4873:openat(AT_FDCWD, "/usr/local/include/bits/types/stack_t.h", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)

It does illustrate the various System headers picked up from places like /usr/include, /usr/lib, /usr/local/include, ...

andrew-m-leonard picture andrew-m-leonard on 14 May 2021

The strace test illustrates the problem of knowing exactly what system files are used to compile with. It is not practical to log, verify and determine the version of every single opened .h file for example.

andrew-m-leonard picture andrew-m-leonard on 14 May 2021

So the above has illustrated logging exactly what you've built with to the Nth degree, ie.exact versions of includes, libs, .. is very difficult. For example, it's very hard to know what version or where did the file /usr/include/bits/sigaction.h come from?
Maybe the best answer is use Docker (ignoring AIX for the moment), as long as the "images" are created in a very secure environment. Then log the exact image SHA digest used for a build. You then know exactly what a build is built with.
This approach then "moves" the problem down to the Docker image creation, at least from the "definition" and "security" perspective.

andrew-m-leonard picture andrew-m-leonard on 14 May 2021

Prototype:

Extend the openjdk-build scripts to log the exact "build info" for a jdk-17 Hotspot xLinux built in Docker

  • [ ] temurin-build SHA
  • [ ] Job "Build Config"
  • [ ] makejdk-any-platform.sh params
  • [ ] BUILD_CONFIG
  • [ ] Docker image and its SHA digest
  • [ ] Any downloaded dependency URLs, SHAs: Alsa, Freetype
  • [ ] All "source" repos and SHAs built with. NOT the values passed in as params, but log output just prior to building
  • [ ] OpenJDK Configure arguments
  • [ ] OpenJDK "make targets"

All information logged in Release meta data file

Not included in this prototype:

  1. ci-jenkins-pipelines repo used
  2. secondary repos used, eg.installer, signing, ...
andrew-m-leonard picture andrew-m-leonard on 14 May 2021
👍1

So the above has illustrated logging exactly what you've built with to the Nth degree, ie.exact versions of includes, libs, .. is very difficult. For example, it's very hard to know what version or where did the file /usr/include/bits/sigaction.h come from?

OK ... Assuming it hasn't been modified since the package was installed (which is another thing that could probably be checked) the answer would come from rpm -qf /usr/include/bits/sigaction.h (RPM based) or dpkg -S /usr/include/bits/sigaction.h (Debian-based). The E_NOENT ones can be ignored as that's just the compiler searching for where in the include path the file is. Once you have the list of files that were successfully opened, dedup-them, run that command on each of them, then de-dup that output. Since you have the output shown above perhaps something like this will probably work (untested as I don't have your file but I'm sure you can see what it's trying to do:

grep -v ENOENT strace.out.4873 | cut -d'"' -f2 | while read F; do dpkg -S "$F" >> packagelist.txt 2>> filesnotinpackage.txt; done
sort packagelist.txt | uniq
sort filesnotinpackage.txt | uniq

_NOTE: On the real build machines there will be more in filesnotinpackage.txt than on your system since, for example, gcc and all it's bits are in /usr/local/gcc* and not from the OS-supplied package as per this ansible role_

sxa picture sxa on 14 May 2021
👍1

BUILD_CONFIG

What is the difference between this and Job "Build Config" - I'd assume one is a superset of the other. I'd also suggest that this isn't quite as important as the other stuff since the bits relevant to build are captured by the script parameters and everything else is mostly about other parts of the pipelines such as a tests/installers etc. so it has the potential to cause some confusion, but no harm in including it in this first pass :-)

secondary repos used, eg.installer, signing, ...

Bear in mind also that for OpenJ9 "secondary" repos would likely include the eclipse/openj9 and eclipse/omr repos that get pullled in.

sxa picture sxa on 14 May 2021

@sxa Job "build Config" is:

{
    "ARCHITECTURE": "x64",
    "TARGET_OS": "linux",
    "VARIANT": "hotspot",
    "JAVA_TO_BUILD": "jdk",
    "TEST_LIST": [
        "sanity.openjdk",
        "sanity.system",
        "extended.system",
        "sanity.perf",
        "sanity.functional",
        "extended.functional"
    ],
    "SCM_REF": "",
    "BUILD_ARGS": "",
    "NODE_LABEL": "build&&linux&&x64",
    "ADDITIONAL_TEST_LABEL": "",
    "KEEP_TEST_REPORTDIR": false,
    "ACTIVE_NODE_TIMEOUT": "0",
    "CODEBUILD": false,
    "DOCKER_IMAGE": "adoptopenjdk/centos6_build_image",
    "DOCKER_NODE": "",
    "DOCKER_REGISTRY": "",
    "DOCKER_CREDENTIAL": "",
    "PLATFORM_CONFIG_LOCATION": "AdoptOpenJDK/openjdk-build/master/build-farm/platform-specific-configurations",
    "CONFIGURE_ARGS": "--enable-dtrace",
    "OVERRIDE_FILE_NAME_VERSION": "",
    "USE_ADOPT_SHELL_SCRIPTS": true,
    "RELEASE": false,
    "PUBLISH_NAME": "",
    "ADOPT_BUILD_NUMBER": "",
    "ENABLE_TESTS": true,
    "ENABLE_INSTALLERS": true,
    "ENABLE_SIGNER": true,
    "CLEAN_WORKSPACE": false,
    "CLEAN_WORKSPACE_AFTER": false,
    "CLEAN_WORKSPACE_BUILD_OUTPUT_ONLY_AFTER": true
}

BUILD_CONFIG:

00:39:19  # ============================
00:39:19  # OPENJDK BUILD CONFIGURATION:
00:39:19  # ============================
00:39:19  BUILD_CONFIG[ADOPT_PATCHES]="true"
00:39:19  BUILD_CONFIG[ASSEMBLE_EXPLODED_IMAGE]="false"
00:39:19  BUILD_CONFIG[BRANCH]="dev"
00:39:19  BUILD_CONFIG[BUILD_FULL_NAME]="linux-x86_64-server-release"
00:39:19  BUILD_CONFIG[BUILD_VARIANT]="hotspot"
00:39:19  BUILD_CONFIG[CLEAN_DOCKER_BUILD]="false"
00:39:19  BUILD_CONFIG[CLEAN_GIT_REPO]="true"
00:39:19  BUILD_CONFIG[CLEAN_LIBS]="false"
00:39:19  BUILD_CONFIG[CONTAINER_NAME]="openjdk_container"
00:39:19  BUILD_CONFIG[COPY_MACOSX_FREE_FONT_LIB_FOR_JDK_FLAG]="false"
00:39:19  BUILD_CONFIG[COPY_MACOSX_FREE_FONT_LIB_F......
andrew-m-leonard picture andrew-m-leonard on 14 May 2021

Prototype for Hotspot, so ignoring openj9 for the moment

andrew-m-leonard picture andrew-m-leonard on 14 May 2021
👍1

Complete build with strace, 56000+ trace files !?!
0.6Gbs of trace
and took 3+ hours to run!

andrew-m-leonard picture andrew-m-leonard on 14 May 2021
👍1

Complete build with strace, 56000+ trace files !?!

Does that include all the ENOENT trace points?

sxa picture sxa on 14 May 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

andrew-m-leonard picture andrew-m-leonard  ·  95Comments
M-Davies picture M-Davies  ·  97Comments
karianna picture karianna  ·  24Comments
DanielMartensson picture DanielMartensson  ·  25Comments
kuenking picture kuenking  ·  21Comments