Temurin-build: openjdk10 builds on CentOS6 are failing

[sxa]

It looks like it's now using the default gcc 4.4.7 instead of the gcc 4.8.2 in /opt/devtoolset-2 Somehow it was picking up the 4.8.2 on the CentOS boxes up until the 16th

Last good build was https://ci.adoptopenjdk.net/job/openjdk10_build_x86-64_linux/119/console and contained this, suggesting there might have been a /usr/bin/gcc -> /opt/devtoolset-2/root/usr/bin/gcc symbolic link in place (/opt/devtoolset-2 is where the official 4.8 gcc goes on RHEL/CentOS6)

configure: Using default toolchain gcc (GNU Compiler Collection)
checking for gcc... /usr/bin/gcc
checking resolved symbolic links for CC... /data/opt/rh/devtoolset-2/root/usr/bin/gcc
configure: Using gcc C compiler version 4.8.2 [gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)]

The /usr/bin/gcc on build-digitalocean-centos69-x64-1 was put in place on March 13s at 18:49 local time and the corresponding package gcc-4.4.7-18.el6_9.2.x86_64 was installed on March 15th which would be consistent with when it started breaking. Same on Friday 16th on build-joyent-centos69-64-1 (which would have been due to the yum update cron that we have to keep the systems patched based on the install timestamps)

The ansible playbook had previously put a symlink in place to make 4.8.4 the default:

- name: Create symlink for /opt/rh/devtoolset-2/root/usr/bin/gcc to gcc
  file:
    src: /opt/rh/devtoolset-2/root/usr/bin/gcc
    dest: /usr/bin/gcc
    force: true
    owner: root
    group: root
    state: link
  when:
    - ansible_distribution == "CentOS"
    - ansible_distribution_major_version == "6"
  tags: build_tools

I think my preferred solution is not to put the symlink in place, and set CC/CXX variables somewhere to force it to 4.8. Alternatively we could put the devtoolset-2 bin directory in the path before /usr/bin. Thoughts?

[bblondin]

Interesting...
I can confirm that yes yum did update gcc on March 16th

yum.log
Mar 16 01:01:01 Updated: gcc-4.4.7-18.el6_9.2.x86_64
Mar 16 01:01:03 Updated: gcc-c++-4.4.7-18.el6_9.2.x86_64

And in the process it likely overwritten the symlink.

On other OS' we leave the default OS gcc alone and set a symlink to point at /usr/bin/gcc-4.8 not sure why we are not doing the same on CentOS.

@gdams do you have any insight into this?

Related to pull: https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/257

[sxa]

gcc 4.8 is not in the default package repository for CentOS/RHEL6 and it's supplied as part of the optional developer toolset package, hence the rather odd path and why it's a bit diferent from other OSs. (I've taken it from https://linux.web.cern.ch/linux/devtoolset/ in the past for other machines)

The playbook is doing the same on CentOS, and it worked for a while, just got overridden by the executable when the security patching was done. I don't think there's any easy way of stopping that, short of completely removing the gcc 4.4.7 package from the machine which would stop it updating the symlink we're putting in place.

[sxa]

I've added this directly into the jenkins xlinux job for now as a stop-gap to demonstrate that this works as a solution. I'm happy to do this across all the architectures, although is this the right way to do it? Should I add /opt/rh/devtoolset-2/root/usr/bin to the start of the PATH instead?

[ -r /opt/rh/devtoolset-2/root/usr/bin/gcc ] && export CC=/opt/rh/devtoolset-2/root/usr/bin/gcc
[ -r /opt/rh/devtoolset-2/root/usr/bin/g++ ] && export CXX=/opt/rh/devtoolset-2/root/usr/bin/g++

Job reference: https://ci.adoptopenjdk.net/job/openjdk10_build_x86-64_linux (#126 includes the above fix)

[bblondin]

Edit /etc/yum.conf and add in exclude=gcc*

https://access.redhat.com/solutions/10185

[sxa]

That would work too, if we're prepared to accept the lack of bug fixes to the 4.4.7 compiler. I suppose the question really is whether we even need gcc 4.4.7 for anything. GIven that we've been building on 4.8 on Ubuntu 16.04 the preferred option may well be just to remove the package completely and let the symlink survive in safety :-)

[sxa]

We are currently building JDK8+HotSpot with gcc 4.4.7 successfully so I suspect it would be sensible to retain it for building JCK8 as a "lowest common denominator", JDK8+Eclipse OpenJ9 requires 4.8 though. Rather disappointingly it seems the devtoolset-2 packages doesn't create a gcc-4.8 in any location, so perhaps we should also create that symlink in /usr/bin in the playbook.

I think my personal preference would be to create a /usr/bin/gcc-4.8 playbook and have the jobs explicitly point at gcc-4.8 which feels like the safest option, but I'd only claim about 70% preference for that solution and would be happy with something else that works.

[bblondin]

Yes I agree with that. This is also what we have been doing for RHEL.

https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/gcc_48/tasks/main.yml

[sxa]

Agreement reached on the call - we will leave the OS doing what it wants to (i.e. /usr/bin/gcc will be the default 4.4.7 on the CentOS6 boxes) and have the playbook create /usr/bin/gcc-4.8 and /usr/bin/g++-4.8 pointing to the appropriate localtions under /opt/rh/devtoolset-2

[karianna]

Effectively resolved and we're no longer concerned with 10.