Openjdk-infrastructure: Cache installed cygwin in order to speed up Windows provisions

[sxa]

I believe the cygwin installation is no more than splatting a load of files into C:\cygwin64. This is one of the slowest parts of the Windows playbooks. We should prototype storing the contents of that directory somewhere and extracting it when we perform a playbook deployment.

[sxa]

The change merged was backed out in https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1231 - reopening this issue

[Willsparker]

I've had another crack at trying what I did last time - wget the archive from ci.adoptopenjdk.net/userContent/ and untar to C:/cygwin64. While that works generally, (i.e. I can access the terminal mintty and do basic operations ), attempting to git clone https://github.com/adoptopenjdk/openjdk-infrastructure returns the same error as when I tried it before:

fatal: unable to access 'https://github.com/adoptopenjdk/openjdk-infrastructure/': SSL: couldn't create a context: error:0E079065:configuration file routines:def_load_bio:missing equal sign

Reinstallation of git via setup-x86_64.exe (The installer for Cygwin) unfortunately didn't fix this issue either.

[Willsparker]

Following https://stackoverflow.com/questions/11744472/how-to-replicate-a-cygwin-install-and-environment-on-another-machine unfortunately didn't give much luck either - same git issue. I'm going to get a fresh installation of Cygwin and repeat the things I did above, on the off-chance that it's the archive that is the issue.

[AdamBrousseau]

Does the machine have Git for Windows installed? Is it first on the path? If so, try removing it.

[Willsparker]

It doesn't have it installed - I'm using a clean machine at the minute as the Cygwin installation shouldn't need anything else :-)

[Willsparker]

Judging from the error message, It is an issue with OpenSSL, not git. Therefore, I tried to run a basic SSL command to see if the error message given was anymore helpful (following a guide here):

$ openssl s_client -connect poftut.com:443
34359738384:error:0E079065:configuration file routines:def_load_bio:missing equal sign:crypto/conf/conf_def.c:391:line 40
34359738384:error:140E6118:SSL routines:ssl_cipher_process_rulestr:invalid command:ssl/ssl_ciph.c:1030:
34359738384:error:140A90A1:SSL routines:SSL_CTX_new:library has no ciphers:ssl/ssl_lib.c:2967:

The first error to pop up is the same as the one that does with the git clone command, so looking at that first. Initially I thought it was an issue with the crypto/conf/conf_def.c, however looking at https://groups.google.com/forum/#!topic/mailing.openssl.dev/PLNOhqXOp94 , it seems it's an issue with the config file (at /etc/pki/tls/openssl.cnf) at line 40.

The contents of the config file at line 40 is:

38  .include = /etc/crypto-policies/back-ends/opensslcnf.config
39 
40  [ new_oids ]

This doesn't appear immediately useful however. I may try replacing the config file with a a default one, and see if if fixes the issue, or changes the error message to confirm it is an issue with that file.

[Willsparker]

Replacing the old config file with one found here fixed the missing equals error listed above. When rerunning the SSL command, the following was output:

34359738384:error:140E6118:SSL routines:ssl_cipher_process_rulestr:invalid command:ssl/ssl_ciph.c:1030:
34359738384:error:140A90A1:SSL routines:SSL_CTX_new:library has no ciphers:ssl/ssl_lib.c:2967:

For added information, the OpenSSL version in the cygwin.64.tar.xz archive is 1.1.1d-1 - running setup.exe shows it can be updated to 1.1.1f-1.

[Willsparker]

On the off-chance that these can be fixed with a reinstallation of openSSL, I gave that a go.
Reinstallation of openssl, libssl and libssl-devel to v1.1.1-d took ~19 minutes, as a lot of other dependent packages had to be reinstalled also (including cygwin.dll and the cygwin package).
Regardless, 2 issues listed above were still there when I attempted to git clone the infra-repo. Thankfully the 'missing equal sign' error didn't crop up again.
I'm beginning to wonder if the cygwin archive at ci.adoptopenjdk.net/userContent/winansible/cygwin64.tar.xz is broken itself. I'll try a fresh installation of Cygwin on a separate machine and try that tar.xz

[sxa]

If it's broken then let's get it updated :-)

[Willsparker]

In the process of creating a new .tar.xz at the minute. I checked the installation before and git clone https://github.com/adoptopenjdk/openjdk-build worked fine.
And for the sake of keeping a track of what I'm doing, I used the 7zip GUI (installed via the playbook), to first create a .tar file that had the C:\cygwin64 and C:\cygwin_packages folders , and then further compressed the .tar file to a .tar.xz file.

[Willsparker]

Creating and extracting a new cygwin64.tar.xz, I was able to run mintty.exe okay, however I had to add /cygdrive/c/cygwin64/cygwin64/bin to the path before I was able to run the commands cygwin installs.
Having tried git clone http://github.com/adoptopenjdk/openjdk-infrastructure , I was met with a new error message:

fatal: unable to access 'https://github.com/adoptopenjdk/openjdk-infrastructure/': error setting certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

Having run the following, I was able to properly git clone a repo!

export GIT_SSL_NO_VERIFY=true

However, this isn't really a good solution as it stops all SSL verification.
Running git clone with export GIT_CURL_VERBOSE=1 , didn't really tell me much more, just that the CApath isn't being set:

* error setting certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Marked for [closure]: Failed HTTPS connection
* multi_done
* The cache now contains 0 members
* Closing connection 0
* Expire cleared (transfer 0x80008e128)
fatal: unable to access 'https://github.com/adoptopenjdk/openjdk-build/': error setting certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

Weirdly, on a system that has Cygwin installed the usual way, it looks like the CAPath isn't be set either, however it doesn't mind:

04:33:37.907322 http.c:756              == Info: ALPN, offering h2
04:33:37.907381 http.c:756              == Info: ALPN, offering http/1.1
04:33:37.966520 http.c:756              == Info: successfully set certificate verify locations:
04:33:37.966590 http.c:756              == Info:   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
04:33:37.972724 http.c:756              == Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):

[Willsparker]

I've found the issue. When compressing a folder with 7-Zip, it doesn't retain symlinks ( https://unix.stackexchange.com/questions/600282/preserve-file-permissions-and-symlinks-in-archive-with-7-zip ), which is incredibly annoying as that was the software used to create both archive.
The particular symlinks that stop git from working are the following ones:

/etc/ssl/certs/ca-bundle.crt -> /etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Manually reinstating them allows git clone to work - however I don't feel very confident that a build would work due to other symlinks possibly being removed, so I'll create yet another archive and compress with a different software

[Willsparker]

@sxa had a good idea to use cygwin's tar to compress a new archive. I did this, and it did retain symlinks, however, it appears to have got confused in the process. Extracting the tar to a fresh machine still results in the error setting certificate verify locations when git clone ing.
So the following symlinks are present:

/etc/ssl/certs -> /cygdrive/c/etc/pki/tls/certs
/etc/pki/tls/certs/ca-bundle.crt -> /cygdrive/c/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

The issue is that the target symlinks have all be prefixed with /cygdrive/c/ - which actually refers to C:/ to windows, so C:/etc doesn't exist, and by proxy the other symlinks.
I think this may be due to compressing the folders in cmd as opposed to mintty.exe.

[sxa]

Do they show /cygdrive/c as the symlink targets if you run tar tvf to look inside the tarball?

[Willsparker]

Oh, they dont

tar tvf cygwin64.tar.xz  | grep /etc/ssl/certs
lrwxrwxrwx  0 Administrator None                0  3 Aug 09:48 cygwin64/etc/ssl/certs -> /etc/pki/tls/certs

[Willsparker]

Looking at that symlink from the Windows file explorer on a machine that had that archive extracted, the symlink's target is C:\etc\pki\tls\certs - Maybe Windows doesn't like the file path /etc/pki/tls/certs/, so it just throws a C:\ on at the beginning, at the time of extracting the file. I'll have a go at extracting it with a version of minnty.exe and tar.exe that comes bundled with cygwin.

[Willsparker]

The Cygwin cache was made by installing Cygwin using the playbook as it was before- the version is cygwin 3.16, and has the packages installed from the devel category at the newest version, except make, which has version 4.2.1-2 and grep which has version 2.27-2.
These version differences were due to https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1231#issue-394240789 and https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/384 respectively.
The archive itself was made using a separate version of cygwin's tar. Window's 7-Zip wouldn't work as it didn't retain symlinks when creating archives.
In addition, cygwin's pigz (and tar) must be used to extract the archive too, otherwise the paths to symlinks can have C:\ added to the front of them, which causes them to break.

[aahlenst]

If I understand it correctly, with #1503 the old playbook vanishes in the Git history. So it might make sense to either create a separate playbook or script that makes it possible to re-create or update the Cygwin archive in the future.

[Willsparker]

Ah I see - I'll have a look into it :-)

[sxa]

The other option would just be to put a few lines of batch script in a comment at the top which shows how to replicate what the playbook did originally:

• Download https://cygwin.com/setup-x86_64.exe, https://mirrors.kernel.org/sourceware/cygwin/x86_64/release/grep/grep-2.27-2.tar.xz and https://mirrors.kernel.org/sourceware/cygwin/x86_64/release/make/make-4.2.1-2.tar.xz
• Run cygwin.exe --quiet-mode --download --local-install --delete-orphans --site https://mirrors.kernel.org/sourceware/cygwin/ --local-package-dir c:\cygwin_packages --root c:\cygwin64 --categories Devel
• Run for %%P in (autoconf cpio libguile2.0_22 unzip zip curl curl-debuginfo libcurl-devel libpng15 libpng-devel perl-text-CSV) do setup-x86-64 -q -P %%P
• Run cygwin bash and extract grep and make tar -C / -xpJf grep.tar.xz; tar -C / -xpzf grep-2.27-2.tar.gz

Would be nice if those two overlays were no longer required at some point though - if we have an issue for why we added those two it's worth linking to that in the playbook regardless. But I'm happy with the above or keeping the original role around (We could just add another bit to such a comment block with a link to the original playbook in the history so it's easy to recover if needed)

Side note, can cygwin -q -p take multiple packages as arguments?

[gdams]

@sxa yes you can install multiple packages in one command. I've been using this 2 liner in the dockerfiles:

Invoke-WebRequest -UseBasicParsing 'https://cygwin.com/setup-x86_64.exe' -OutFile 'C:\temp\cygwin.exe';
Start-Process -Wait -FilePath 'C:\temp\cygwin.exe' -ArgumentList '--packages wget,bsdtar,rsync,gnupg,git,autoconf,make,gcc-core,mingw64-x86_64-gcc-core,unzip,zip,cpio,curl,grep,perl --quiet-mode --download --local-install --delete-orphans --site https://mirrors.kernel.org/sourceware/cygwin/ --local-package-dir C:\cygwin_packages --root C:\cygwin64';