Openjdk-infrastructure: System unavailable: build-alibaba-win2012r2-x64-[12]

[sxa]

This will prevent alibaba windows builds working as they are currently tied to these machines.

[Willsparker]

-1 is back as I've come to it, I'll look at -2 :-)

[Willsparker]

Rather interestingly, neither of the machines have the Jenkins Agent installed as a service. They appeared to be running the agent in a cygwin terminal window. I'll install it on both

[Willsparker]

I can't install them on both, due to the lack of IcedTea-Web. I can install it, but it seems that the machines have a stripped down version of the playbook running on them, and I'm unsure of the reason for that. (asked about it here: https://adoptopenjdk.slack.com/archives/C53GHCXL4/p1610356948467700 )
In the meanwhile, I'll get the Jenkins agent running in a Cygwin terminal again, so they're at least usable.

[Willsparker]

@Haroon-Khel said he's installing the missing packages on the machines (ref: https://adoptopenjdk.slack.com/archives/C53GHCXL4/p1610356948467700 ).

[Haroon-Khel]

Missing packages have been installed on both alibaba machines, except for OpenSSL packages. Both experienced the error

TASK [Install OpenSSL-1.1.1i 64-bit (VS2013)] ******************************************************************************************************************************************
task path: /Users/hkhel/AdoptOpenJDK/openjdk-infrastructure/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/OpenSSL/tasks/main.yml:73
fatal: [8.208.87.18]: FAILED! => {"changed": true, "cmd": "set PATH=C:\\Strawberry\\perl\\bin;C:\\openjdk\\nasm-2.13.03;%PATH% && .\\vcvarsall.bat AMD64 && cd C:\\temp\\OpenSSL-1.1.1i && perl C:\\temp\\OpenSSL-1.1.1i\\Configure VC-WIN64A --prefix=C:\\openjdk\\OpenSSL-1.1.1i-x86_64-VS2013 && nmake install > C:\\temp\\openssl64-VS2013.log &&
 nmake -f makefile clean", "delta": "0:00:03.448210", "end": "2021-01-11 12:34:31.186987", "msg": "non-zero return code", "rc": 1, "start": "2021-01-11 12:34:27.738777", "stderr": "'nmake' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n", "stderr_lines": ["'nmake' is not recognized as an internal or external command,", "operable program or batch file."], "stdout": "The specified configuration type is missing.  The tools for the\r\nconfiguration might not be installed.\r\nConfiguring OpenSSL version 1.1.1i (0x1010109fL) for VC-WIN64A\r\nUsing os-specific seed configuration\r\nCreating configdata.pm\r\n

Looking into it

[sxa]

We're having some issuejs on these machines after (a) running the rest of the playbooks and (b) Switching the jenkins agent to run as the jenkins user. While most of them have now been resolved I'm still getting the following issue (even after a reboot) on -1 which I haven't yet been able to fully diagnose ... Still working on it but any crazy ideas welcome :-)

17:00:23  Running gradle with /cygdrive/c/openjdk/jdk-11 at /cygdrive/c/workspace/openjdk-build/workspace/.gradle
17:00:23  Exception in thread "main" java.io.FileNotFoundException: \cygdrive\c\workspace\openjdk-build\workspace\.gradle\wrapper\dists\gradle-6.5-bin\6nifqtx7604sqp1q6g8wikw7p\gradle-6.5-bin.zip.lck (Access is denied)

[Haroon-Khel]

OpenSSL 64 bit VS2013 also isnt installed on either -1 or -2 due to vcvarsall.bat not being available in in the VS2013 folders. Reinstalling VS2013 didnt seem to solve this

[Willsparker]

@sxa Have you tried running it with a different JDK (or reinstalled JDK11) ? Presuming you've already looked at all the permissions of the folders and everything.

[Haroon-Khel]

Latest failure https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-dragonwell/38/console
Still the same error, but running the same build command on a cygwin shell, as the jenkins user, on build-alibaba-win2012r2-x64-1 in an rdp session doesnt seem to hit this error

[Haroon-Khel]

I changed the variable CYGWIN_WORKSPACE to C:\Users\Jenkins\workspace (it was C:\Jenkins\workspace before). This may have done the trick https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-hotspot/884/console (the hotspot builds were failing for the same reason too)

[Haroon-Khel]

https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-dragonwell/40/console
A dragonwell build on alibaba -1 passed, but failed at the installer stage. I think the variable change helped to circumvent the gradle error

[Haroon-Khel]

Running the dragonwell jdk8 job on alibaba-1, https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-dragonwell/46/console, jenkins seems to have a problem with clearing the C:\Users\Jenkins\workspace workspace

[Haroon-Khel]

Re ran the jdk11 dragonwell job on alibaba-1, same error https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-dragonwell/41/console. Oddly this wasnt a problem yesterday when I ran both jdk11 hotspot and dragonwell jobs on the same machine one after the other

[Haroon-Khel]

I deleted the C:\Users\Jenkins\workspace directory. I re ran the jdk11 hotspot and dragonwell and jdk 8 dragonwell jobs one after the other. Jenkins didnt seem to complain about not being able to delete workspaces. The CYGWIN_WORKSPACE variable is still C:\Users\Jenkins\workspace for alibaba-1

[Haroon-Khel]

Regarding the 2013 compiler on alibaba-2, jdk 8 hotspot can build fine. jdk 8 dragonwell exits with this error

Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
27 errors
make[2]: *** [CompileJavaClasses.gmk:336: /cygdrive/c/cygwin/home/jenkins/openjdk-build/workspace/build/src/build/windows-x86_64-normal-server-release/jdk/classes/_the.BUILD_JDK_batch] Error 1
make[1]: *** [BuildJdk.gmk:64: classes-only] Error 2
make: *** [/home/jenkins/openjdk-build/workspace/build/src//make/Main.gmk:117: jdk-only] Error 2

[sxa]

Looking at https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-dragonwell/53/consoleFull I think that might be the same error occurring on one of the other build machines, so it could well be a problem in the codebase at the moment as opposed to a problem with that machine, so at least for now I wouldn't worry too much about that error.

[Haroon-Khel]

Just an update:
It was identified that the alibaba machines are having the same problem as https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1662, in which the leftover _the.. file prevents jenkins from deleting the workspace before running its job. This has affected other windows boxes, hence the pr https://github.com/AdoptOpenJDK/openjdk-build/pull/2204, so I have put in a similar pr https://github.com/AdoptOpenJDK/openjdk-build/pull/2400.

Related issue https://github.com/AdoptOpenJDK/openjdk-build/issues/2205

[Haroon-Khel]

I have also changed the CYGWIN_WORKSPACE variable on both alibaba machines to C:\Jenkins\temp since C:\Jenkins\workspace results in the gradle error

17:00:23  Running gradle with /cygdrive/c/openjdk/jdk-11 at /cygdrive/c/workspace/openjdk-build/workspace/.gradle
17:00:23  Exception in thread "main" java.io.FileNotFoundException: \cygdrive\c\workspace\openjdk-build\workspace\.gradle\wrapper\dists\gradle-6.5-bin\6nifqtx7604sqp1q6g8wikw7p\gradle-6.5-bin.zip.lck (Access is denied)

[sxa]

Gut feel at this point is that it's a party length issue so I suspect any directory with 9 characters like "workspace" would have the issue. We could switch to C:\workspace which works be more consistent with what we have on the other machines

[Willsparker]

If you change the workspace variable to C:\workspace , I _think_ the build PR will be unnecessary, as it should be covered by https://github.com/AdoptOpenJDK/openjdk-build/blob/102237341c7f0737f0dd4dc57fcc7e9e3ffe3bd5/pipelines/build/common/openjdk_build_pipeline.groovy#L915 .

[Haroon-Khel]

To reiterate my comments in the build pr, changing the workspace to C:\workspace\openjdk-build (the rm command removes workspaces in C:\workspace\openjdk-build not C:\workspace\) caused the gradle error to appear again. It's possible that your gut feeling is right @sxa, since C:\Jenkins\temp as a workspace seemed to work fine (I assume its fine if the total path exceeds 9 characters, so long as each directory doesnt?)

[sxa]

Yep it's the total length that'll make a difference, each individual component shouldn't matter too much (there may be a limit but I haven't hit it before in any normal scenario)

[sxa]

@Haroon-Khel If you set the workspace to just C:\workspace does it work ok? I think openjdk-build will get created as a subdirectory of the workspace dir so setting th workspace to c:\workspace\openjdk-build definitely wouldn't have the deisred effect of taking advantage of the line suggested by Will

[Haroon-Khel]

Setting it to C:\workspace didnt work. It hit the same 'could not delete workspace error', (https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-dragonwell/79/console). I even tried C:/workspace and it hit the same error.

Also, I dont think the openjdk-build directory is created on its own as a subdirectory (but I may be wrong); by setting the workspace to C:\workspace, the subdirectories that are created are C:\workspace\workspace\build ... which therefore wont be deleted by the existing rm command, which deletes rm -rf C:/workspace/openjdk-build/workspace/build/src/build/*/jdk/gensrc. Its only if I set the workspace to be C:\workspace\openjdk-build\ do I get C:\workspace\openjdk-build\workspace\build ... (unless I am terribly mistaken).

(EDIT: I might be slightly mistaken, but I am certain the openjdk-build subdirectory does not get created automatically since I cant find it in any of the workspaces used aside from when I used C:\workspaceopenjdk-build as the workspace)

Another note, I cant find a single windows machine in jenkins with the CYGWIN_WORKSPACE variable to be C:\workspace, so I am not sure why https://github.com/AdoptOpenJDK/openjdk-build/blob/102237341c7f0737f0dd4dc57fcc7e9e3ffe3bd5/pipelines/build/common/openjdk_build_pipeline.groovy#L915 was put there in the first place, unless those machines have since been decommissioned

[Haroon-Khel]

https://github.com/AdoptOpenJDK/openjdk-build/issues/1855#issue-636830242 Ahh, Softlayer machines which we dont have anymore

[sxa]

AdoptOpenJDK/openjdk-build#1855 (comment) Ahh, Softlayer machines which we dont have anymore

They were replaced with the -ibmcloud- ones.

[sxa]

Setting it to C:\workspace didnt work. It hit the same 'could not delete workspace error', (https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-dragonwell/79/console). I even tried C:/workspace and it hit the same error.

We need to understand why - looks like you've changed it back to be the openjdk-buildsubdirectory. I'm going to reset it and try again.

[sxa]

(EDIT: I might be slightly mistaken, but I am certain the openjdk-build subdirectory does not get created automatically since I cant find it in any of the workspaces used aside from when I used C:\workspaceopenjdk-build as the workspace)

Worth remembering that setting it to something that long causes other problems, so we know that won't work.

Unclear why the default directory used for deletion included the openjdk-build if that's the case though. Suspect we may want to adjust the defult so it excludes the openjdk-build but it might be safer for now to just add a new delete as you've done in the PR.

Just running a test just now and I've queued up another one using c:\Jenkins\temp as per your PR and make it use your branch with the extra rm. You have access to run that job if needed

[sxa]

The last run mentioned in the previous comment looks ok, as does a subsequent one on the machine from the same c:/Jenkins/temp directory using your PR.

[Haroon-Khel]

The build-ibmcloud machines use E:/jenkins/tmp as their CYGWIN_WORKSPACE (hence the need for https://github.com/AdoptOpenJDK/openjdk-build/blob/102237341c7f0737f0dd4dc57fcc7e9e3ffe3bd5/pipelines/build/common/openjdk_build_pipeline.groovy#L919) so I dont think that rm command is of any use anymore

[sxa]

Slightly confused by the last comment - you don't think which rm command is of use?

[Haroon-Khel]

Sorry, the command which deletes the C:\workspace\openjdk-build\ ... directory, since I think this was specific to the softlayer machines

[sxa]

Yes agreed your new line could probably replace that one

[Haroon-Khel]

Can this issue be closed now? Or do we want to use this issue to discuss the mysterious _the.. file?

[sxa]

Can this issue be closed now? Or do we want to use this issue to discuss the mysterious _the.. file?

We haven't yet enabled all the tags to build etc. overnight so it is still "unavailable". I've ran a couple of test jobs though (https://ci.adoptopenjdk.net/view/Test_system/job/Test_openjdk15_j9_sanity.system_x86-64_windows/ 152 and 153 - one on each machine) and that looks ok, so I think I'll re-enable all the tags and let it run tonight on the JDK16 builds and see what happens. I'm also running https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-hotspot/907 on alibaba-1 to test VS2013 but it looks like it doesn't have a valid JDK7 boot dir configured

[Haroon-Khel]

but it looks like it doesn't have a valid JDK7 boot dir configured

Im not sure why this is, just yesterday I ran a jdk8 dragonwell job which made it past that stage on alibaba-1 https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-dragonwell/78/consoleFull

[sxa]

Dragonwell 8 does not use a JDK7 boot JDK. Hotspot does. We need this machine to be able to build all variants including HotSpot

[Haroon-Khel]

Ok, that would explain it

[Haroon-Khel]

Ive changed the JDK7_BOOT_DIR on alibaba-1 to /cygdrive/c/openjdk/jdk-7 (the - was missing). Done the same on alibaba-2

[sxa]

There was another problem showing up whereby the builds would fail if the jenkins agent was running as a service relative to starting it from a cygwin shell. The difference is that the default PATH on the system had the Windows GIT client first, whereas the cygwin shell had it's own one first. Adjusting the system PATH to have C:\cygwin\bin first resolved that problem, therefore the machines now have it running correctly as a service.

[sxa]

• jdk8u/hs/windows built ok on build-alibaba-win2012r2-x64-1
• I've also queued up a JDK11/J9XL build and a JDK17/HS so if they both work I think we're good to enable this machine for building again.

[sxa]

@Haroon-Khel Ref the first of the job links above, it looks like alibaba-1 doesn't have cmake on it - I thought you'd run most of the playbooks on it - is that not the case? https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9-windowsXL/635/console

[sxa]

JDK17/HS didn't run properly either :-(

23:22:58  Building targets 'product-images legacy-jre-image test-image' in configuration 'windows-x86_64-server-release'
23:22:58  Compiling 8 files for BUILD_TOOLS_LANGTOOLS
23:22:58  error: file not found: \cygdrive\c\jenkins\temp\workspace\build\src\build\windows-x86_64-server-release\buildtools\langtools_tools_classes\_the.BUILD_TOOLS_LANGTOOLS_batch.filelist
23:22:58  make[3]: *** [ToolsLangtools.gmk:37: /cygdrive/c/jenkins/temp/workspace/build/src/build/windows-x86_64-server-release/buildtools/langtools_tools_classes/_the.BUILD_TOOLS_LANGTOOLS_batch] Error 3
23:22:58  make[2]: *** [make/Main.gmk:74: buildtools-langtools] Error 2
23:22:58  make[2]: *** Waiting for unfinished jobs....
23:23:01  

[Haroon-Khel]

@Haroon-Khel Ref the first of the job links above, it looks like alibaba-1 doesn't have cmake on it - I thought you'd run most of the playbooks on it - is that not the case? https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9-windowsXL/635/console

Im certain I did. Will look into it

[Haroon-Khel]

I ran the cmake role on -1. Though ansible said it installed cmake, I couldnt find it on the machine, nor did it update the path. So I manually installed cmake in Program Files\CMake and added it to the path. I was unable to install it manually in cygwin/bin due to the installer not having sufficient privileges (eventhough I was running it as the Administrator user)

It should be noted that ansible checks for an already installed cmake in cygwin64\bin, while these machines have only a cygwin\bin directory. Would it suffice simply to rename the directory to cygwin64? Or must cygwin be reinstalled completely for this?

[Haroon-Khel]

Rerunning a jdk11 openj9 job on -1. If it succeeds, ill install cmake this way onto -2
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/907/console

[Haroon-Khel]

The job made it passed the cmake check, but failed with this

12:25:48  Compiling 13 properties into resource bundles for jdk.javadoc
12:25:49  /usr/bin/bash: /cygdrive/c/Program: No such file or directory
12:25:49  make[3]: *** [/cygdrive/c/Jenkins/temp/workspace/build/src/closed/OpenJ9.gmk:414: /cygdrive/c/Jenkins/temp/workspace/build/src/build/windows-x86_64-normal-server-release/vm/cmake.stamp] Error 127
12:25:49  make[2]: *** [/cygdrive/c/Jenkins/temp/workspace/build/src/closed/custom/Main.gmk:51: j9vm-build] Error 2
12:25:49  make[2]: *** Waiting for unfinished jobs....
12:25:49  Compiling 19 properties into resource bundles for jdk.compiler
12:25:49  Compiling 12 properties into resource bundles for jdk.jdeps
12:25:57  
12:25:57  ERROR: Build failed for targets 'product-images legacy-jre-image test-image debug-image' in configuration 'windows-x86_64-normal-server-release' (exit code 2) 
12:25:57  
12:25:57  No indication of failed target found.
12:25:57  Hint: Try searching the build log for '] Error'.
12:25:57  Hint: See doc/building.html#troubleshooting for assistance.
12:25:57  
12:25:57  make[1]: *** [/cygdrive/c/Jenkins/temp/workspace/build/src/make/Init.gmk:305: main] Error 2
12:25:57  make: *** [/cygdrive/c/Jenkins/temp/workspace/build/src/make/Init.gmk:186: product-images] Error 2

[Haroon-Khel]

The 12:25:49 /usr/bin/bash: /cygdrive/c/Program: No such file or directory suggests that its having trouble with the spacing in windows directory names

[Willsparker]

Program Files (or the x86 equivalent) doesn't have a Short Name, would be my guess. See #1250 #1598 #1672 (and their referenced issues)

[Haroon-Khel]

02/02/2021  08:07 PM    <DIR>          PROGRA~1     Program Files
01/14/2021  06:44 PM    <DIR>          PROGRA~2     Program Files (x86)

They do seem to have Shortnames enabled

[Haroon-Khel]

Ive created a symlink in cygwin/bin/ which links to Program Files\CMake\bin. Re running at https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/911/console

[Haroon-Khel]

I think it may have found cmake, but is hitting a different error now

13:45:33  CMake Error: Error processing file: /cygdrive/c/Jenkins/temp/workspace/build/src/openj9/runtime/cmake/caches/win_x86-64_cmprssptrs.cmake
13:45:33  CMake Error: The source directory "/cygdrive/c/Jenkins/temp/workspace/build/src/openj9" does not exist.
13:45:33  Specify --help for usage, or press the help button on the CMake GUI.
13:45:33  loading initial cache file /cygdrive/c/Jenkins/temp/workspace/build/src/openj9/runtime/cmake/caches/win_x86-64_cmprssptrs.cmake
13:45:33  make[3]: *** [/cygdrive/c/Jenkins/temp/workspace/build/src/closed/OpenJ9.gmk:414: 

[sxa]

The 12:25:49 /usr/bin/bash: /cygdrive/c/Program: No such file or directory suggests that its having trouble with the spacing in windows directory names

The 12:25:49 /usr/bin/bash: /cygdrive/c/Program: No such file or directory suggests that its having trouble with the spacing in windows directory names

The question is: What's different between how it's being picked up on the alibaba machines vs the Azure/IBMCloud ones?

[Haroon-Khel]

On build-azure-win2012r2-x64-1, cmake is in cygwin64/bin/cmake.exe. But I think the symlink I created managed to solve the spaced directory problem. We're now getting a Cmake error, which I have posted above.
The azure machine machine's cmake is version 3.14.5, while the alibaba's is 3.7.2 the version that ansible installs

[sxa]

On build-azure-win2012r2-x64-1, cmake is in cygwin64/bin/cmake.exe. But I think the symlink I created managed to solve the spaced directory problem. We're now getting a Cmake error, which I have posted above

So where did the cmake.exe one come from oin the machines that work - is it a cygwin one as opposed to one from a separate cmake installation?

[sxa]

OK This is rather odd ... The cmake role seems to check for its presence in c:\cygwin64\bin\cmake.exe and then use that to decide whether to install the one from cmake.org which won't update C:\cygwin64\bin\cmake.exe. What a mess...

@AdamBrousseau Do the OpenJ9 Windows machines use cmake from cygwin or from cmake.org? If the former (and assuming Haroon confirms how the other machines are set up - cmake --version will hopefully help figure it out) then we should possibly standardise on that version and remove the cmake role.

[Haroon-Khel]

So where did the cmake.exe one come from oin the machines that work - is it a cygwin one as opposed to one from a separate cmake installation?

Im not too sure, but I would assume that it came with cygwin since its in the cygwin directory. When I manually had to use the installer to install cmake on alibaba, its default installation directory was Program Files\CMake, therefore I assume the way our cmake role works is that it checks in cygwin64\bin\ for a prepackaged cmake, else it installs it separately in Program Files

[Haroon-Khel]

Yeah until now I assumed that the cmake role was responsible for installing cmake in cygwin64\bincmake. I guess not

[Haroon-Khel]

Im going to install cmake on alibaba-2 and see if I get the same error

[sxa]

I assume the way our cmake role works is that it checks in cygwin64\bin\ for a prepackaged cmake, else it installs it separately in Program Files

While that looks like it's what it's doing it doesn't seem particuarly sensible since it'll be repeatedly trying to install the seperate one in the case where there isn't a copy in c:\cygwin64\bin - it shouldn't be trying to reinstall the separate cmake if it's already there under C:\Program Files

[Haroon-Khel]

I just ran the cmake role on -2. It didnt install it in either Program Files or cygwin\bin, depsite it saying changed. Thats confusing

TASK [Download cmake installer] *********************************************************************************************************************************************************************************************************************************************************
task path: /Users/hkhel/AdoptOpenJDK/openjdk-infrastructure/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/cmake/tasks/main.yml:12
changed: [8.208.87.18] => {"changed": true, "checksum_dest": "8b0cbfc6be83e31a058c8ef282fe204862809ffcd8788bc19a8f0eb457f71187", "checksum_src": "8b0cbfc6be83e31a058c8ef282fe204862809ffcd8788bc19a8f0eb457f71187", "dest": "C:\\temp\\cmake.msi", "elapsed": 3.1962816, "msg": "OK", "size": 18235900, "status_code": 200, "url": "https://cmake.org/files/v3.7/cmake-3.7.2-win64-x64.msi"}

TASK [Install cmake] ********************************************************************************************************************************************************************************************************************************************************************
task path: /Users/hkhel/AdoptOpenJDK/openjdk-infrastructure/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/cmake/tasks/main.yml:22
changed: [8.208.87.18] => {"changed": true, "rc": 0, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

[sxa]

You'll need to go onto the machine and try and install it manually using the command in the playbook and see what happens and/or search the whole machine for cmake.exe to see if the playbook has put it anywhere, but the point I was making before is whether it's even used at all on the other machines or if they are actually using cmake from cygwin for the OpenJ9 builds.

[Haroon-Khel]

Both build-azure machines use 3.14.5, while both build-ibmcloud machines use 3.17.3. Both use cygwin's cmake

[Haroon-Khel]

Before your comment, I installed cmake on -2 via the msi. The same error appeared on -2
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/914/console

[sxa]

Before your comment, I installed cmake on -2 via the msi. The same error appeared on -2
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/914/console

Is that different from what happened on the machine before you installed cmake from the MSI? If every other machine is using the cmake from cygwin I'm not sure we have a requirement for the other one

[Haroon-Khel]

Is that different from what happened on the machine before you installed cmake from the MSI?

Yes. Before installing it, there wasnt a cmake on the machine so would give a cmake not found error. I installed it via the msi on -2 just to see if I could recreate the error. It can always be uninstalled.

In terms of next steps, the only thing I can think of is to reinstall cygwin using the playbooks to get the cygwin cmake

[sxa]

Yes. Before installing it, there wasnt a cmake on the machine so would give a cmake not found error. I installed it via the msi on -2 just to see if I could recreate the error. It can always be uninstalled.

Hmmm - has it also added itself to the system PATH? We don't add that directory during the build scripts

[sxa]

In terms of next steps, the only thing I can think of is to reinstall cygwin using the playbooks to get the cygwin cmake

I don't think it needs a reinstall - from memory you can add packages by repliacting the command-line parameters to the cygwin installer like https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/81d61d27006e6832c063905c80421a6ca3cd0db9/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/cygwin/tasks/main.yml#L20

Or worst case you just re-run the installer and select the new packages.

[Haroon-Khel]

Using the installer, there isnt an option to install cmake as a package

[Haroon-Khel]

Even the command line arguments you posted doesnt include cmake. I can only assume it comes with the cygwin package from the checklist I posted?

[sxa]

Using the installer, there isnt an option to install cmake as a package

Hmmm that's a bit odd. Since we have it on the others machines it must havecome from somewhere

Even the command line arguments you posted doesnt include cmake. I can only assume it comes with the cygwin package from the checklist I posted?

It's possible it disappeared in the changes made to speed up the cygwin installs a few months ago in which case it needs to be added back in. Although that shoulkd have been picked up by VagrantPlaybookCheck (NOTE: It's just about possible we haven't run a JDK11/J9 build on VPC since it was added I suppose)

[Haroon-Khel]

It's possible it disappeared in the changes made to speed up the cygwin installs a few months ago in which case it needs to be added back in. Although that shoulkd have been picked up by VagrantPlaybookCheck (NOTE: It's just about possible we haven't run a JDK11/J9 build on VPC since it was added I suppose)

I checked for changes in the cygwin role, I dont think that cmake was ever in the arguments list

On alibaba-1, I tried uninstalling cygwin by removing the cygwin directory. This partially worked; some files were prevented from being deleted due to a permission error eventhough I tried to delete the directory as the admin user. Nonetheless, I ran the playbook's cygwin role onto the machine which ran fine. Cygwin installed itself in the C:cygwin64 directory, independent of the existing cygwin directory. The only problem is that this didnt come with a cmake install, so I do not know from where cmake is installed on the azure or ibmcloud machines

[Haroon-Khel]

Like before, Ive created a symlink for the Program Files\CMake in the cygwin64\bin directory so jenkins can use it, but like before it will likely result in a CMake error

[Haroon-Khel]

Update: https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1958 allows for cmake to get installed alongside cygwin. Cmake is now on both alibaba machines.
openj9 jdk11 job passed on -2 https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/929/console

The same job is now running on -1
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/933/console

Openssl has been updated from i to j on both machines too

[sxa]

Build seems to have worked ok :-)

[sxa]

I've added the build tag back onto -1 so it should get used for tonight's builds

[sxa]

@Haroon-Khel Is there any outstanding work to be done here?

[Haroon-Khel]

This issue can be closed. The cmake issue was the last missing thing for these machines. Thats now been resolved