Openjdk-infrastructure: Allow build-ibmcloud-win2012r2-x64-1 to run installer (WiX) jobs

[sxa]

Please put the name of the software product (and affected platforms if relevant) in the title of this issue

• [x] Missing install
• [ ] Bug in ansible playbook
• [ ] Request for new playbook addition

Details: At the moment the create_installer_windows jobs cannot be run on anything other than the build-azure- machines. Loss of them both this morning caused the jobs to back up. Ideally we want to enable at least one at an alternative provider

[Willsparker]

Apparently it already has Wix stuff installed - I'm not sure of what else would need to be installed for the installer jobs, or if there's any other config?

[Willsparker]

From @gdams:

it needs the codesign cert and the 8dot3 shortname for the wix dir

I double checked and that build machine has it all, so I think it should already be good for it. I've added the Wix tag to the machine, to see how it fairs :-)
I would presume that -2 also has all the dependencies for the Wix tag, so we could add it to that as well, if you like? @sxa

[Willsparker]

Unfortunately some other setup is required:

Generate OpenJDK setup "hotspot" for "x64" platform "jdk"
****************************************************
looking for .\SourceDir\OpenJDK16\hotspot\x64\jdk-16.0.0
looking for .\SourceDir\OpenJDK16\hotspot\x64\jdk16u0-b36
looking for .\SourceDir\OpenJDK16\hotspot\x64\jdk-16+36
Source dir used : .\SourceDir\OpenJDK16\hotspot\x64\jdk-16+36
PRODUCT_ID: {C327398B-AF80-482A-89ED-23E48A043B27}
SOURCE_TEXT_GUID (without displaying secret UPGRADE_CODE_SEED) : jdk-16-x64-hotspot
Constant PRODUCT_UPGRADE_CODE: {588F312B-DC7A-76BB-21EB-17E095F33BAA}
copy "Main.hotspot.wxs"
        1 file(s) copied.
HEAT

E:\workspace\build-scripts\release\create_installer_windows\wix>"!WIX!bin\heat.exe" dir "!REPRO_DIR!" -out Files-!OUTPUT_BASE_FILENAME!.wxs -gg -sfrag -scom -sreg -srd -ke -cg "AppFiles" -var var.ProductMajorVersion -var var.ProductMinorVersion -var var.ProductVersionString -var var.MSIProductVersion -var var.ReproDir -dr INSTALLDIR -platform !PLATFORM! 
The system cannot find the path specified.

This is from https://ci.adoptopenjdk.net/job/build-scripts/job/release/job/create_installer_windows/12541/console - however, I image the full log will be lost considering how many installer jobs are run. Every run with that machine has the same error though.

[Willsparker]

From Philippe : https://adoptopenjdk.slack.com/archives/C09NW3L2J/p1614066011009100
WIX isn't located properly, and should be set as :
set WIX="C:\PROGRA~2\WIXTOO~1.14\"

[Willsparker]

The issue is that build-ibmcloud-win2012r2-x64-1's Program Files (x86) shortname is PROGRA~1 , not PROGRA~2. I'll try and swap them now as the machine isn't doing anything just this second. This will require me to stop any processes in both Program Files folders, so some unexpected stuff may happen. Will update once it's done (and put a PR to make it PROGRA~2 by default)

[Willsparker]

Okay, so for Program Files (x86), I had to stop a couple of Incredibuild processes, and for Program Files, I had to stop the Jenkins agent java process, and an SQL server process. None of those appeared to cause issues, though I briefly stopped the Jenkins agent service to stop the process restarting.

Now, Program Files = PROGRA~1 and Program Files (x86) = PROGRA~2, so should hopefully work!

[sxa]

Why couldn't we just set the WIX variable to the correct location?

[Willsparker]

As far as I'm aware, the installer is hardcoded to look for PROGRA~2 specifically - if we changed it to PROGRA~1, we'd have to change the shortnames for the already existing WIX machines.

[Willsparker]

https://ci.adoptopenjdk.net/job/build-scripts/job/release/job/create_installer_windows/12571/
All other installer jobs look green too! Closing issue :-)

[sxa]

As far as I'm aware, the installer is hardcoded to look for PROGRA~2 specifically - if we changed it to PROGRA~1, we'd have to change the shortnames for the already existing WIX machines.

Hmmm ... I'm going to reopen this as my queries have not been addresed (maybe we'll just spin off another issue) and I don't think hardcoding WIX in the create_installer_linux job with a shortname is necessarily the right thing to do so if we can do anything to improve the detection, or use a variable on the machine (for example) would be preferable. Shouldn't be hard to have a slightly more scalable solution.

Related: We've got most other jenkins jobs in pipelines under version control now, so there may be potential to do the same for the create_installer_windows job too.

[Willsparker]

I would have thought this would constitute an installer issue (though, I can't find any instance of PROGRA~ with the github search). I agree that a variable would probably be best - I thought there was one, honestly.

[douph1]

I'm agree that the install job not set static path.
Tree options:
1) The infra take care of putting WIX on the PATH
2) The infra take care of setting WIX with the installed path
3) The build pass a param to the installer job to tell him where wix is installed

I prefer 2) over 1) which polluting PATH and over 3) which add complexity not required
If you are agree with me, tell me when the WIX variable will be available and I will delete it from the installer job.

Related: We've got most other jenkins jobs in pipelines under version control now, so there may be potential to do the same for the create_installer_windows job too.

I can't find any instance of PROGRA~ with the github search

100% Agree, I thinking about it since long.... time but have not found any time to spare on it since.
@gdams as started something new .. using github action to build installer.
I just want to know if it is intended to replace jenkins create_installer_windows job or not before starting to work on a Jenkinsfile

[sxa]

I would have thought this would constitute an installer issue (though, I can't find any instance of PROGRA~ with the github search). I agree that a variable would probably be best - I thought there was one, honestly.

As I said the WIX definition is hard coded in the Jenkins job, not in GitHub. My last paragraph was proposing moving that job definition into GitHub

[sxa]

I'm agree that the install job not set static path.
Tree options:
1) The infra take care of putting WIX on the PATH
2) The infra take care of setting WIX with the installed path
3) The build pass a param to the installer job to tell him where wix is installed

Can it be made to work with the long name path (i.e. with a space in it?) For some other things we've made links under c:\openjdk which works be another option (I don't like that one so much though)

I don't think I've seen that actions proposal anywhere in an issue

[Willsparker]

Can we open a new issue for this in the Installer repo? build-ibmcloud-win2012r2-x64-1 can now run the installer jobs, and I presume that the job definition will be in the Installer repo if/when it's moved.

[sxa]

We still need to verify whether it can run with the full path (i.e. not requiring shortnames) which is independent of moving the scripts into github, so yes creating another issue in the installer repo is a good idea, but I would suggest this can't be closed yet.

[sxa]

For what it's worth, it doesn't work without additional modifications - this was from setting WIX to the full path "C:\Program Files (x86)\Wix Toolset v3.14":

J:\jenkins\workspace\build-scripts\release\create_installer_windows\wix>"!WIX!bin\heat.exe" dir "!REPRO_DIR!" -out Files-!OUTPUT_BASE_FILENAME!.wxs -gg -sfrag -scom -sreg -srd -ke -cg "AppFiles" -var var.ProductMajorVersion -var var.ProductMinorVersion -var var.ProductVersionString -var var.MSIProductVersion -var var.ReproDir -dr INSTALLDIR -platform !PLATFORM! 
'""C:\Program' is not recognized as an internal or external command,
operable program or batch file.

[Willsparker]

Can you try setting it as C:\Program Files (x86)\Wix Toolset v3.14 - without the double quotes?

This part of the error message: '""C:\Program' makes me think that they're being closed prematurely

[douph1]

Yes @Willsparker when settings this WIX var because we call it inside double quotes already, the var must not be set with quote too.
I just tested with SETLOCAL ENABLEDELAYEDEXPANSION we use and double quotes during call is not needed to protect again space. I can remove it to allow Wix var to work in both way :
SET WIX=C:Program Files (x86)\Wix Toolset v3.14
or
SET WIX="C:Program Files (x86)\Wix Toolset v3.14"

[douph1]

Tested with github action in place: env var WIX is set without double quote so the script who use the var must enclosed it with double quote.
To protect the space we need to enclose it with double quote in assignement OR usage but not both.

To keep compatibility with github action we must stick with "!WIX!bin\heat.exe" and never set "..." on var assignment
and it must work with long path with space
So the playbook can set var like this :
WIX=C:Program Files (x86)\Wix Toolset v3.14
but not
WIX="C:Program Files (x86)\Wix Toolset v3.14"
I close the linked merge-request 297

[Willsparker]

Ah, hang on - If you set a variable like that, i.e. set WIX=C:\Program Files (x86), it would cut off at C:\Program, as there's a space in it.I suppose this is why it uses Shortnames

[douph1]

Who will cut it ? Ansible ?

[Willsparker]

Powershell (or whichever shell you're using to run set WIX ...etc). To set variables with a space in them, you need to have those double quotes, otherwise the first space of the variable will be considered the end of the variable.

[douph1]

Not true for batch file (.cmd)
set calcu=W:\Philippe\Folder with space
work as expected and keep the whole line in the variable
Wich "shell" are we using to defined WIX env var ?
I don't know Ansible but it seems to handle value with space without quote:

• name: Set an environment variable for all users
  ansible.windows.win_environment:
  state: present
  name: TestVariable
  value: Test value
  level: machine

https://docs.ansible.com/ansible/latest/collections/ansible/windows/win_environment_module.html

I think the debate must stay focus in what techno/shell we use to set the WIX var .. I assume it was ansible but don't find it on the repo.
Is it Wix installer who set the env var itself ?

[Willsparker]

We don't set any particular variable in our playbooks: https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/WiX/tasks/main.yml

So, if we can set variables with spaces, without double quotes, then that should work, if I'm reading this correctly?

As for where the variable is set, I'm not sure! I thought it was within the Jenkins Job? I suppose the installer could have set it, itself

[douph1]

Ok, I have tested on standalone_create_installer_windows and finally changed on create_installer_windows result here:
https://ci.adoptopenjdk.net/job/build-scripts/job/release/job/create_installer_windows/12897/
With jenkins longpath with space only work if we delay expansion of variable.

I have added a job parameter WIXPATH with default value to the long form of the path : C:Program Files (x86)\Wix Toolset v3.14
so it is transparent and configurable.
The demonstration that the installer job can use path with space is done (without double quotes).

I propose a last step :
The ansible playbook set directly a WIX env var with the installed path (with or without space) but always without double quotes as now
This will enabled the capability to have different version of Wix on different host
because the path is not related to the installer job or the upstream caller job but only from the provisioned host choosed dynamicaly by jenkins at job start

[Willsparker]

Just checked on the build-ibmcloud-win2012r2-x64-1 machine, the WiX installer does set that variable, without double quotes:

C:\Users\Administrator\Desktop>echo %WIX%
C:\Program Files (x86)\WiX Toolset v3.14\

(this is in cmd).

So it should be a case of setting the Jenkins Job variable to be whatever is shown in %WIX% :-)

[douph1]

build-azure-win2012r2-x64-1 :

IF DEFINED WIX ECHO WIX var is already set to : %WIX%
WIX var is already set to : C:Program Files (x86)\WiX Toolset v3.14\

It is already present .. probably on all host where wix is installed (set by the wix setup) !
So I will try to remove all that mess and rely on the predefined WIX var if @sxa is ok with that end ?

[sxa]

We'd need to verify that it is defined on all hosts, but if that works it seems like a good idea - thanks! I'd suggest waiting until after we have the JDK16 GA releases out though (They have all been built at the moment but until they have been shipped on Tuesday or Wednesday next week there is always a chance we will have to rebuild them for some reason)

[douph1]

Ok, I have rollbacked the early morning change (when adding WIXPATH) to be as previously
and waiting the JDK16 GA releases

[Willsparker]

We'd need to verify that it is defined on all hosts

Just checked, build-azure-win2012r2-x64-2 also has that environment variable set, so that's all machines with the WIX tag set :+1:

[karianna]

Is this complete now?

[Willsparker]

I _think_ a change needs to be made, so the Installer Jenkins Jobs just uses the %WIX% variable, if I'm interpreting the above conversation correctly :)

[sxa]

Correct @Willsparker

[douph1]

Can we consider the release week of JDK 16 is ended so I can make the change ?

[sxa]

Can we consider the release week of JDK 16 is ended so I can make the change ?

Yep - this can go ahead now :-)

[douph1]

set WIX="C:\PROGRA\~2WIXTOO~1.14\"

removed from all jobs

[sxa]

Runnig a new JDK8 pipeline with all Windows variants at https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/1564/ to see if it shows any problems on the machines it runs on :-)

[sxa]

Of all the variants only 13065 went onto that machine, but it seemed to work successfully. 13070 on build-ibmcloud-2 passed as well so I'veadded the wix label to that machine as well, giving us four that we can run on.

Thanks @douph1 :-)