Openj9: CUDA should be installed on build machines

[keithc-ca]

My first attemp to enable CUDA in openj9 builds yields this error:

cygpath: cannot create short name of C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0

which, I presume, means that the folder is not present. CUDA version 9.0 should be installed on OpenJ9's build machines, just like it is at adoptopenjdk.net (see [1]).

[1] https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/NVidia_Cuda_Toolkit/tasks/main.yml

[keithc-ca]

FYI @jdekonin @AdamBrousseau

[keithc-ca]

See also https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_windows_Personal/285/consoleText

[keithc-ca]

Linux machines need work too (at least ppc64le): see https://github.com/eclipse/openj9/pull/7429#issuecomment-540831345

[jdekonin]

Looks like its only installed on win2012r2-x86-1, I'll get it on others in the morning when workload is light(er).

[jdekonin]

I have the playbook applied for win2012r2-x86-2-6. Please advise if there is anything else on this issue that requires work.

Linux machines to be worked through #7429

[pshipton]

Testing Windows https://github.com/eclipse/openj9/pull/7430#issuecomment-543776287

[pshipton]

The build didn't work, but not obviously a machine problem this time. Keith will look at it.

[jdekonin]

7429 is a PR, didn't realize that at first; so I guess I'll use this issue as it is abstract is generic enough to include plinux. :)

[keithc-ca]

The relevant part of the jdk11 build log [1] is this:

00:04:56.902  configure: Rewriting cuda_home to "/cygdrive/c/program files/nvidia~2/cuda/v9.0"
00:04:57.339  checking CUDA_HOME... C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.0 [C:/program
00:04:57.339  files/nvidia~2/cuda/v9.0]
00:04:57.339  checking for cuda... yes (explicitly set)

The first line quoted above should say

00:04:56.902  configure: Rewriting cuda_home to "/cygdrive/c/progra~1/nvidia~2/cuda/v9.0"

The configuration script uses cygpath -m -s which should have shortened the "Program Files" segment to "PROGRA~1" (which is then converted to lower-case).
Perhaps the version of cygwin is out-dated?

[1] https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_windows_Personal/311/

[pshipton]

@jdekonin any comment on Keith's reference to using an outdated version of cygwin? What version are we using? @keithc-ca what version did you test with?

[keithc-ca]

$ cygpath --version
cygpath (cygwin) 2.11.1
Path Conversion Utility
Copyright (C) 1998 - 2018 Cygwin Authors

It's also possible that a short name doesn't exist for C:\Program Files - we can check with dir/x:

>dir /x C:\
 Volume in drive C has no label.
 Volume Serial Number is 7258-9D8E

 Directory of C:\

2018-06-20  11:53    <DIR>                       cygwin64
2018-12-04  15:05    <DIR>                       LLVM-32
2018-12-04  15:04    <DIR>                       LLVM-64
2018-12-06  10:58    <DIR>          MINGW-~1     mingw-w64
2019-01-29  14:18    <DIR>                       openjdk
2013-08-22  11:52    <DIR>                       PerfLogs
2019-07-15  14:39    <DIR>          PROGRA~1     Program Files
2019-10-07  18:34    <DIR>          PROGRA~2     Program Files (x86)
2019-10-11  13:56    <DIR>                       space
2019-01-29  14:00    <DIR>          STRAWB~1     Strawberry
2019-07-15  14:40    <DIR>                       temp
2018-11-07  14:21    <DIR>                       Users
2019-10-07  10:19    <DIR>                       Windows

A third possibility is that the root directory is not readable by the build userid. [1] says

GetShortPathName may fail when it is unable to query the parent directory of a path component to determine the short name for that component

[1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getshortpathnamew

[jdekonin]

$ cygpath --version
cygpath (cygwin) 2.10.0

Short name doesn't exist on win2012r2-x86-1.

09/05/2018  09:32 AM    <DIR>          MINGW-~1     mingw-w64
01/21/2019  02:46 PM    <DIR>          OPENSS~2.1-X OpenSSL-1.1.1-x86_32
01/21/2019  02:46 PM    <DIR>          OPENSS~1.1-X OpenSSL-1.1.1-x86_64
...
11/11/2018  12:15 AM    <DIR>                                     Program Files
10/22/2018  07:06 AM    <DIR>          PROGRA~2     Program Files (x86)
04/13/2018  08:11 AM    <DIR>          STRAWB~1     Strawberry

[keithc-ca]

There is API to create the short name after the fact: [1], and it's available via the fsutil tool, [2].

> cd/d C:\
> fsutil file setshortname "Program Files" "PROGRA~1"

[1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setfileshortnamea
[2] https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/fsutil-file

[jdekonin]

To do this we need to boot into safe mode which it appears we cannot do remotely following method 2 [1]. It appears that win2012r2-x86-1 is no longer reachable for builds until we can get that fixed.

1. https://www.top-password.com/blog/boot-windows-server-2012-into-safe-mode/

[keithc-ca]

When I tried that command, safe mode was not needed.

[jdekonin]

In a cmd shell or power shell run as administrator I get this output on all the machines.

C:\Users\Administrator>cd/d C:\

C:\>fsutil file setshortname "Program Files" "PROGRA~1"
Error:  Access is denied.

[jdekonin]

Not ideal, but I did a MKLINK /D PROGRA~1 "c:\Program Files" which means the path I think you are trying to use will be present.

[pshipton]

Still failing the same way.
https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_windows_Personal/346/
win2012r2-x86-2
custom-spec.gmk:50: *** invalid syntax in conditional

[keithc-ca]

Not ideal, but I did a MKLINK /D PROGRA~1 "c:\Program Files" which means the path I think you are trying to use will be present.

I don't think that will work. What is the output of dir /x C:\ now?

[pshipton]

I don't think that will work.

Indeed, it did not work. https://github.com/eclipse/openj9/issues/7432#issuecomment-546438817

[keithc-ca]

@jdekonin Any updates on getting the build machines fixed (so there's a short name for each segment of the path C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.0)?

[jdekonin]

@rajdeepsingh1 has a plan for this but we want to wait until we have access to the management console for these systems. We currently only have 4 of 6 online, I don't want to give the go ahead on a functional system to have that one accidentally removed from availability as well.

[keithc-ca]

Can you share a summary of that plan? Thanks.

[rajdeepsingh1]

@keithc-ca

It seems the Program Files directory short name was not being set due to access denied errors. The directory has a tendency of being locked while the user (administrator) is logged in, since it is housing system files.

On my Win2012 test machine I was able to schedule a task (similar to a cronjob) with the Task Scheduler utility.

The task will initiate the following command immediately at system startup fsutil file setshortname "C:\Program Files" "PROGRA~1"

Once the task is created, a soft boot would be required in order for the command to propagate before login.

[keithc-ca]

I tried that too, but maybe the trigger I specified was not early enough. Let's hope that works.

[rajdeepsingh1]

@keithc-ca

I've tested the scenario of win2012r2-5 but it did not work. The trigger specified was at startup

I exported the task from my test machine and copied it to win2012r2-5

The machines both have the exact same permissions and reg settings for setting short names, but still did not work. A test directory was created with a folder inside of Program Files on win2012r2-5 and the task was able to rename it with no issues.

Still need to figure out how to set shortname for the Program Files dir.

[keithc-ca]

Here's another suggestion which doesn't require using safe mode. Create Multi-String registry entry HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations with the value:

\??\C:\Program Files
\??\C:\PROGRAMS
\??\C:\PROGRAMS
\??\C:\Program Files

At boot time, Program Files is first renamed to PROGRAMS which should remove the short name (it would be the same as the long name). Then PROGRAMS is renamed to Program Files which should trigger creation of the normal short name.

[rajdeepsingh1]

win2012r2-3
win2012r2-4
win2012r2-5
win2012r2-6

All of the systems above now have the shortnames set to PROGRA~1 and PROGRA~2

win2012r2-1 is currently being rebuilt, playbook is running - the shortnames were successfully applied to this machine aswell.

[keithc-ca]

Build on win2012r2-x86-3 for #7430 passed:
https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_windows_Personal/390/