Openj9: subst command does not work on Windows build at ci.eclipse

[Mesbah-Alam]

System tests are failing in ci.eclipse openj9 Windows build with :

systemtest-prereqs has been processed, and set to: C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64_cmprssptrs/jvmtest/systemtest/systemtest_prereqsSubstituting test_dir C:\Users\jenkins\workspace\Test-sanity.system-JDK8-win_x86-64_cmprssptrs\openjdk-tests\TestConfig\test_output_15435624397491\ClassLoadingTest_0\20181130-012042-ClassloadingLoadTest with T: to avoid long path lengths
Running subst T: /d
subst T: /d returned: 256
Access denied - T:
Running subst T: C:\Users\jenkins\workspace\Test-sanity.system-JDK8-win_x86-64_cmprssptrs\openjdk-tests\TestConfig\test_output_15435624397491\ClassLoadingTest_0\20181130-012042-ClassloadingLoadTest
subst T: C:\Users\jenkins\workspace\Test-sanity.system-JDK8-win_x86-64_cmprssptrs\openjdk-tests\TestConfig\test_output_15435624397491\ClassLoadingTest_0\20181130-012042-ClassloadingLoadTest returned: 256
Access denied - C:\Users\jenkins\workspace\Test-sanity.system-JDK8-win_x86-64_cmprssptrs\openjdk-tests\TestConfig\test_output_15435624397491\ClassLoadingTest_0\20181130-012042-ClassloadingLoadTest
Retrieving amount of free space on drive containing C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64_cmprssptrs/openjdk-tests/\TestConfig\test_output_15435624397491\ClassLoadingTest_0
Uncaught exception from user code:
    Not able to write arguments to file 'T://stf_parameters.properties' at C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64_cmprssptrs/jvmtest/systemtest/stf/stf.core/scripts/stfArguments.pm line 235.
    stfArguments::write_arguments_to_file("T://stf_parameters.properties", "C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64"..., "C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64"..., "C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64"...) called at C:/Users/jenkins/workspace/Test-sanity.system-JDK8-win_x86-64_cmprssptrs/openjdk-tests///..//jvmtest\systemtest\stf\stf.core\scripts\stf.pl line 410

Job link: https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-win_x86-64_cmprssptrs/91/

Note:
1)subst command is failing on Windows machine in ci.eclipse. It works fine in Adopt, e.g, https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/475/consoleFull
2) Stf uses subst to substitute long folder paths to a virtual drive T: https://github.com/AdoptOpenJDK/stf/blob/21a60ab0fecaf127142882aa37ffce1cf9438fa4/stf.core/scripts/stf.pl#L353
3) Nothing I found on the internet suggests that subst should not work using normal user privileges.

We need to understand why subst is not working in the Windows machines in ci.eclipse while they work fine in Adopt.

[Mesbah-Alam]

FYI... @AdamBrousseau

[Mesbah-Alam]

Note: This blocks Windows system testing at ci.eclipse: https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-win_x86-64_cmprssptrs/95/tapResults/

[jdekonin]

The eclipse/openj9 windows machines are connected to jenkins via cygwin ssh, which defaults the login shell to bash, not a command prompt (cmd.exe) like how I believe the Adopt machines are connected. The subst command works fine when running through a cmd.exe, but fails when run through cygwin bash, because bash is finding it on the path (/cygdrive/c/Windows/system32/).

This is on the cygwin path because c:\Windows\system32 is in the windows environment variable PATH which cygwin only adds/prefixes too. From what I could see there is no subst for cygwin.

I am not entirely sure what the subst use case is, can you detail why its needed? This won't help the fact that cygwin won't support it, but maybe there is an alternative that can be used within a OSTYPE=cygwin environment.

[Mesbah-Alam]

This won't help the fact that cygwin won't support it

Are you saying Cygwin will not support subst?

In STF, subst is used to substitute a file path if it is larger than 260 characters, on Windows : https://github.com/AdoptOpenJDK/stf/blob/21a60ab0fecaf127142882aa37ffce1cf9438fa4/stf.core/scripts/stf.pl#L345

It is also used in TestConfig settings.mk in multiple places: https://github.com/eclipse/openj9/blob/master/test/TestConfig/settings.mk

@jdekonin @renfeiw

[AdamBrousseau]

Is subst even required for a Cygwin environment?

[Mesbah-Alam]

@AdamBrousseau - good point!

I did some googling and found that from version 1.7.0, for Cygwin, PATH_MAX=4096: https://cygwin.com/ml/cygwin-announce/2009-02/msg00018.html

Are we using Cygwin 1.7.0 or newer? If so, when we are on Cygwin environment, we do not need to use subst (in System test). I'll make the change and try a personal build to see how far I get... (I fear we will now break in TestConfig's use of subst).

[AdamBrousseau]

OpenJ9 machines have 2.10.0

jenkins@oj9-win2012r2-1 ~
$ cygcheck --version
cygcheck (cygwin) 2.10.0
System Checker for Cygwin
Copyright (C) 1998 - 2018 Cygwin Authors
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[Mesbah-Alam]

And the Adopt machines? Can you find out the Cygwin version over there?

[AdamBrousseau]

test-azure-win2012r2-x64-1

cygcheck (cygwin) 2.10.0
System Checker for Cygwin
Copyright (C) 1998 - 2018 Cygwin Authors
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[renfeiw]

There are multiple places we are using subst in TKG, so it looks like we still need it.

[renfeiw]

The functional test builds have no issue on windows: https://ci.eclipse.org/openj9/view/Test/job/Test-sanity.functional-JDK8-win_x86-64_cmprssptrs/380/

[AdamBrousseau]

I don't see subst mentioned in that console output. Is it being used? Can we see how it is being used?

[renfeiw]

It is used in TestConfig settings.mk in multiple places: https://github.com/eclipse/openj9/blob/master/test/TestConfig/settings.mk
For example https://github.com/eclipse/openj9/blob/82d9955f5b161cc1fb90742ca6d23272d929cbfe/test/TestConfig/settings.mk#L157

[AdamBrousseau]

This usage looks different than the one that's failing in systemtest. Afaict this replaces characters in a variable. The systemtest is using it to create short dir paths.

[Mesbah-Alam]

I have removed the use of subst in stf.pl in my stf branch (Please see discussion here: https://github.com/AdoptOpenJDK/stf/issues/49).

Ran a Grinder in internal jenkins and it got passed the previous issue (seems to be working normally now): https://hyc-runtimes-jenkins.swg-devops.com/view/Test_grinder/job/Grinder/1010/console

@AdamBrousseau - Are we using the same set of Windows machines in openj9 and the internal Jenkins?

[jdekonin]

Complete different machines, but setup with the same AdoptOpenJDK playbook as a guide.

[Mesbah-Alam]

Tested with a grinder in openj9 (seems to be working fine there too) https://ci.eclipse.org/openj9/view/Test/job/Test-Grinder/122/console

[Mesbah-Alam]

Grinder link for Adopt: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/582/console

[Mesbah-Alam]

https://github.com/AdoptOpenJDK/stf/issues/49 has been merged. System tests are running fine on Windows now in Adopt, OpenJ9 and internal jenkins. This issue may be closed at this point.