Openjdk-tests: Multiple JTReg JDI Tests failing on Windows

[M-Davies]

Describe the bug
We currently have approximately 36 JTReg tests consistently failing on JDK8/hs/x32_windows and JDK8/hs/x64_windows&&x32_windows nightlies. It's the same group every night since the 31st March and they're all com/sun/jdi/*.sh tests.

• https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/111/#showFailuresLink
• https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-32_windows/95/#showFailuresLink

FAILING TESTS

com/sun/jdi/ArrayLengthDumpTest.sh.ArrayLengthDumpTest
com/sun/jdi/BreakpointWithFullGC.sh.BreakpointWithFullGC
com/sun/jdi/CatchAllTest.sh.CatchAllTest
com/sun/jdi/CatchCaughtTest.sh.CatchCaughtTest
com/sun/jdi/CatchPatternTest.sh.CatchPatternTest
com/sun/jdi/CommandCommentDelimiter.sh.CommandCommentDelimiter
com/sun/jdi/DeoptimizeWalk.sh.DeoptimizeWalk
com/sun/jdi/EvalArgs.sh.EvalArgs
com/sun/jdi/EvalArraysAsList.sh.EvalArraysAsList
com/sun/jdi/EvalInterfaceStatic.sh.EvalInterfaceStatic
com/sun/jdi/GetLocalVariables3Test.sh.GetLocalVariables3Test
com/sun/jdi/GetLocalVariables4Test.sh.GetLocalVariables4Test
com/sun/jdi/JdbLockTest.sh.JdbLockTest
com/sun/jdi/JdbMethodExitTest.sh.JdbMethodExitTest
com/sun/jdi/JdbMissStep.sh.JdbMissStep
com/sun/jdi/JdbVarargsTest.sh.JdbVarargsTest
com/sun/jdi/MixedSuspendTest.sh.MixedSuspendTest
com/sun/jdi/NotAField.sh.NotAField
com/sun/jdi/NullLocalVariable.sh.NullLocalVariable
com/sun/jdi/Redefine-g.sh.Redefine-g
com/sun/jdi/RedefineAbstractClass.sh.RedefineAbstractClass
com/sun/jdi/RedefineAddPrivateMethod.sh.RedefineAddPrivateMethod
com/sun/jdi/RedefineAnnotation.sh.RedefineAnnotation
com/sun/jdi/RedefineChangeClassOrder.sh.RedefineChangeClassOrder
com/sun/jdi/RedefineClasses.sh.RedefineClasses
com/sun/jdi/RedefineClearBreakpoint.sh.RedefineClearBreakpoint
com/sun/jdi/RedefineException.sh.RedefineException
com/sun/jdi/RedefineFinal.sh.RedefineFinal
com/sun/jdi/RedefineImplementor.sh.RedefineImplementor
com/sun/jdi/RedefineIntConstantToLong.sh.RedefineIntConstantToLong
com/sun/jdi/RedefineMulti.sh.RedefineMulti
com/sun/jdi/RedefinePop.sh.RedefinePop
com/sun/jdi/RedefineStep.sh.RedefineStep
com/sun/jdi/RedefineTTYLineNumber.sh.RedefineTTYLineNumber
com/sun/jdi/StringConvertTest.sh.StringConvertTest
com/sun/jdi/WatchFramePop.sh.WatchFramePop

[M-Davies]

@smlambert can you take a look? It seems like most are thread & permission failures

[sophia-guo]

Those tests were moved from extended bucket to sanity on March 31 as jdk_jdi is part of tier1 openjdk tests. #1695

[smlambert]

Yes, so its likely not a new failure, they used to be in extended.sanity, where they may have been failing for some time now, though I see that I had not enabled weekly runs of extended.openjdk on windows like the other platforms, as we were low on machines (so I can't look at a history).

The interesting part is that its tests that are .sh scripts that fail. We should check if only on certain machines, are there permissions that are not allowing the shell scripts to run on all or some our windows machines.

[M-Davies]

I've kicked off a few sanity.openjdk grinder runs on several different machines that could be affected.

| Machine | Grinder Link | Result
| ------------- | ------------- | ------------- |
| test-aws-win2019-x64-2 | https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2739/ | FAIL |
| test-godaddy-win2016-x64-1 | https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2740/ | FAIL |
| test-godaddy-win2016-x64-2 | https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2741/ | FAIL |
| test-godaddy-win2016-x64-4 | https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2742/ | FAIL |
| test-azure-win2012r2-x64-1 | https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2743/ | FAIL |

[smlambert]

sanity.openjdk will no longer show those failures as the tests have been moved to the extended level ignore that... it will show them!

however, there is no need to run the top-level target (which takes a long time), it is sufficient to run the sub-target, jdk_jdi in the Grinder

(and thanks for launching some Grinders @M-Davies!)

[M-Davies]

I aborted them all just before saw your edit @smlambert 😆. No harm done, I've edited the table above with links to new grinders. Should be a little quicker than from the top level target (thanks for that hint)

[M-Davies]

Looks like it's all of them. Possibly a silly question, but do the jenkins users have permission to make their own folders/files? I noticed a lot of the tests are spitting out these sort of errors

16:52:54  mkdir: cannot create directory '/aa7364': Permission denied

[smlambert]

I believe you have hit upon the root cause of these failures. I know we have had permissions problems on windows very early on in this project, though do not know that these are the same, and not sure specifically what infra actions resolved them, but think this may need to become an infra issue.

[M-Davies]

@sxa555 Could you take a look at this? IIRC we've been having perm failures recently with some boxes? Or is that my imagination

[sxa]

Looks like it's all of them. Possibly a silly question, but do the jenkins users have permission to make their own folders/files? I noticed a lot of the tests are spitting out these sort of errors

16:52:54  mkdir: cannot create directory '/aa7364': Permission denied

Why would anything be trying to write into the root of the filesystem? The jobs shoudn't have access to write to that location on any system. If it does I'd consider it a bug ...

[smlambert]

https://github.com/openjdk/jdk11u/blob/master/test/jdk/com/sun/jdi/ShellScaffold.sh#L287-L330

[sxa]

https://github.com/openjdk/jdk11u/blob/master/test/jdk/com/sun/jdi/ShellScaffold.sh#L287-L330

That is an odd piece of logic but suggests the TESTCLASSES variable isn't set.

[smlambert]

yes, indeed its not set... not sure what happens on other platforms from this line in that code:
TESTCLASSES=cat $tmpFile | sed -e 's@\\\\@/@g'

[M-Davies]

Exclude PR https://github.com/AdoptOpenJDK/openjdk-tests/pull/1724

It should be possible to reproduce this in a grinder. I could investigate this. See what the test is doing further up

[M-Davies]

~Although ShellScaffold.sh is not one of the failing tests and on a JDK11u repo. I'll see if any of the failing tests in the list mess around with paths at all~

Ah I see https://github.com/M-Davies/openjdk-jdk8u/blob/baec42c5baec170b8da5a2fae4388104530b6262/jdk/test/com/sun/jdi/ArrayLengthDumpTest.sh#L92-L104 and other tests use ShellScaffold.sh to set itself up.

[M-Davies]

I ran a grinder with some debug output from https://github.com/M-Davies/openjdk-jdk8u/blob/5715a65e47dac6060cf925c26d408f2bbd5db457/jdk/test/com/sun/jdi/ShellScaffold.sh#L277-L284 to see what the values of some key variables are.

15:51:04  STDOUT:
15:51:04  osname before build break is: 
15:51:04  CYGWIN_NT-10.0
15:51:04  SystemRoot before build break is:
15:51:04  C:\Windows
15:51:04  tmpFile before build break is:
15:51:04  C:\Windows/tmp.1117
15:51:04  TESTCLASSES before build break is:
15:51:04  C:/Users/jenkins.EC2AMAZ-T7I06S9/workspace/Grinder/openjdk-tests/TKG/test_output_15863574524135/jdk_custom_0/work/classes/0/com/sun/jdi

$TESTCLASSES appears to be set correctly but $tmpFile is the one pointing at the root dir. Generated above, the variable is a tmp. double dollar sign. Double dollar sign returns the current process id (according to this SF article anyway) which is what results in the weird numbered file name it's trying to create.

I also see that tmpFile has a forward slash in there when it should really have a backslash to comply with windows file path syntax's. Im not sure if this matters here? @sxa ?

[sxa]

In your code the blog that sets tmpfile before the debug messages is:

         if [ -w "$SystemRoot" ] ; then
            tmpFile=$SystemRoot/tmp.$$
         elif [ -w "$SYSTEMROOT" ] ; then
            tmpFile=$SYSTEMROOT/tmp.$$
         else
            tmpFile=tmp.$$
         fi

Whereas the original code that Shelley pointed at was:

            if [ -w "$Temp" ] ; then
                tmpFile=$Temp/tmp.$$
            elif [ -w "$TEMP" ] ; then
                tmpFile=$TEMP/tmp.$$
            else
                tmpFile=tmp.$$
            fi

Has something changed? $TEMP should be /tmp in a cygwin shell which would make tmpFile be /tmp/tmp.$$ which should be ok. Using the Windows SYSTEMROOT(Defaults to C:\Windows) doesn't seem like a good idea (if for no other reason than because the jenkins user, as a non-Administrator, wouldn't generally be able to write to it

[M-Davies]

This is a discrepancy between 8 and 9+. This was fixed in https://hg.openjdk.java.net/jdk-updates/jdk11u/rev/3ea9823cb503 (Bug ID https://bugs.openjdk.java.net/browse/JDK-8172188) but was not backported to jdk8 for some reason??? Seeing as it is affecting a large number of tests, this should be probably be contributed upstream

@sxa @smlambert do you agree?

[sophia-guo]

@M-Davies Also those shell scripts have been rewritten to java from jdk12+. https://bugs.openjdk.java.net/browse/JDK-8201652

I've talked with @jerboaa in jbs channel https://adoptopenjdk.slack.com/archives/CF3CGHGHW/p1586294477011100 and believe the reason it's not backported to jdk8 is just help-needed:) https://adoptopenjdk.slack.com/archives/CF3CGHGHW/p1586335628012900

[M-Davies]

Ah ok. Thanks @sophia-guo . I've opened up a request to the team here in Hursley to see if anyone can pick this up as there's quite a lot on my plate at the moment

[M-Davies]

Same story for https://github.com/AdoptOpenJDK/openjdk-tests/issues/1718

[M-Davies]

I haven't seen any indication of these tests failing in the last few days. Closing...

[sophia-guo]

same as #1737. Reopen.