Openjdk-infrastructure: System unavailable: test-azure-win2012r2-x64-1 cannot clean workspace
test-azure-win2012r2-x64-1 is failing to run tests due to not being able to clean ots workspace: https://ci.adoptopenjdk.net/job/Test_openjdk8_j9_extended.system_x86-64_windows/361/
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
Cannot delete workspace: null
Option not to fail the build is turned on, so let's continue
[Pipeline] sh
+ find /cygdrive/c/temp -name '*core*' -print -exec rm -f '{}' ';'
/cygdrive/c/temp/vs2013_install_log_005_vc_compilercore86.txt
/cygdrive/c/temp/vs2013_install_log_005_vc_librarycore86.txt
/cygdrive/c/temp/vs2013_install_log_006_vc_compilercore86res.txt
/cygdrive/c/temp/vs2013_install_log_006_vc_librarycore.txt
/cygdrive/c/temp/vs2013_install_log_007_vc_compilercore.txt
/cygdrive/c/temp/vs2013_install_log_008_vc_compilercoreres.txt
/cygdrive/c/temp/vs2013_install_log_008_vs_professionalcore.txt
/cygdrive/c/temp/vs2013_install_log_015_tfs_objectmodelcore_x64.txt
/cygdrive/c/temp/vs2013_install_log_016_tfs_objectmodelcoreres_x64_enu.txt
/cygdrive/c/temp/vs2013_install_log_019_netfx_451mtpackcore.txt
/cygdrive/c/temp/vs2013_install_log_020_netfx_451mtpackcorelp.txt
/cygdrive/c/temp/vs2013_install_log_023_vs_minshellcore.txt
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin\java.exe
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin\java.exe
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin\java.exe
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove directory C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin with directory contents: [C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\bin\java.exe]
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:250)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9dmp29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9dmp29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9dmp29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9gc29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9gc29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9gc29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9hookable29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9hookable29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9hookable29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9jit29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9jit29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9jit29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9jvmti29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9jvmti29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9jvmti29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9prt29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9prt29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9prt29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9shr29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9shr29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9shr29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9thr29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
Also: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9thr29.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary\j2sdk-image\jre\bin\compressedrefs\j9thr29.dll
at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 13.68.134.204/13.68.134.204:50345
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.FilePath.act(FilePath.java:1070)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.deleteRecursive(FilePath.java:1304)
at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
jenkins.util.io.CompositeIOException: Unable to delete 'C:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-64_windows\openjdkbinary'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 29 additional exceptions)
at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:99)
at hudson.Util.deleteRecursive(Util.java:295)
at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1310)
at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1306)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3122)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:375)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:118)
at java.lang.Thread.run(Unknown Source)
Finished: FAILURE
lumpfish
All 25 comments
These are mostly covered under https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1573
sxa
on 9 Nov 2020
Hmmm there doesn't obviously seem to be a process using that file at the moment - I've queued up a new version of that job on the macine so we can see if it recurs when it runs (@lumpfish you have Grinder running just now so it will happen once your run is finished)
sxa
on 9 Nov 2020
Now Running https://ci.adoptopenjdk.net/job/Test_openjdk8_j9_extended.system_x86-64_windows/365 on test-azure-win2012r2-x64-1 after renaming that directory from the workspace (which seems to have been successful
sxa
on 9 Nov 2020
(ref #1396 )
Willsparker
on 10 Nov 2020
@Willsparker Can you take a look on this given that moving the directory doesn't appear to have resolved this (the job in the previous comment failed so I've taken it offline) - let's try and understand what's going on here rather than just deleting / rebooting as it wasn't immediately obvious to me what was causing the problem here
sxa
on 10 Nov 2020
Yep - pretty sure it's leftover Java processes (again)
So, the Java process 14984, is apparently using various .jar files in the workspace, stopping it from being deleted.
(On the same machine, there's another java process that is affecting the Test_openjdk8_j9_sanity.openjdk_x86-64_windows folder too, in the same way).
Tests aren't clearing up their java processes properly, and Jenkins isn't handling them properly. So- 2 options. We run your SXA-ProcessCheck Jenkins job (or an equivalent Java process checker) after every windows test (As I've suggested here and here). OR, we build the same functionality into each test run - I can see in this log, that make -f ./openjdk-tests/TKG/testEnv.mk testEnvTeardown is run at the end of the tests, though it doesn't look like it does much (https://github.com/AdoptOpenJDK/TKG/blob/master/testEnv.mk). I'm not too clued up on Makefiles so I'm not sure if we could use that to query if there's any Java processes in the same workspace?
As the machine is disabled, I'm going to leave those 2 java processes running, just in case we want to trial anything. I'd like to try running SXA-ProcessCheck on the machine to see if it does pick up those rogue processes, if you don't want to use them for any other reason.
Willsparker
on 11 Nov 2020
Can you tell how long the processes have been there for? I recently fixed this issue which would have meant system tests did not clean up processes: https://github.com/AdoptOpenJDK/stf/pull/89 (merged 05/11/2020). That would only account for processes started via extended.system or sanity.system (and also potentially a few grinders).
lumpfish
on 11 Nov 2020
The issue I have with using SXA-processCheck as-is for this is that you can't always identify within the test suite that the problem is occuring. A better option would be to incorporate the same check/delete code into the test jobs and then fail the job with a warning if processes have been left around - that way it can all be part of the same job.
Looking at azure-test-1 if I run ps -ef | grep java it doesn't show anything so the logic in SXA-procesCheck is not sufficient to correctly detect these problems on Windows.
sxa
on 11 Nov 2020
@lumpfish I can't see how long the process has been running, but the folders it's locking up is from the 2nd / 3rd of November.
@sxa tasklist works; but it doesn't say which is the Jenkins agent - but it does pick up the problematic one, 14984
Willsparker
on 11 Nov 2020
Tasklist unfortunately doesn't seem to let you filter on the whole command line. There's probably a way of doing it through PowerShell but I'm not sure what the right incantation is - we'd likely need to that to be able to reliable filter out the jenkins agent process (and determine which test is causing the problem) ...You can see the details in the process' properties in task manager for each individual process but that's not scriptable.
The tlist tool will do it but that's not a default part of a windows installation AFAIK. If we can do it with standard systems tools that would be preferable
sxa
on 11 Nov 2020
Got it- in Powershell.
PS Get-Process java | select Path,ID
Path ID
---- ---
C:\Program Files\Java\jre1.8.0_171\bin\java.exe 392
C:\Users\jenkins\workspace\Test_openjdk8_j9_extendeded.sys... 14984
C:\Users\jenkins\workspace\Test_openjdk8_j9_sanity.openj... 28560
C:\Users\jenkins\workspace\Test_openjdk8_j9_sanity.openj... 30948
(typed, apologies for bad formatting).
And Jenkins agent is running the Program Files java :-)
Willsparker
on 12 Nov 2020
Did you kill off the jobs - I can't see those last three on the machine any more?
sxa
on 12 Nov 2020
No? :confused:
Willsparker
on 12 Nov 2020
~This also doesn't work reliably across platforms 馃槩 - on Windows 2016 for example it looks like the Path select doesn't show anything~
Scratch that - it needs to be run as Administrator to get that information
sxa
on 13 Nov 2020
Had to clear out several machines today which had a load of leftover processes, mostly from the recently added extended test suite runs. Details below:
https://ci.adoptopenjdk.net/computer/test-godaddy-win2016-x64-1/
https://ci.adoptopenjdk.net/computer/test-godaddy-win2016-x64-4/
https://ci.adoptopenjdk.net/computer/test-azure-win2012r2-x64-3/
sxa
on 13 Nov 2020
Okay, so I had a look at Get-Process as a Jenkins user. I'm presuming that we can't just make this run as Admin, so I was trying to find something else of use. Turns out, you can get the start time of the process as the Jenkins user:
From the StartTime, it's fairly obvious that the process w/Id 3236 is the Jenkins Agent
I'm fairly sure that this wouldn't be airtight though- for example, a test runs and leaves a process over, the Jenkins agent fails and is then restarted, the oldest process is no longer the Jenkins agent. (This is assuming that the Test process doesn't automatically stop when the Jenkins agent process stops.)
So, what about if we go with removing Java processes that had started between the start of the Jenkins Job and the end of it. Something like the following works for the Jenkins user
PS Get-Process java | select id,starttime | where {$_.StartTime -gt (Get-Date).AddHours(-1)}
This will return the ID and starttime of the Java Process who has started within the last hour. If there's a way we can get the start-time of the Jenkins run and convert it to a Powershell Datetime object, we can just compare them :-)
Willsparker
on 13 Nov 2020
There we go, I've found a more generalised solution that will find all the Java Processes that have been created since the Jenkins job started running:
Ran in mintty.exe ; I tested this with cmd and it seemed to work
export jobStart=$(date -R)
... # Run Test
powershell -c "Get-Process java | select id,starttime | where {\$_.StartTime -gt (Get-Date -Date \"$jobStart\")}"
Thanks Will!
@smlambert @andrew-m-leonard Does the above look feasible for inclusion into the test scripts? It should kill off any processes created since the start of the job.
Worth considering what we want to do in this situation - should the test job be marked as "failed" or "warning"? I suspect we should display the list and (at the very least) make it a warning condition.
sxa
on 16 Nov 2020
Sounds good to me
andrew-m-leonard
on 16 Nov 2020
As we add new testing environment checks (https://github.com/AdoptOpenJDK/TKG/issues/45) into TKG, we have started with warnings.
smlambert
on 16 Nov 2020
@smlambert Do you agree that we should add the above proposed operations from Will into the testing scripts? Would adding it into maketest.sh be appropriate and would it be best "guarded" so that it only terminates processes when executed via jenkins (e.g. if EXECUTOR_NUMBER is defined or similar)?
sxa
on 16 Nov 2020
I linked to https://github.com/AdoptOpenJDK/TKG/issues/45 (see 4th entry in checklist) because these checks may more properly belong in TKG where we have a set of machine checks.
smlambert
on 16 Nov 2020
So, I guess I misunderstood what Will's solution does, which is using Jenkins info as a corral around the processes started. This does not help us identify issues in other CI systems, or when someone runs on their laptop via commandline. It would be good to cover those cases (which is why we proposed putting them into TKG). Would be worth a design discussion so we consider all options.
(FYI @renfeiw @llxia)
smlambert
on 16 Nov 2020
@Willsparker Solution is simply a "run command to store the start time" then "check for new java processes since then". It's not tied to jenkins at all (unless we tie it to the presence of EXECUTOR_NUMBER - that was mostly to ensure that any other java processes run outside the test during the test wouldn't be affected) which is why my first proposal was to put it into maketest.sh which is always executed. Happy for it to go into TKG instead if there are specific targets in there that are guaranteed to be executed at the start and end (including if the tests fail)
sxa
on 16 Nov 2020
I think I'm going to close this and move follow on discussion to https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1573 to avoid fragmented discussions - we will add in @Willsparker proposed code into maketest.sh
sxa
on 18 Nov 2020
Related issues
smlambert
路
6Comments
aahlenst
路
6Comments
Willsparker
路
9Comments
sxa
路
7Comments
LongyuZhang
路
4Comments