Openjdk-tests: SDK fetch: Curl retries refuse to overwrite file.

Created on 5 Mar 2020 · 7Comments · Source: AdoptOpenJDK/openjdk-tests

Describe the bug
There is an option added to the curl command which will retry if the fetch fails. The issue is if the file is created at all on the filesystem, the retry will refuse to overwrite it.
Ex job
https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_linux_Nightly/291/console

00:02:17.821  + ./openjdk-tests/get.sh -s /home/****/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_Nightly -t /home/****/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_Nightly/openjdk-tests -p x86-64_linux -r customized -j 11 -i openj9 -c https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/test-images.tar.gz https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/debug-image.tar.gz --openj9_repo https://github.com/eclipse/openj9.git --openj9_branch master --openj9_sha 9130918dd6cd886679403df0efa8f22870fd814e --tkg_repo https://github.com/AdoptOpenJDK/TKG.git --tkg_branch master
00:02:17.821  TESTDIR: /home/****/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_Nightly/openjdk-tests
00:02:17.821  get jdk binary...
00:02:17.821  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz
00:04:55.438  curl: (56) GnuTLS recv error (-9): A TLS packet with unexpected length was received.
00:04:55.438  curl error code: 56. Sleep 300 secs, then retry 1...
00:09:53.832  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz
00:09:53.832  curl: (23) Failed writing body (0 != 15607)
00:09:53.832  curl error code: 23. Sleep 300 secs, then retry 2...
00:14:50.956  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz
00:14:50.956  curl: (23) Failed writing body (0 != 15607)
00:14:50.956  curl error code: 23. Sleep 300 secs, then retry 3...
00:19:56.649  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz
00:19:56.649  curl: (23) Failed writing body (0 != 15607)
00:19:56.649  curl error code: 23. Sleep 300 secs, then retry 4...
00:24:51.270  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz
00:24:51.270  curl: (23) Failed writing body (0 != 15607)
00:24:51.270  curl error code: 23. Sleep 300 secs, then retry 5...
00:29:48.770  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz
00:29:49.577  curl: (23) Failed writing body (0 != 15607)
00:29:49.577  curl error code: 23
00:29:49.577  Failed to retrieve https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz, exiting. This is what we received of the file and MD5 sum:
00:29:49.577  ls: cannot access 'https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz': No such file or directory
00:29:49.577  md5sum: 'https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/306/OpenJ9-JDK11-x86-64_linux-20200303-030650.tar.gz': No such file or directory

Rerunning a similar command by hand without -s

$ curl -OLJSk  https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_ppc64le_linux_Personal/731/test-images.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Refusing to overwrite test-images.tar.gz: File exists
  0 24.6M    0 15698    0     0  17186      0  0:25:05 --:--:--  0:25:05 17175
curl: (23) Failed writing body (0 != 15698)

Refusing to overwrite test-images.tar.gz: File exists

From what I found from a quick search this is not necessarily easily fixed with another option.
https://unix.stackexchange.com/questions/19608/how-can-i-specify-that-curl-via-command-line-overwrites-a-file-if-it-already-e/19609

To Reproduce
Touch a file with he same name as the one you are about to download.

Expected behavior
In my opinion, since these are build scripts, the behaviour should be to overwrite the file(s).

bug

Source

AdamBrousseau

All 7 comments

fyi @pshipton

AdamBrousseau on 5 Mar 2020

Seems like remove the -J option gives me the desired behaviour.

 -J, --remote-header-name  Use the header-provided filename (H)
 -S, --show-error    Show error. With -s, make curl show errors when they occur
 -s, --silent        Silent mode (don't output anything)
 -O, --remote-name   Write output to a file named as the remote file
 -L, --location      Follow redirects (H)
 -k, --insecure      Allow connections to SSL sites without certs (H)

Related #1425 #1150
I see we no longer use the curl retry and instead use a loop. Could we add a cleanup to the loop? Perhaps in the inner loop after the sleep?

AdamBrousseau on 5 Mar 2020

👍1

Thinking about cases where the url is an api request and does not contain a filename. The header returned contains the filename.
ex

$ curl -OLJSkv  https://api.adoptopenjdk.net/v3/binary/latest/8/ea/aix/ppc64/jdk/openj9/normal/adoptopenjdk
...
< Content-Disposition: attachment; filename=OpenJDK8U-jdk_ppc64_aix_openj9_2020-03-03-01-09.tar.gz
...

AdamBrousseau on 5 Mar 2020

👍1

smlambert on 6 Mar 2020

It would be nice if this could be fixed soon, since lately there are artifactory download failures every night in the nightly testing at the OpenJ9 project.

pshipton on 17 Mar 2020

In the original code, we did use curl retry options (#1099), but the retry did not work in the nightly builds (curl retry did not get triggered due to a different return code?). To work around the issue, we switched to use a loop (#1150).

For a quick fix, we can update the script to remove the file if the URL contains the filename (nightly build URL contains filename).

IMO, for the url that does not contain filename (mentioned in https://github.com/AdoptOpenJDK/openjdk-tests/issues/1667#issuecomment-595323207), I think we should not write the script to figure the filename from the header and delete the file. I feel this is too much hacking, which may cause more problems and it is not needed atm. I think we should investigate the problem and fix it properly.