Upload-artifact: Failed to upload Artifact: ECONNRESET and Socket Disconnection

Created on 17 Aug 2020 · 13Comments · Source: actions/upload-artifact

Describe the bug
While try to upload the upload fails with either ECONNRESET or socket disconnected.

Version

[ ] V1
[x] V2

Environment

[ ] self-hosted
[x] Linux
[ ] Windows
[ ] Mac

Run actions/[email protected]
  with:
    name: undionly.kpxe
    path: ipxe/src/bin/undionly.kpxe
    if-no-files-found: error
  env:
    GITHUB_TOKEN: ***
    filename: undionly.kpxe
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]Search path '/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin/undionly.kpxe'
##[debug]File:/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin/undionly.kpxe was found using the provided searchPath
With the provided path, there will be 1 file(s) uploaded
##[debug]Root artifact directory is /home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin
##[debug]Artifact Url: https://pipelines.actions.githubusercontent.com/bINcheClQHqTZNbVjfeRDuwAx9xssj4r3uOpnOJv4gG2xmi8qY/_apis/pipelines/workflows/212542507/artifacts?api-version=6.0-preview
##[error]Client network socket disconnected before secure TLS connection was established
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Upload bin/undionly.kpxe artifact as undionly.kpxe

Run actions/[email protected]
  with:
    name: ipxe.efi
    path: ipxe/src/bin-x86_64-efi/ipxe.efi
    if-no-files-found: error
  env:
    GITHUB_TOKEN: ***
    filename: ipxe.efi
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]Search path '/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin-x86_64-efi/ipxe.efi'
##[debug]File:/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin-x86_64-efi/ipxe.efi was found using the provided searchPath
With the provided path, there will be 1 file(s) uploaded
##[debug]Root artifact directory is /home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin-x86_64-efi
##[debug]Artifact Url: https://pipelines.actions.githubusercontent.com/bINcheClQHqTZNbVjfeRDuwAx9xssj4r3uOpnOJv4gG2xmi8qY/_apis/pipelines/workflows/212542507/artifacts?api-version=6.0-preview
##[error]read ECONNRESET
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Upload bin-x86_64-efi/ipxe.efi artifact as ipxe.efi

Run/Repo Url
https://github.com/killermoehre/custom-ipxe/runs/995234331?check_suite_focus=true
https://github.com/killermoehre/custom-ipxe/runs/995234353?check_suite_focus=true

How to reproduce
I don't know. I just run the run.

Additional context
It started suddenly with the linked run. I didn't change anything, my files are quite small, too.

bug

Source

killermoehre

👍10

Most helpful comment

Should be good now. Retries with exponential back-off have been added to all HTTP calls so any socket disconnects, timeouts, 503s or ECONNRESET errors should gracefully now retry and the artifact upload should go through (previously only a subset of HTTP calls during upload were retried). Recent updates should significantly improve reliability.

If this issue appears again, please open a new issue

konradpabjan on 4 Jan 2021

🎉1 👍1

All 13 comments

Btw, it works now again. This was probably a problem in the actual GitHub API. But even than shouldn't the action handle this better?

killermoehre on 18 Aug 2020

I have also had this failure, but re-running the jobs fixes it. Flakey builds are annoying!

https://github.com/UAlbertaALTLab/gunaha/actions/runs/214200167
https://github.com/UAlbertaALTLab/gunaha/actions/runs/212474456

eddieantonio on 18 Aug 2020

I _think_ I know what is going on here. During artifact upload, there are effectively 3 internal steps

1 POST call to create the artifact
Multiple concurrent PUT calls to upload everything (depends on the size of the artifact and the number of files)
1 PATCH call to indicate we are done uploading the artifact

Historically, the second step has been the flakiest since this can be hundreds or even thousands of http calls. There is a robust retry mechanism for this part of the upload (multiple retries with exponential backoff if something goes wrong). Now the first POST call (same with the last PATCH call) does not have any retry mechanism and I suspect something is failing along the way and there is no retry so it aborts and the action fails.

The specific section of code is here for those interested: https://github.com/actions/toolkit/blob/main/packages/artifact/src/internal/upload-http-client.ts#L54:L100

konradpabjan on 19 Sep 2020

I am experiencing this ECONNRESET issue as well. I'm trying to upload coverage artifacts. In total there are 92 files but they are all very small and the overall size of the bundle is only a few hundred kb.

dmbarry86 on 16 Nov 2020

I'm also having this problem. I've seen it a couple of times in the past, but suddenly it's become quite frequent today. I'm trying to upload a JSON coverage artifact - the file isn't very big.

EDIT:
This is now happening frequently.

kiancross on 17 Nov 2020

I'm having the same issue: a lot of workflow executions failed in the last 3 days with the following error:

Run actions/upload-artifact@v2
With the provided path, there will be 31 file(s) uploaded
Error: read ECONNRESET

kakuzei on 19 Nov 2020

This is now happening for us frequently in the last couple of days.

peteryule on 20 Nov 2020

👍2

It seems to me that it would be possible to reduce the likelihood of those failures by implementing a retry mechanism with exponential backoff:

try upload;
if it fails, print warning message and wait 1s and retry;
if it still fails, print warning message and wait 2s and retry;
if it still fails, print warning message and wait 4s and retry;
if it still fails, print warning message and wait 8s and retry;
if it still fails, print warning message and wait 16s and retry;
if it still fails, print error message and abort.

However I have never coded in TypeScript so I am not confident enough to try to contribute this myself.

ogrisel on 23 Nov 2020

Hello,
We are still encountering the upload artifact issue during the execution of our workflows:

2020-11-30T16:14:03.8299997Z ##[group]Run actions/upload-artifact@v2
2020-11-30T16:14:03.8300677Z with:
2020-11-30T16:14:03.8301354Z   name: ...
2020-11-30T16:14:03.8302112Z   path: ...
2020-11-30T16:14:03.8302824Z   if-no-files-found: warn
2020-11-30T16:14:03.8303422Z env:
...
2020-11-30T16:14:03.8429868Z   ACTIONS_STEP_DEBUG: true
2020-11-30T16:14:03.8430458Z ##[endgroup]
2020-11-30T16:14:03.9223335Z With the provided path, there will be xxx file(s) uploaded
2020-11-30T16:14:05.6239533Z Total size of all the files uploaded is xxx bytes
2020-11-30T16:14:05.6390587Z ##[error]read ECONNRESET

Do you understand why I don't have any debug messages in the logs even if I set the environment variable ACTIONS_STEP_DEBUG to true ?

kakuzei on 1 Dec 2020

Just adding a note here to say that I too ran into this several times this morning, paired with intermittent occurrences of https://github.com/actions/download-artifact/issues/55. I'm curious if the telemetry mentioned over in https://github.com/actions/upload-artifact/issues/137#issuecomment-726914916 might be showing a trend that matches the reports in this issue.