Upload-artifact: ECONNRESET / A 503 status code has been received, will attempt to retry the upload run failed

[jamtur01]

Describe the bug
Artifact uploads fail.

Version

• [ ] V1
• [X] V2

Environment

• [ ] self-hosted
• [X] Linux
• [ ] Windows
• [ ] Mac

Screenshots

Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #1. Waiting for 5134.365661710544 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #2. Waiting for 11357.338554431655 milliseconds before continuing the upload at offset 0
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #2, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #3. Waiting for 19352.64343273028 milliseconds before continuing the upload at offset 0
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #3, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #4. Waiting for 21939.224307833832 milliseconds before continuing the upload at offset 0
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #4, continuing with upload
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (4.8%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (9.7%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (14.6%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (19.5%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (24.4%)
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #1. Waiting for 6609.559098017793 milliseconds before continuing the upload at offset 41943040
Finished backoff for retry #1, continuing with upload
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #2. Waiting for 13000.143736695301 milliseconds before continuing the upload at offset 41943040
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #2, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #3. Waiting for 13960.07476170299 milliseconds before continuing the upload at offset 41943040
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #3, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #4. Waiting for 20167.756830186514 milliseconds before continuing the upload at offset 41943040
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #4, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #5. Waiting for 30945.872415478836 milliseconds before continuing the upload at offset 41943040
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #5, continuing with upload
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (29.3%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (34.2%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (39.1%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (44.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (48.9%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (53.8%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (58.7%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (63.6%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (68.5%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (73.4%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (78.3%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (83.2%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (88.1%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (93.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Uploading /home/runner/work/vector/vector/target/x86_64-unknown-linux-musl/debug/vector (97.9%)
Total size of all the files uploaded is 171217036 bytes
Error: read ECONNRESET

Run/Repo Url

https://github.com/timberio/vector/runs/1282740454?check_suite_focus=true

How to reproduce
Intermittent.

[nschonni]

Maybe related to https://www.githubstatus.com/incidents/k46g5dn2c4lt

[jamtur01]

Could be - we've seen it happen previously. It's possible the retry code is not overly resilient.

[scottjacobsen]

Run actions/upload-artifact@v2
With the provided path, there will be 9 file(s) uploaded
##### Begin Diagnostic HTTP information #####
Status Code: 503
Status Message: Service Unavailable
Header Information: {
  "cache-control": "no-store",
  "content-length": "228",
  "content-type": "text/html",
  "server": "Microsoft-IIS/10.0",
  "x-msedge-ref": "Ref A: A55C5AFC5CB94F6D87EA4D5D3C9890D3 Ref B: DB3EDGE1214 Ref C: 2020-10-20T20:18:33Z",
  "date": "Tue, 20 Oct 2020 20:18:39 GMT",
  "connection": "close"
}
###### End Diagnostic HTTP information ######
Error: Unable to create a container for the artifact coverage-engines at https://pipelines.actions.githubusercontent.com/1oitshaZUNk3UUOFHei9Y085uSJABzRgBL4u7hLQKBhwLX3GZu/_apis/pipelines/workflows/318302862/artifacts?api-version=6.0-preview

Also getting a 503 on each build. And noticed "server": "Microsoft-IIS/10.0", 😱

[musicEnfanthen]

Same here some minutes ago:

Run actions/upload-artifact@v2
With the provided path, there will be 1 file(s) uploaded
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
Total file count: 1 ---- Processed file #0 (0.0%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 6159.675238993805 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total size of all the files uploaded is 3685 bytes
Error: read ECONNRESET

[mrmundt]

Commenting to say we are also experiencing this with Linux and MacOS.

Run actions/upload-artifact@v2
  with:
    name: build_osx_3.5
    path: codecov.result

    if-no-files-found: warn
  env:
    PYTHONWARNINGS: ignore::UserWarning
    PYTHON_REQUIRED_PKGS: ply six nose coverage

    PYTHON_BASE_PKGS: cython dill ipython networkx openpyxl pathos pint pymysql pyro4 pyyaml sphinx_rtd_theme sympy xlrd wheel python-louvain

    PYTHON_NUMPY_PKGS: numpy scipy pyodbc pandas matplotlib seaborn

    CACHE_VER: v2
    GHA_JOBNAME: osx_3.5
    CURL_HOME: /Users/runner/work/pyomo/pyomo
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.5.10/x64
    PYTHON_EXE: /Users/runner/hostedtoolcache/Python/3.5.10/x64/bin/python
    NOSETESTS: /Users/runner/hostedtoolcache/Python/3.5.10/x64/bin/nosetests
    TPL_DIR: /Users/runner/work/pyomo/pyomo/cache/tpl
    DOWNLOAD_DIR: /Users/runner/work/pyomo/pyomo/cache/download
    PYOMO_CONFIG_DIR: /Users/runner/work/pyomo/pyomo/config
    COVERAGE_RCFILE: /Users/runner/work/pyomo/pyomo/.cover_rc
    COVERAGE_PROCESS_START: /Users/runner/work/pyomo/pyomo/.cover_rc
With the provided path, there will be 1 file(s) uploaded
Total size of all the files uploaded is 5 bytes
Error: read ECONNRESET

[jaffinito]

I am seeing the same on Windows as well. Only started about an hour ago for me, before that it was working fine. Here is a link to the run: https://github.com/jaffinito/newrelic-dotnet-agent/runs/1283433397?check_suite_focus=true

[scottjacobsen]

In case it helps anyone else out - on our builds the upload-artifacts steps are generally not critical to the build and I added
continue-on-error: true
to those steps so they can continue without failing the build.

[chingc]

Been getting more upload errors with the following error:
A 429 status code has been received, will attempt to retry the upload

GitHub status page doesn't seem to be reporting any outages today, but maybe it's just hasn't caught up yet.

[flawedworld]

Hi @konradpabjan this seems to be related to #84 , my workflow is unable to upload files and I get slammed with the exact issue as OP. Would really be handy if this gets resolved. Jenkins is being phased out at my firm and I would seriously appreciate if this gets resolved soon as its a lot more elegant than uploading to S3 (which I suggest people do in the meantime until this gets sorted).

[konradpabjan]

We had an outage when this issue was created. https://github.com/actions/upload-artifact/issues/137#issuecomment-713127764

Internal telemetry also indicates there was a big spike in upload-artifact and download-artifact failures.

This action has a lot of built-in retries with exponential back-offs however due to the incident HTTP calls were being dropped significantly on our end so any retries on the client side (upload-artifact action) were futile.

With regards to 429 responses, those are by design. If an individual user is uploading a lot of artifacts then it's a throttling mechanism.

There are similar issues reporting ECONRESET so I'm closing this issue, tracking here: https://github.com/actions/upload-artifact/issues/116