Dpl: github releases - 502 errors and feature "max-upload-tries"

[jakoch]

Hey!

I'm using file_glob to upload multiple build assets to github releases, but this fails often with 502 - Error uploading asset.

Deploying application
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-3.8.0/lib/octokit/response/raise_error.rb:16:in `on_complete': 
POST https://uploads.github.com/repos/WPN-XM/WPN-XM/releases/1027211/assets?name=WPNXM-0.8.1-Standard-Setup-php56-w64.exe: 
502 - Error uploading asset (Octokit::BadGateway)

My current workaround is to delete the partly uploaded file via the interface (Github "Delete and try uploading this file again.") and then restart the build, in order to get the other files uploaded.
This works, because of the already_exists checks on tag and files, so only the missing file are uploaded. (https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/releases.rb#L110)

My feature request is the addition of something like max-upload-tries=x (or upload-retries=x),
where the raised error is catched and the upload would be tried for a number of times.
This would make the deployment more stable in case of "short lived" connection problems.

Regards, Jens

[kenorb]

Same here (119627600). One re-try worked, another failed on another error:

Deploying application
No stash found.
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/client/releases.rb:86:in `initialize': No such file or directory - EURUSD1_0.fxt.gz (Errno::ENOENT)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/client/releases.rb:86:in `new'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/client/releases.rb:86:in `upload_asset'

and another one:

Deploying application
EURUSD10080_0.fxt.gz already exists, overwriting.
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/response/raise_error.rb:16:in `on_complete': POST https://uploads.github.com/repos/FX31337/FX-BT-Data-EURUSD-DS/releases/2921050/assets?name=EURUSD10080_0.fxt.gz: 502 - Error uploading asset (Octokit::BadGateway)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/faraday-0.9.2/lib/faraday/response.rb:9:in `block in call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/faraday-0.9.2/lib/faraday/response.rb:57:in `on_complete'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/faraday-0.9.2/lib/faraday/response.rb:8:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/middleware/follow_redirects.rb:61:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/faraday-0.9.2/lib/faraday/connection.rb:177:in `post'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/sawyer-0.7.0/lib/sawyer/agent.rb:94:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/connection.rb:154:in `request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/client/releases.rb:94:in `upload_asset'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.14/lib/dpl/provider/releases.rb:130:in `upload_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.14/lib/dpl/provider/releases.rb:115:in `block in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.14/lib/dpl/provider/releases.rb:102:in `each'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.14/lib/dpl/provider/releases.rb:102:in `push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.14/lib/dpl/provider.rb:145:in `block in deploy'

After re-trying again then both works.

This happens quiet often, especially for files ~80-100MB in size.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[kenorb]

I think this is still the case.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[BanzaiMan]

@kenorb Do you have a recent failure?

[kenorb]

Won't be able to test it anytime soon. Will try when I can. If you think it should be fixed, feel free to close.

[cgiroua]

@BanzaiMan ... unfortunately I just bumped into this today, our artifact in this case is ~75MB. This is the first time I've seen this failure, it has run successfully dozens of times (or hundreds if you count parallel jobs) ...

https://travis-ci.org/open-horizon/horizon-deb-packager/jobs/445846377
Pasting the error as we rebuilt ^^ successfully ...
```
/home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/response/raise_error.rb:16:in 'on_complete': POST https://uploads.github.com/repos/open-horizon/horizon-deb-packager/releases/13640553/assets?name=v2.19.0.ubuntu.yakkety.assets.tar.gz: 502 - Error uploading asset (Octokit::BadGateway)
from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.15.3/lib/faraday/response.rb:9:in 'block in call'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.15.3/lib/faraday/response.rb:61:in 'on_complete'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.15.3/lib/faraday/response.rb:8:in 'call'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/middleware/follow_redirects.rb:73:in 'perform_with_redirection'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/middleware/follow_redirects.rb:61:in 'call'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.15.3/lib/faraday/rack_builder.rb:143:in 'build_response'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.15.3/lib/faraday/connection.rb:387:in 'run_request'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.15.3/lib/faraday/connection.rb:175:in 'post'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/sawyer-0.8.1/lib/sawyer/agent.rb:94:in 'call'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/connection.rb:154:in 'request'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/client/releases.rb:94:in 'upload_asset'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.10.3/lib/dpl/provider/releases.rb:144:in 'upload_file'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.10.3/lib/dpl/provider/releases.rb:125:in 'block in push_app'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.10.3/lib/dpl/provider/releases.rb:115:in 'each'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.10.3/lib/dpl/provider/releases.rb:115:in 'push_app'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.10.3/lib/dpl/provider.rb:196:in 'block in deploy'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.10.3/lib/dpl/cli.rb:41:in 'fold'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.10.3/lib/dpl/provider.rb:196:in 'deploy'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.10.3/lib/dpl/cli.rb:32:in 'run'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.10.3/lib/dpl/cli.rb:7:in 'run'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.10.3/bin/dpl:5:in ' from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in 'load'
from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in '

'
failed to deploy

[BanzaiMan]

We can add some logic to retry, but beyond that, I am not sure what else we can do here.

[matyasselmeci]

Since this seems to be a random failure, I think retry logic would be fine.

[iriberri]

I did some tests myself in the past with this and the retry logic did help a lot - adding this would be great indeed :)

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[kenorb]

Yes, got another one today in build 513636620, the files aren't big.

/home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/openssl/buffering.rb:325:in `syswrite': Broken pipe (Faraday::ConnectionFailed)
    from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/openssl/buffering.rb:325:in `do_write'
    from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/openssl/buffering.rb:343:in `write'
    from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/protocol.rb:229:in `write0'
    from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/protocol.rb:203:in `block in write'
    from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/protocol.rb:220:in `writing'
...
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/octokit-4.6.2/lib/octokit/client/releases.rb:94:in `upload_asset'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-releases-1.10.8/lib/dpl/provider/releases.rb:163:in `upload_file'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-releases-1.10.8/lib/dpl/provider/releases.rb:136:in `block in push_app'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-releases-1.10.8/lib/dpl/provider/releases.rb:119:in `each'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-releases-1.10.8/lib/dpl/provider/releases.rb:119:in `push_app'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.8/lib/dpl/provider.rb:199:in `block in deploy'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.8/lib/dpl/cli.rb:41:in `fold'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.8/lib/dpl/provider.rb:199:in `deploy'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.8/lib/dpl/cli.rb:32:in `run'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.8/lib/dpl/cli.rb:7:in `run'
    from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.8/bin/dpl:5:in `<top (required)>'
    from /home/travis/.rvm/gems/ruby-2.4.1/bin/dpl:23:in `load'
    from /home/travis/.rvm/gems/ruby-2.4.1/bin/dpl:23:in `<main>'

I'll just re-try.