dpl 1.9.4: uninitialized constant Aws::S3::Resource (NameError)

[kennytm]

We are having deployment failures on Travis starting from dpl 1.9.2 released several hours ago, and the error is still happening as of 1.9.4.

Error logs:

• https://travis-ci.org/rust-lang/rust/jobs/355835936 (1.9.2, errored at 2018-03-20T14:22:12Z)

  Deploying application
  uploading "deploy/b405b3e7b2a49128a24cc50f7bb02c4129efd984/rust-docs-nightly-x86_64-unknown-cloudabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "deploy/b405b3e7b2a49128a24cc50f7bb02c4129efd984/rust-std-nightly-aarch64-unknown-fuchsia.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "deploy/b405b3e7b2a49128a24cc50f7bb02c4129efd984/rust-docs-nightly-x86_64-unknown-linux-gnux32.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}uploading "deploy/b405b3e7b2a49128a24cc50f7bb02c4129efd984/rust-analysis-nightly-sparcv9-sun-solaris.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  
  /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-s3-1.9.2/lib/dpl/provider/s3.rb:108:in `rescue in deploy': uninitialized constant Aws::S3::Errors (NameError)
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-s3-1.9.2/lib/dpl/provider/s3.rb:107:in `deploy'
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.2/lib/dpl/cli.rb:32:in `run'
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.2/lib/dpl/cli.rb:7:in `run'
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.2/bin/dpl:5:in `<top (required)>'
      from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `load'
      from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `<main>'
  

• https://travis-ci.org/rust-lang/rust/jobs/355875866 (1.9.3, errored at 2018-03-20T15:54:10Z)

  Deploying application
  uploading "deploy/7708b71325795829bae09293e76ace0298f0ccca/rust-docs-nightly-x86_64-unknown-redox.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "deploy/7708b71325795829bae09293e76ace0298f0ccca/rust-std-nightly-aarch64-unknown-linux-musl.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "deploy/7708b71325795829bae09293e76ace0298f0ccca/rust-docs-nightly-armv5te-unknown-linux-gnueabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "deploy/7708b71325795829bae09293e76ace0298f0ccca/rust-docs-nightly-wasm32-unknown-emscripten.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-s3-1.9.3/lib/dpl/provider/s3.rb:108:in `rescue in deploy': uninitialized constant Aws::S3::Errors (NameError)
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-s3-1.9.3/lib/dpl/provider/s3.rb:107:in `deploy'
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.3/lib/dpl/cli.rb:32:in `run'
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.3/lib/dpl/cli.rb:7:in `run'
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.3/bin/dpl:5:in `<top (required)>'
      from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `load'
      from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `<main>'
  

• https://travis-ci.org/rust-lang/rust/jobs/355919748 (1.9.4, errored at 2018-03-20T17:14:29Z)

  Deploying application
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-x86_64-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-aarch64-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-aarch64-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-aarch64-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-arm-linux-androideabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-arm-linux-androideabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-armv7-linux-androideabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-armv7-linux-androideabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-aarch64-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-i686-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-x86_64-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-aarch64-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-arm-linux-androideabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-i686-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-armv7-linux-androideabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-arm-linux-androideabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-i686-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-x86_64-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-arm-linux-androideabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-arm-linux-androideabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-aarch64-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-std-nightly-x86_64-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-armv7-linux-androideabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-x86_64-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-i686-linux-android.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-x86_64-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-analysis-nightly-i686-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-armv7-linux-androideabi.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-i686-linux-android.tar.gz" with {:content_type=>"application/gzip", :acl=>"public-read"}
  uploading "eb725bb0c14f55f8fb1f0b5bd3ceae0c769209f4/rust-docs-nightly-armv7-linux-androideabi.tar.xz" with {:content_type=>"application/x-xz", :acl=>"public-read"}
  /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-s3-1.9.4/lib/dpl/provider/s3.rb:12:in `api': uninitialized constant Aws::S3::Resource (NameError)
      from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-s3-1.9.4/lib/dpl/provider/s3.rb:97:in `block (2 levels) in upload_multithreaded'
  

Relevant part of our .travis.yml:

language: shell
sudo: required
dist: trusty

...

deploy:
  - provider: s3
    bucket: rust-lang-ci2
    skip_cleanup: true
    local_dir: deploy
    upload_dir: rustc-builds
    acl: public_read
    region: us-west-1
    access_key_id: <snip>
    secret_access_key:
      secure: <snip>
    on:
      branch: auto
      condition: $DEPLOY = 1

Ruby versions:

rvm version
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]

[BanzaiMan]

Thanks for the report. Is any of these files shown in https://travis-ci.org/rust-lang/rust/jobs/355919748 actually uploaded?

[kennytm]

@BanzaiMan Yes, everything lists there are uploaded.

[BanzaiMan]

That is quite bizarre. That means dpl did initialize the api object using Aws::S3::Resource, but somehow it was forgotten.

[BanzaiMan]

How big are these files?

Maybe multithreading is not working well for this repo?

Could you try reducing the number of threads to use for uploading?

deploy:
  provider: s3
  max_threads: 2 # default is 5
  ⋮ # rest

[kennytm]

@BanzaiMan

The files range from 366 KB to 27 MB, with an average of 12 MB among the 27 files uploaded. I'll see if reducing the number of threads helps, thanks.

[BanzaiMan]

It could be a multi-threading issue. See, for example: https://aws.amazon.com/blogs/developer/threading-with-the-aws-sdk-for-ruby/

I've added Aws.eager_autoload! to see if that solves the problems generally. Could you also try:

deploy:
  provider: s3
  edge:
    branch: s3-eager-autoload
  ⋮ # rest

Thanks!

[kennytm]

@BanzaiMan Thanks! That sounds more plausible 😊 The job is created at https://travis-ci.org/rust-lang/rust/builds/356018991 and we should get the result in 1 to 2 hours.

[BanzaiMan]

@kennytm Thanks for your help. I've got my finger crossed! 🤞

[BanzaiMan]

One complete deployment: https://travis-ci.org/rust-lang/rust/jobs/356018996#L9945

[BanzaiMan]

There is a blank line in https://travis-ci.org/rust-lang/rust/jobs/356018993#L9975; could someone confirm that the deployment was complete?

[alexcrichton]

@BanzaiMan looks like it did indeed succeed!

[BanzaiMan]

Super! I'll merge #789 and release 1.9.5 tomorrow. Sorry for the troubles today.

[BanzaiMan]

1.9.5 is out.