Dpl: S3 deploy errors with SignatureDoesNotMatch

[tsloughter]

Since Sept 1 our upload to S3 from https://github.com/rebar/rebar3 has begun to fail with SignatureDoesNotMatch error. Failed job can be viewed here https://travis-ci.org/rebar/rebar3/jobs/79769044 and the travis.yml file here: https://github.com/rebar/rebar3/blob/master/.travis.yml

/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.20/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.20/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.20/lib/aws-sdk-core/plugins/param_converter.rb:21:in `call'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.20/lib/seahorse/client/plugins/response_target.rb:21:in `call'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.20/lib/seahorse/client/request.rb:70:in `send_request'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.20/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.20/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.20/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.20/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.20/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.20/lib/aws-sdk-resources/services/s3/object.rb:210:in `upload_file'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider/s3.rb:56:in `block (2 levels) in push_app'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider/s3.rb:48:in `glob'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider/s3.rb:48:in `block in push_app'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider/s3.rb:47:in `chdir'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider/s3.rb:47:in `push_app'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider.rb:143:in `block in deploy'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/cli.rb:41:in `fold'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider.rb:143:in `deploy'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/provider/s3.rb:73:in `deploy'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/cli.rb:32:in `run'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/lib/dpl/cli.rb:7:in `run'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.2/bin/dpl:5:in `<top (required)>'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `load'

    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `<main>'

failed to deploy

[BanzaiMan]

The date corresponds to our switch to AWS SDK v2 (with https://github.com/travis-ci/dpl/pull/246). A similar error (multipart upload error, but the same signature error) has also been reported in https://travis-ci.org/octoblu/gateblu-forever/builds/79543775.

[BanzaiMan]

I'm leaning toward devising a way to revert to v1. (Catch these errors and display helpful message.)

[BanzaiMan]

@tsloughter Could you try adding the following to .travis.yml to test this?

deploy:
  provider: s3
  edge:
    branch: s3-backward-compat
  ⋮ # rest

With this, we first try uploading with SDK v2, and if it fails, retry with v1.

[tsloughter]

Yup, that worked. Thanks.

[BanzaiMan]

Hmm. Looks like aws-sdk-2.1.21 fixed the issue. https://travis-ci.org/rebar/rebar3/jobs/80520689 shows that it uploaded the file with the gem without failing.

[BanzaiMan]

I'm going to close this now, since falling back to SDK v1 is no longer necessary.

[jaitaiwan]

Seem to be having the same error on CentricWebEstate/pieisreal. Has there been some sort of regression?

[BanzaiMan]

I've opened a new issue to track this regression. https://github.com/aws/aws-sdk-ruby/issues/965

[BanzaiMan]

Is this still an issue? I understand that aws-sdk 2.2.0 was released yesterday, and that may have some impact on this issue. The issue above hints at a potential fix with 2.2.0, but I've seen reports of breakage with 2.2.0 as well.

[jaitaiwan]

I think I did try this and had no luck. Its been awhile though

On Sat, 14 Nov 2015 8:13 am Hiro Asari [email protected] wrote:

Is this still an issue? I understand that aws-sdk 2.2.0 was released
yesterday, and that may have some impact on this issue. The issue above
hints at a potential fix with 2.2.0, but I've seen reports of breakage with
2.2.0 as well.

—
Reply to this email directly or view it on GitHub
https://github.com/travis-ci/travis-ci/issues/4776#issuecomment-156573771
.

[devdazed]

I started hitting this issue today. Last week everything was working.

/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method.

My assumption is that the upgrade to 2.2.0 caused the regression.

https://github.com/aws/aws-sdk-ruby/issues/965 seems to validate the assumption as well.

adding

    edge:
      branch: s3-backward-compat

does not work

[BanzaiMan]

@devdazed that branch no longer exists; it was for pinning the gem to 1.x.

[devdazed]

hmm, is there any known workarounds? we currently can not deploy our apps which is kinda frustrating.

[BanzaiMan]

@devdazed I do not have a complete grasp of the problem. However, there is at least one report of successful deployments when aws-sdk is pinned to 2.1.x:

before_deploy:
  - rvm 1.9.3 do gem install aws-sdk -v 2.1.36

[devdazed]

@BanzaiMan thanks, that worked.

[awood45]

@devdazed can you provide a bit more context around the issue you're seeing with 2.2.0? I'd like to investigate that from the SDK side.

[devdazed]

@awood45 what would you like to know? I am using travis-pro. I set the AWS access key and secret using the settings page in Travis-CI.

My .travis.yml looks like this:

language: python
python:
  - "2.7"

sudo: false

addons:
  apt:
    packages:
      - libev-dev

cache:
  apt: true

install:
  - python setup.py install
  - pip install flake8
  - pip install pyaml
  - pip install coveralls

after_success:
  - coveralls

script:
  - make lint
  - make coverage

deploy:
  - provider: releases
    skip_cleanup: true
    file_glob: true
    api_key:
      secure: <snip>
    file: dist/*
    on:
      tags: true
  - provider: s3
    skip_cleanup: true
    file_glob: true
    bucket: <snip>
    local_dir: dist
    on:
      tags: true

Which causes the issue, adding:

before_deploy:
  - rvm 1.9.3 do gem install aws-sdk -v 2.1.36

fixed it.

[awood45]

Probably a question for @BanzaiMan then - I haven't been able to recreate this issue using 2.2.x of the SDK, I'd like to have some more context around what is causing this so that I can reproduce and fix as needed.

[BanzaiMan]

@awood45 I have not been able to reproduce this with my tests, either. All reports have been from paying customers using private repositories, so I can't really see what differences are there to account for the issues.

[awood45]

@BanzaiMan what kind of differences could exist within private repositories that I could reproduce?

So far, I haven't been able to generate S3 upload signature errors with 2.2.x, nor have I heard of any other reports. However, part of the 2.2 release did change how we did signing (making Signature Version 4 a global default) so I am paying extra close attention to any possibilities.

I'd like to do whatever I can to at least pin down the cause.

[BanzaiMan]

@awood45 That is the question! I want a reproduction that I can share with you, so that you can look at what differences may explain this peculiar issue. I wish I had any better idea! :-(

[awood45]

I can see from the stack trace we do have that it is happening in the upload logic. I'm starting by hitting that from every angle I can think of (no failures so far), but if you can share the code context around that, or if you see any other information you can share, please do add it to the issue.

[jaitaiwan]

I'll give another go to pieisreal and see if the same issue is still cropping up...

[devdazed]

here is a full stack trace if it will help:

Deploying application
uploading "share_processor-0.0.0-py2.7.egg"
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/seahorse/client/request.rb:70:in `send_request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.0/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.0/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.0/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.0/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.0/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.0/lib/aws-sdk-resources/services/s3/object.rb:231:in `upload_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider/s3.rb:56:in `block (2 levels) in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider/s3.rb:48:in `glob'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider/s3.rb:48:in `block in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider/s3.rb:47:in `chdir'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider/s3.rb:47:in `push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider.rb:143:in `block in deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/cli.rb:41:in `fold'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider.rb:143:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/provider/s3.rb:73:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/cli.rb:32:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/lib/dpl/cli.rb:7:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.7/bin/dpl:5:in `<top (required)>'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `load'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `<main>'
failed to deploy

outside of this there is little more I can give. I can say that all tests passed when ran. no extra ruby packages were installed (it's a python app). and (from the yml above) the only apt package installed was lebev-dev

here is the list of python packages installed if it helps:

coverage==4.0.2
flake8 >= 2.5.0
mock >= 1.0.0
nose >= 1.3.7
blinker==1.4
cassandra-driver==2.5.0
dogstatsd-python==0.5.6
gevent==1.1b1
gnsq==0.3.0
grequests==0.2.0
kitchen==1.2.1
pycassa==1.11.1
pymongo==3.0.3
redis==2.10.3
requests==2.7.0
simplejson==3.8.0
slumber==0.7.1
blist==1.3.6
pyOpenSSL==0.15.1
ndg-httpsclient==0.4.0
pyasn1==0.1.9
tornado>=4.2
iso8601
inflection

[jaitaiwan]

It seems we have a similar trace to the one above:

home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/request.rb:70:in `send_request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/object.rb:231:in `upload_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:56:in `block (2 levels) in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:48:in `glob'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:48:in `block in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:47:in `chdir'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:47:in `push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider.rb:145:in `block in deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/cli.rb:41:in `fold'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider.rb:145:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:73:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/cli.rb:32:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/cli.rb:7:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/bin/dpl:5:in `<top (required)>'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `load'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `<main>'

[coreyhulen]

our travis file on a private github repo running on travis-ci.com

nothing very complex

language: python
python:
- 2.7
install:
- pip install sphinx sphinx-autobuild
- pip install recommonmark
script:
- make html

deploy:
- provider: s3
  access_key_id: <removed>
  secret_access_key: 
     secure: <removed>
 bucket: docs.mattermost.org
  local_dir: "build/html"
  acl: public_read
  region: us-east-1
  skip_cleanup: true
  detect_encoding: true
  on:
    repo: mattermost/docs
    branch: master
    python: 2.7

Fetching: dpl-1.8.8.gem (100%)
Successfully installed dpl-1.8.8
1 gem installed
dpl.1
Installing deploy dependencies
Fetching: jmespath-1.1.3.gem (100%)
Successfully installed jmespath-1.1.3
Fetching: aws-sdk-core-2.2.1.gem (100%)
Successfully installed aws-sdk-core-2.2.1
Fetching: aws-sdk-resources-2.2.1.gem (100%)
Successfully installed aws-sdk-resources-2.2.1
Fetching: aws-sdk-2.2.1.gem (100%)
Successfully installed aws-sdk-2.2.1
4 gems installed
Fetching: mime-types-2.6.2.gem (100%)
Successfully installed mime-types-2.6.2
1 gem installed
dpl.2
Preparing deploy
Logging in with Access Key: ****************ZDUA
dpl.3
Deploying application
uploading "help/Manage-Members.html"
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/request.rb:70:in `send_request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.2.1/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/object.rb:231:in `upload_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:56:in `block (2 levels) in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:48:in `glob'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:48:in `block in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:47:in `chdir'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:47:in `push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider.rb:145:in `block in deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/cli.rb:41:in `fold'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider.rb:145:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/provider/s3.rb:73:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/cli.rb:32:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/lib/dpl/cli.rb:7:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.8/bin/dpl:5:in `<top (required)>'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `load'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `<main>'
failed to deploy

[coreyhulen]

Our private repo at mattermost/docs will eventually be made public so I don't mind granting someone readonly access to it.

[awood45]

So the question is, what is the DPL gem doing?

I'm unfamiliar, does anyone have a link to that?

I'm also looking at adding some debug logic to a future release that would show more information about signature errors.

[awood45]

Nevermind, found that code and looking in to it.

[awood45]

What I can see is that all deployments are hard-coded to us-east-1, which has changed signers from the "s3" signer to the "v4" signer, so signature errors shouldn't be happening but a delta exists.

I'm going to work on the change that gives you additional signer information, using this code would allow us to get a more useful debug output that we could use to investigate possible bugs in the signer implementation.

[BanzaiMan]

@awood45 You can see the implementation here: https://github.com/travis-ci/dpl/blob/v1.8.8/lib/dpl/provider/s3.rb#L44-L70

It creates an Aws::S3::Resource object (https://github.com/travis-ci/dpl/blob/v1.8.8/lib/dpl/provider/s3.rb#L10) then upload the files for each file (https://github.com/travis-ci/dpl/blob/v1.8.8/lib/dpl/provider/s3.rb#L56).

[devdazed]

I see @coreyhulen is using Python, we are as well. I know that one of the eggs that we install also uses the cryptography egg which relies/installs some OpenSSL stuff. Maybe one of those packages is interfering with the SSL here?

[coreyhulen]

I finally got it to work. I did a bunch of stuff but in my case I think it might have been user error. I deleted and re-gen'ed all my keys and moved the repo from private to public, but still no luck. The trick was travis encrypt --add deploy.secret_access_key != travis encrypt secret_access_key=secretvalue + copy. I had setup a similar deploy several months back so looking at that file I just googled travis secure and was taken to http://docs.travis-ci.com/user/encryption-keys/, to which I was like, this is what I must have done to generate the secure: field since I couldn't remember. When I ran the travis encrypt command it prints out something that looks like I must have copied into the deploy section of the other script so I just replaced the secure section. I randomly ran across http://docs.travis-ci.com/user/deployment/s3/ which gave me the correct encrypt command.

A better error from travis would be nice, or maybe some validation check with travis lint which said everything was fine.

[awood45]

This is working with 2.2.x? Curious what you feel the issue was, I didn't quite follow the changes here.

[devdazed]

I'm not convinced this is entirely the case. We aren't encrypting our keys in the .travis.yml. We are using the environment variables in the project settings in Travis-CI. Everything was working without any changes until the upgrade.

[coreyhulen]

I'm totally guessing here that the travis encrypt command salts/hashes in the entire key name which should have been deploy.secret_access_key vs just secret_access_key notice one has deploy which jacks up the decrypt and in turn the real secret access key.

OR ....I'm just really lucky :)

[coreyhulen]

Sorry maybe it was the move from the private repo to a public repo and the travis encrypt stuff is just a red herring.

Here is the successful build log and it looks almost identical. It started working after I did the travis encrypt cmd, but maybe I just got lucky. I would rather be lucky than good :)

Fetching: dpl-1.8.8.gem (100%)
Successfully installed dpl-1.8.8
1 gem installed
11.15s
dpl.1
Installing deploy dependencies
Fetching: jmespath-1.1.3.gem (100%)
Successfully installed jmespath-1.1.3
Fetching: aws-sdk-core-2.2.1.gem (100%)
Successfully installed aws-sdk-core-2.2.1
Fetching: aws-sdk-resources-2.2.1.gem (100%)
Successfully installed aws-sdk-resources-2.2.1
Fetching: aws-sdk-2.2.1.gem (100%)
Successfully installed aws-sdk-2.2.1
4 gems installed
Fetching: mime-types-2.6.2.gem (100%)
Successfully installed mime-types-2.6.2
1 gem installed
dpl.2
Preparing deploy
Logging in with Access Key: ****************ZP4Q
dpl.3
Deploying application
uploading "help/Manage-Members.html"
uploading "help/Messaging.html"
uploading "help/Search.html"
uploading "help/Sign-in.html"
uploading "help/Team-Settings.html"
uploading "help/system-console/Team-Statistics.html"
uploading "_sources/help/Manage-Members.txt"
uploading "_sources/help/Messaging.txt"
uploading "_sources/help/Search.txt"
uploading "_sources/help/Sign-in.txt"
uploading "_sources/help/Team-Settings.txt"
uploading "_sources/help/system-console/Team-Statistics.txt"
uploading "_sources/index.txt"
uploading "_sources/install/Administration.txt"
uploading "_sources/install/Amazon-Elastic-Beanstalk.txt"
uploading "_sources/install/Configuration-Settings.txt"
uploading "_sources/install/Docker-Single-Container.txt"
uploading "_sources/install/Production-Debian.txt"
uploading "_sources/install/Production-Ubuntu.txt"
uploading "_sources/install/Release-Numbering.txt"
uploading "_sources/install/Requirements.txt"
uploading "_sources/install/SMTP-Email-Setup.txt"
uploading "_sources/install/Troubleshooting.txt"
uploading "_sources/install/Upgrade-Guide.txt"
uploading "_sources/integrations/Single-Sign-On/Gitlab.txt"
uploading "_sources/integrations/services/Gitlab-Integration-Service-for-Mattermost.txt"
uploading "_sources/integrations/webhooks/Incoming-Webhooks.txt"
uploading "_sources/integrations/webhooks/Outgoing-Webhooks.txt"
uploading "_sources/process/documentation-guidelines.txt"
uploading "_sources/process/release-process.txt"
uploading "_sources/usage/Markdown.txt"
uploading "index.html"
uploading "install/Administration.html"
uploading "install/Amazon-Elastic-Beanstalk.html"
uploading "install/Configuration-Settings.html"
uploading "install/Docker-Single-Container.html"
uploading "install/Production-Debian.html"
uploading "install/Production-Ubuntu.html"
uploading "install/Release-Numbering.html"
uploading "install/Requirements.html"
uploading "install/SMTP-Email-Setup.html"
uploading "install/Troubleshooting.html"
uploading "install/Upgrade-Guide.html"
uploading "integrations/Single-Sign-On/Gitlab.html"
uploading "integrations/services/Gitlab-Integration-Service-for-Mattermost.html"
uploading "integrations/webhooks/Incoming-Webhooks.html"
uploading "integrations/webhooks/Outgoing-Webhooks.html"
uploading "process/documentation-guidelines.html"
uploading "process/release-process.html"
uploading "usage/Markdown.html"
uploading "genindex.html"
uploading "search.html"
uploading "_static/pygments.css"
uploading "_static/jquery.js"
uploading "_static/jquery-1.11.1.js"
uploading "_static/underscore.js"
uploading "_static/file.png"
uploading "_static/down-pressed.png"
uploading "_static/comment.png"
uploading "_static/doctools.js"
uploading "_static/basic.css"
uploading "_static/comment-bright.png"
uploading "_static/ajax-loader.gif"
uploading "_static/plus.png"
uploading "_static/websupport.js"
uploading "_static/searchtools.js"
uploading "_static/underscore-1.3.1.js"
uploading "_static/up-pressed.png"
uploading "_static/comment-close.png"
uploading "_static/down.png"
uploading "_static/minus.png"
uploading "_static/up.png"
uploading "_static/alabaster.css"
uploading "searchindex.js"
uploading "objects.inv"
Done. Your build exited with 0.

[BanzaiMan]

Hello! Is aws-sdk 2.2.2 or 2.2.3 helping you?

[tsloughter]

Mines now broken again but with a different error: https://travis-ci.org/rebar/rebar3/jobs/93442097#L619

/home/travis/.rvm/rubies/ruby-1.9.3-p551/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:inrequire': cannot load such file -- mime-types (LoadError)`

[BanzaiMan]

@tsloughter Sorry, but that's an entirely different issue, a variation of #5145, which is fixed by a recent dpl release. Please drop https://github.com/rebar/rebar3/blob/901620fb603d98b7c030e30bb43f63a8a5a6ed08/.travis.yml#L23-L24.

[tsloughter]

Thanks @BanzaiMan, it works again.

[jesusoterogomez]

I am still getting an error even when applying

before_deploy:
  - rvm 1.9.3 do gem install aws-sdk -v 2.1.36

I also tried regenerating the encrypted secrets, though I haven't tried using the secret without encryption.
Is there anything else that could be causing this error?.

If you need some additional information, don't hesitate to let me know

Here's the log of the error.

/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.36/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.36/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.36/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.36/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.36/lib/seahorse/client/request.rb:70:in `send_request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.36/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.36/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.36/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.36/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.36/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.1.36/lib/aws-sdk-resources/services/s3/object.rb:231:in `upload_file'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider/s3.rb:56:in `block (2 levels) in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider/s3.rb:48:in `glob'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider/s3.rb:48:in `block in push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider/s3.rb:47:in `chdir'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider/s3.rb:47:in `push_app'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider.rb:145:in `block in deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/cli.rb:41:in `fold'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider.rb:145:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/provider/s3.rb:73:in `deploy'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/cli.rb:32:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/lib/dpl/cli.rb:7:in `run'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/dpl-1.8.11/bin/dpl:5:in `<top (required)>'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `load'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/bin/dpl:23:in `<main>'
failed to deploy

[BanzaiMan]

@jesusoterogomez Could you point to a build log URL that shows your problem?

[BanzaiMan]

Is this still an issue? Is the recent aws-sdk 2.2.7 working any better?

[jesusoterogomez]

@BanzaiMan sorry for not responding sooner, I somehow had my notifications turned off, here is the build log URL, https://travis-ci.com/Penneo/penneo-fe-react/builds/18697235, but I managed to get it working ever since. the current aws-sdk default doesn't show this issue either.

I think I had a problem with the key encryption. I used the travis CLI tool to encrypt the keys while I was on my fork repository. However, I did specify the main repository as an argument with --repo (https://github.com/travis-ci/travis.rb#encrypt).

It worked when I cloned my main repository, and retried the encryption, letting the CLI detect the settings.

So I might have either been doing something entirely wrong (:disappointed: sorry), or the CLI might be targeting the current clone and ignoring the --repo argument (unlikely).

[BanzaiMan]

Hi, everyone. Is anyone seeing this issue recently?

[jaitaiwan]

I redid the encryption key and things are working well now

On Thu, 4 Feb 2016 06:34 Hiro Asari [email protected] wrote:

Hi, everyone. Is anyone seeing this issue recently?

—
Reply to this email directly or view it on GitHub
https://github.com/travis-ci/travis-ci/issues/4776#issuecomment-179449215
.

[dougnukem]

I'm still seeing the issue, but I did the fix to pin the aws-sdk -v 2.1.36 and that fixed it:

before_deploy:
  # Fix s3 deploy issues https://github.com/travis-ci/travis-ci/issues/4776
  - rvm 1.9.3 do gem install aws-sdk -v 2.1.36

[shmendo]

For people who may have a similar problem, try using the setup command to override your deploy section.

travis setup s3 --force

I've noticed that when I encrypt the key myself using travis encrypt secret_access_key'<removed>, it never works. If I use travis setup s3 --force and provide the requested information it does work.

And it's definitely not user error. Myself and 4 other people spent a considerable amount of time testing this :(

[dougnukem]

I tested recently and am still noticing this issue which requires the manual fix https://github.com/travis-ci/travis-ci/issues/4776#issuecomment-180847210.

I've tried @shmendo advice to run manually:
travis setup s3 --force

but it still was an issue.

[BanzaiMan]

I'm sorry that this is cropping up every now and then. How are things looking with the latest version of aws-sdk (2.2.37)?

[dbernstein]

Hi BanzaiMan:
I'm trying out the Elastic Beanstalk provider. I'm running into what looks like the same issue:

travis_fold:end:dpl.3
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise': The request signature we calculated does not match the signature you provided. Check your key and signing method. (AWS::S3::Errors::SignatureDoesNotMatch)

Looking at the master branch it looks like the elastic beanstalk provider is using aws sdk v1 which is consistent with what I'm seeing in the above log line. I'm not well versed in ruby. Should adding

before_deploy:
  - rvm 1.9.3 do gem install aws-sdk -v 2.2.37

to my .travis.yml fix Elastic Beanstalk as well?

[svenlaater]

I was also encountering this issue but it seems that in my case the problem was in defining the bucket w/ s3:// prefix e.g. bucket: s3://my-bucket-name. This resulted in Aws::S3::Errors::SignatureDoesNotMatch error with aws-sdk 2.x
After changing it to bucket: my-bucket-name I got rid of the problem.

What led me to the root of the problem is the response upon sdk1 fallback. I hope this helps.

uploading "blaah.apk"
Encountered an error while uploading with AWS SDK v2. Retrying with v1. See https://github.com/travis-ci/travis-ci/issues/4776 for details.
...
Successfully installed aws-sdk-v1-1.66.0
...
uploading "blaaah.apk"
....
/Users/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise': The specified bucket does not exist (AWS::S3::Errors::NoSuchBucket)

[dbernstein]

In my case, I believe the upload to S3 is done under the covers by the beanstalk cli.. Were you using the elasticbeanstalk provider? If so did you need to specify a bucket? Perhaps that's what I'm missing.

[dbernstein]

Update: I tried setting before_deploy to use the v2 aws-sdk (2.2.37) - no luck. elastic_beanstalk provider still uses the v1 sdk. (same error as before).

[BanzaiMan]

@dbernstein Beanstalk and OpsWorks providers still rely on SDK v1. We have a plan to move them to v2, but no ETA yet.

[dbernstein]

@BanzaiMan: So is there any workaround or are you saying it is broken for the time being? Just trying to understand the status.

[BanzaiMan]

@dbernstein It is not clear to me. I've seen reports of this error (this thread exists because it does happen!), but I have never been able to figure out how to reproduce it. Various versions of the aws-sdk library have been said to be more reliable than others, but I am not sure which versions they are.

If we can nail down the issue, we can help @awood45 fix it for good.

[BanzaiMan]

Hi, everyone. I still do not have a solid explanation of why this pesky intermittent issue is happening, but one possible explanation is that the clock on our system might be somehow far off from that of the S3 servers.

If you are experiencing this issue, I want to know:

1. Is it intermittent or persistent?
2. Do you run your builds on containers or on GCE (sudo: required)?

[DarynHolmes]

Unfortunately I am getting this error on a private repo. It is persistent, and I run the builds on containers.

This did not work:

before_deploy:
  - rvm 1.9.3 do gem install aws-sdk -v 2.1.36

Neither did this:

before_deploy:
  - rvm 1.9.3 do gem install aws-sdk -v 2.2.37

Here is part of the error message:

/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/aws-sdk-core/plugins/s3_dualstack.rb:23:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/aws-sdk-core/plugins/s3_accelerate.rb:33:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/seahorse/client/request.rb:70:in `send_request'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.5.7/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-resources-2.5.7/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'

[ghost]

I think this may be a compound issue depending on your setup, with AWS with no fix-all solution.

We had this issue building for both iOS and Android. It was building our Android build a week ago fine.

We had tried changing the version of the AWS SDK and it always fell through to the backwards compatibility v1 of the SDK which then failed.

We did get it working by encrypting the keys using:

travis encrypt XXXXXXXXXXXXXXX --add deploy.secret_access_key

instead of:

travis encrypt secret_access_key=XXXXXXXXXXXX --add deploy.secret_access_key

[ricard0javier]

In my case I got it working setting the two environment variables in travis for my repository: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. and this is how my .travis.xml file looks like:

language: node_js

node_js:
- '6'

script: npm run build

deploy:
  provider: s3
  access_key_id: $AWS_ACCESS_KEY_ID
  secret_access_key: $AWS_SECRET_ACCESS_KEY
  bucket: dev.1989generationinitiative.org
  skip_cleanup: true
  region: eu-central-1
  local_dir: dist
  cache_control: max-age=604800

cache:
  yarn: true
  directories:
  - node_modules

[Andrei-Pozolotin]

this works for us:

travis setup s3 --force

[steveswinsburg]

travis setup s3 --force is the only way it has worked for me also. I was using the travis encrypt commandline tools to do it and none of the encrypted keys were working.

[tomschenkjr]

Experienced the same error. We were re-using keys between two separate projects for S3 deployment. Created new keys with the same permissions in IAM and the error disappeared.

[floric]

Ah, I stumbled across this problem and spent hours with no progress at all...

I found one advise in #602: if the generated secret key contains "+" or "/" characters, this error will occur. Simply regenerating keys should have solved it, but it didn't.

In the end I added the keys as env variables directly in Travis. This was the only way that worked.

[CyanoKobalamyne]

I stumbled upon this today while trying to set up automatic deployment to AWS but nothing worked. I tried regenerating keys, adding them manually and via the CLI, also adding them as environment variables but I keep getting this error.

[InnovativeInventor]

I'm getting this problem too (thought I fixed it, but it isn't working). Also, the generated secret key does not contain "+" or "/". This seems like a pretty important issue.

[cmelendez]

Moving AWS's credential to environment variables fixed it for us.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[wongm]

Finally solved this problem for my deploy.

Had to do this:

travis encrypt XXXXXXXXXXXXXXX

instead of:

travis encrypt secret_access_key=XXXXXXXXXXXX

This is different to what the documentation says:

https://docs.travis-ci.com/user/environment-variables/#encrypting-environment-variables

[thaniri]

I ran into this problem. Below is my fix. Hopefully you too can save ~45 minutes of your time.

What ended up working for me was to generate a new access key without a "+" character in it. Then I went into my repository settings in the travis web UI and added an environment variable with my access key. I ended up hardcoding the key ID into my .travis.yml file.

Here is the deploy section of my .travis.yml file:

deploy:
  provider: s3
  access_key_id: <hidden>
  secret_access_key: $SECRET_ACCESS_KEY #this is an environment variable I made in the settings of my repository on the travis website
  bucket: <hidden>
  local-dir: public
  skip_cleanup: true
  region: us-east-1