Dpl: CodeDeploy provider "Aws::S3::Errors::Forbidden" error with dpl 1.8.46 but not dpl 1.8.45

[AdrienAgnel]

Hello,

I have been experiencing deployment issues since yesterday and it seems to me it is related to the upgrade from dpl 1.8.45 to 1.8.46.

I get the following error in my Travis builds :

It occurs during a deployment with CodeDeploy provider.

I managed to reproduce the bug locally, using the command detailed here : https://github.com/travis-ci/dpl#examples-32 (the 'key' option appears to be missing in the doc)

• first step, install dpl 1.8.46 and manually run the deployment command for CodeDeploy provider
  
  
  
  • the command fails with the same error stack as above
  
  
• second step, install dpl 1.8.45 and manually run the exact same deployment command for CodeDeploy provider
  
  
  
  • the command is successful and a deployment is correctly created
  
  

I have no experience with debugging ruby gems, do you have any idea of what is happening ?

[BanzaiMan]

It is complaining about the credentials when we call #head_object on S3: https://github.com/travis-ci/dpl/blob/af2c5c6fb330011731937e5c019eee36f7d95f92/lib/dpl/provider/code_deploy.rb#L47 Perhaps your key does not have enough authorization for this? And if so, our code should take this possibility into account.

In the mean time, you can force deployment with version 1.8.45:

deploy:
  provider: code_deploy
  edge:
    branch: v1.8.45
  ⋮ # rest

[effata]

This change broke all our travis deployments, which is very unfortunate. Is there an estimate on when this can be fixed? Since you can't give HeadObject without giving GetObject on S3, a user must have access to download all objects being deployed to CodeDeploy, which I would argue should not be a necessary dependency.
Thank you for the instruction for a work around, even though I'm not looking forward to manually updating all repositories we're deploying through travis.

[BanzaiMan]

@effata Understood. I'll work on a fix.

[brdmr]

Just a note, the example needs to be prefixed with a v to work. I.e. branch: v1.8.45.
Thanks for the quick workaround and fast response time. 👍

[BanzaiMan]

@brdmr Indeed. I edited the comment above. Thank you.

[AdrienAgnel]

Thanks for the tip about the edge property and for working on a fix, it's actually better if we can avoid giving new rights to the user triggering CodeDeploy.

[BanzaiMan]

Pushed a potential fix to ignore the exception and work as before. I appreciate your testing it:

deploy:
  provider: code_deploy
  edge:
    branch: code_deploy-head_object
  ⋮ # rest

Thank you.

[BanzaiMan]

@jakubholynet Could you also test the above, as your repository also uses the code. Thanks!

[AdrienAgnel]

Unfortunately it appears to be still failing :

I just modified my .travis.yml file as follows :

  - provider: codedeploy
    edge:
      branch: code_deploy-head_object

[BanzaiMan]

@AdrienAgnel Thanks for testing. It was rescueing a wrong class. Could you retry (restarting this failed job should suffice)?

[AdrienAgnel]

@BanzaiMan it's working :+1: thanks !

[BanzaiMan]

@AdrienAgnel Super!

Could someone else also confirm?

[effata]

@BanzaiMan We're seeing the following error in Travis now:

Building dpl gem locally with source travis-ci/dpl and branch code_deploy-head_object

$ git clone https://github.com/travis-ci/dpl travis-ci/dpl
Cloning into 'travis-ci/dpl'...

$ git checkout code_deploy-head_object
Switched to a new branch 'code_deploy-head_object'
Branch 'code_deploy-head_object' set up to track remote branch 'code_deploy-head_object' from 'origin'.
$ git show-ref -s HEAD
af2c5c6fb330011731937e5c019eee36f7d95f92
$ rvm $(travis_internal_ruby) --fuzzy do ruby -S gem build dpl.gemspec
WARNING:  pessimistic dependency on rspec (~> 3.0.0, development) may be overly strict
  if rspec is semantically versioned, use:
    add_development_dependency 'rspec', '~> 3.0', '>= 3.0.0'
WARNING:  open-ended dependency on rspec-its (>= 0, development) is not recommended
  if rspec-its is semantically versioned, use:
    add_development_dependency 'rspec-its', '~> 0'
WARNING:  open-ended dependency on rake (>= 0, development) is not recommended
  if rake is semantically versioned, use:
    add_development_dependency 'rake', '~> 0'
WARNING:  open-ended dependency on coveralls (>= 0, development) is not recommended
  if coveralls is semantically versioned, use:
    add_development_dependency 'coveralls', '~> 0'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: dpl
  Version: 1.8.47.travis.850.1
  File: dpl-1.8.47.travis.850.1.gem





ERROR:  Could not find a valid gem 'dpl-*.gem' (>= 0) in any repository
The command "rvm $(travis_internal_ruby) --fuzzy do ruby -S gem install dpl-*.gem --local --pre" failed and exited with 2 during .

[BanzaiMan]

@effata Whoa. That is strange. It is saying that it built the gem locally, but it can't find it. I assume that the issue persists if you restart it?

[holyjak]

Regarding permissions, dpl newly needs the Head Object permission on the
archive to be deployed (to fetch its metadata). That is why some builds
could start failing.

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html
ons. 24. jan. 2018 kl. 23:36 skrev Hiro Asari [e-mail]:

@effata https://github.com/effata Whoa. That is strange. It is saying
that it built the gem locally, but it can't file it. I assume that the
issue persists if you restart it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/travis-ci/dpl/issues/741#issuecomment-360297476, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAmJPtOGOWfiARYbhQKThQPQJ9p2ze65ks5tN7BcgaJpZM4RrCBa
.

[holyjak]

I will try to test later today.

Beware that if you lack Head Object, revisions will not be registered with
code deploy and the CD CLI and UI will show nonsense revision information.

If Travis can upload the archives, is it such a problem to allow it to also
read them?

It is very unfortunate you need Read to do Head :-(
tor. 25. jan. 2018 kl. 09:45 skrev Jakub Holý [e-mail]:

Regarding permissions, dpl newly needs the Head Object permission on the
archive to be deployed (to fetch its metadata). That is why some builds
could start failing.

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html
ons. 24. jan. 2018 kl. 23:36 skrev Hiro Asari [e-mail]:

@effata https://github.com/effata Whoa. That is strange. It is saying
that it built the gem locally, but it can't file it. I assume that the
issue persists if you restart it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/travis-ci/dpl/issues/741#issuecomment-360297476, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAmJPtOGOWfiARYbhQKThQPQJ9p2ze65ks5tN7BcgaJpZM4RrCBa
.

[holyjak]

Alternative solutions:

1. Dont register revisions
2. Make the S3 deploy step read the response (eTag, version) and somehow
   make them svailable to the CD step
3. Merge the S3 upload and CD step into one, ie teach CD to upload to S3
   (to achirve #2)
   tor. 25. jan. 2018 kl. 09:56 skrev Jakub Holý [e-mail]:

I will try to test later today.

Beware that if you lack Head Object, revisions will not be registered with
code deploy and the CD CLI and UI will show nonsense revision information.

If Travis can upload the archives, is it such a problem to allow it to
also read them?

It is very unfortunate you need Read to do Head :-(
tor. 25. jan. 2018 kl. 09:45 skrev Jakub Holý [e-mail]:

Regarding permissions, dpl newly needs the Head Object permission on the
archive to be deployed (to fetch its metadata). That is why some builds
could start failing.

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html
ons. 24. jan. 2018 kl. 23:36 skrev Hiro Asari [e-mail]:

@effata https://github.com/effata Whoa. That is strange. It is saying
that it built the gem locally, but it can't file it. I assume that the
issue persists if you restart it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/travis-ci/dpl/issues/741#issuecomment-360297476,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmJPtOGOWfiARYbhQKThQPQJ9p2ze65ks5tN7BcgaJpZM4RrCBa
.

[effata]

@BanzaiMan Yes, the issue persist if we restart the build. The strange thing is it seems to work fine on a ruby 2.2.2 project, but on this 2.3.3 project it breaks.

@jakubholynet It is indeed very unfortunate that you need to give full read permissions. For us it would mean that we would have to grant full read permissions to all our production builds to an external user, which is problematic in itself. I don't feel that concerned about travis itself, but consider the case of someone getting hold of that access key, and being able to read all our deployed code. It would be much worse than just being able to blindly write to the same bucket.
Regarding your alternative solutions, 2 and 3 would not work for us, since we use jara to release our artifacts, and not the travis s3 provider.

[holyjak]

For me the edge fails at data = code_deploy.create_deployment with

Installing deploy dependencies
Logging in with Access Key: ****************XYZZ
/home/travis/.rvm/gems/ruby-2.2.7/gems/aws-sdk-core-2.10.123/lib/aws-sdk-core/param_validator.rb:28:in `validate!': parameter validator found 7 errors: (ArgumentError)
  - unexpected value at params[:revision][:s3_location][:accept_ranges]
  - unexpected value at params[:revision][:s3_location][:last_modified]
  - unexpected value at params[:revision][:s3_location][:content_length]
  - unexpected value at params[:revision][:s3_location][:etag]
  - unexpected value at params[:revision][:s3_location][:version_id]
  - unexpected value at params[:revision][:s3_location][:content_type]
  - unexpected value at params[:revision][:s3_location][:metadata]
    from /home/travis/.rvm/gems/ruby-2.2.7/gems/aws-sdk-core-2.10.123/lib/aws-sdk-core/param_validator.rb:13:in `validate!'
...
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.47.travis.114.1/lib/dpl/provider/code_deploy.rb:89:in `push_app'

Build log: https://travis-ci.com/TeliaSoneraNorge/rapidshop-b2b/builds/64294338

@effata I see :-( Then I guess the fallback to the old behavior is the best solution for you, though you will need to live with meaningless release info in CD :(
Maybe CD could have params etag and version so you could supply them yourself, after somehow getting them from Jarå/AWS?

[BanzaiMan]

@effata Do you have a build log URL you can share?

[effata]

@BanzaiMan I believe we're already talking via the travis support mail? I just emailed you another failing build.

[BanzaiMan]

@effata Ah! Yes. I see it. Thanks a bunch!

[BanzaiMan]

(@effata The pics looked a whole lot different, so I didn't connect the dots. 😆)

[BanzaiMan]

@effata FYI https://github.com/travis-ci/travis-build/pull/1298 resolves the "gem not found" error.

[BanzaiMan]

@effata @jakubholynet I've pushed another commit to the code_deploy-head_object branch that changes the shape of S3-based deployment (which aligns with what @jakubholynet had in v1.8.46).

Could you test it (restarting the build should suffice)?

[holyjak]

It passed now and deployed successfully!

[BanzaiMan]

@jakubholynet Is your setup for a key with head_object privilege or without? Or were you able to test both?

[holyjak]

With the privilege. I can revoke it and rerun when I have time, ie Monday,
unless sb. else can verify it earlier.
lør. 27. jan. 2018 kl. 16:24 skrev Hiro Asari [e-mail]:

@jakubholynet https://github.com/jakubholynet Is your setup for a key
with head_object privilege or without? Or were you able to test both?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/travis-ci/dpl/issues/741#issuecomment-360991826, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAmJPuiFCA7jjrFBD4KVYWDJfPUTCYzkks5tOz-IgaJpZM4RrCBa
.

[holyjak]

@BanzaiMan I have revoked the S3 bucket read perms and rerund and it worked (did not register the release but did deploy it)

[BanzaiMan]

Awesome. Sounds like we can have a release Monday. Thanks, everyone for all your help!

[holyjak]

Thank you for your great effort :-) And sorry for not thinking about this
case earlier!
søn. 28. jan. 2018 kl. 21:53 skrev Hiro Asari [e-mail]:

Awesome. Sounds like we can have a release Monday. Thanks, everyone for
all your help!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/travis-ci/dpl/issues/741#issuecomment-361094972, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAmJPrThkdAhAJiuH8zpKTYAy4weZEq3ks5tPN5LgaJpZM4RrCBa
.

[BanzaiMan]

I've released 1.8.47. Thanks!