Google-cloud-ruby: Errors in storage acceptance test

Created on 26 Apr 2021  路  19Comments  路  Source: googleapis/google-cloud-ruby

On Ruby 2.5, we get this. This is from the nightly on 2021-04-26.

  1) Error:
Google::Cloud::Storage::Bucket::generate_signed_post_policy_v4::storage#test_0002_generates a signed post object v4 using signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:355:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:69:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1971:in `generate_signed_post_policy_v4'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:83:in `block (2 levels) in <top (required)>'

  2) Error:
Google::Cloud::Storage::Bucket::post_object::v2::storage#test_0002_generates a signed post object using signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:113:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1811:in `post_object'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:83:in `block (2 levels) in <top (required)>'

  3) Error:
Google::Cloud::Storage::signed_url::v2::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url using IAM signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:128:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1619:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:132:in `block (3 levels) in <top (required)>'

  4) Error:
Google::Cloud::Storage::signed_url::v4::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url v4 using IAM signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:214:in `block in service_account_signer'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:124:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1632:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:132:in `block (3 levels) in <top (required)>'

On Ruby 3.0, we get this. From the same nightly.

  1) Error:
Google::Cloud::Storage::signed_url::v2::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url using IAM signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:128:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1619:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:132:in `block (3 levels) in <top (required)>'

  2) Error:
Google::Cloud::Storage::Bucket::post_object::v2::storage#test_0002_generates a signed post object using signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:113:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1811:in `post_object'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:83:in `block (2 levels) in <top (required)>'

  3) Error:
Google::Cloud::Storage::signed_url::v4::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url v4 using IAM signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:214:in `block in service_account_signer'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:124:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1632:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:132:in `block (3 levels) in <top (required)>'

  4) Error:
Google::Cloud::Storage::Bucket::generate_signed_post_policy_v4::storage#test_0002_generates a signed post object v4 using signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:355:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:69:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1971:in `generate_signed_post_policy_v4'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:83:in `block (2 levels) in <top (required)>'
storage p2 bug
Source
picture dazuma

All 19 comments

I could not reproduce the Google::Apis::ClientError: Invalid request locally with Ruby 3.0.1.

I did reproduce the ArgumentError: unknown keyword: payload locally with Ruby 2.6.6 and 2.7.3.

The errors come from this usage pattern:

    signer = lambda do |string_to_sign|
      request = {
        "payload": string_to_sign,
      }
      resource = "projects/-/serviceAccounts/#{issuer}"
      response = iam_client.sign_service_account_blob resource, request
      response.signed_blob
    end

@dazuma Do you know of any reason why the hash form of the request would no longer be accepted? The payload attribute appears unchanged in SignBlobRequest. The signature of sign_service_account_blob also appears unchanged since https://github.com/googleapis/google-api-ruby-client/pull/2139.

quartzmo picture quartzmo on 27 Apr 2021

The only change to google-apis-iamcredentials_v1 was the trivial doc update in https://github.com/googleapis/google-api-ruby-client/pull/3603 that was released in v0.3.0.

quartzmo picture quartzmo on 27 Apr 2021

Before the v0.3.0 release, I thoroughly checked for keyword argument errors in acceptance tests when I did https://github.com/googleapis/google-cloud-ruby/pull/10575.

quartzmo picture quartzmo on 27 Apr 2021

The unknown keyword: payload error can be avoided by using a string key in the request hash:

request = {
  "payload" => string_to_sign,
}

However, once this problem is solved on the Ruby 2.x runs, the Google::Apis::ClientError: Invalid request is then seen on those Ruby versions as well.

I am now able to reproduce the error on Ruby 3.0.1. The tests fail for me as shown above with Google::Apis::ClientError: Invalid request.

quartzmo picture quartzmo on 27 Apr 2021

@frankyn Any idea why we might be seeing errors in calls to IAMCredentialsService#sign_service_account_blob?

quartzmo picture quartzmo on 27 Apr 2021

Hi @quartzmo,

This has not come up (yet) in other languages. Could you log request JSON bodies for payload : and payload =>? I'm wondering if there's a gotcha / issue with the json body between the two.

frankyn picture frankyn on 28 Apr 2021

Thanks @frankyn, will do! I'll also log the unencoded string_to_sign.

quartzmo picture quartzmo on 28 Apr 2021

@frankyn, here are the logged requests for the three different styles of request object, using Ruby 3.0.1 to avoid ArgumentError: unknown keyword: payload in Ruby 2.x.

Style 1

request = { "payload": string_to_sign }

request object in Google::Apis::Core::ApiCommand:

{:payload=>"GOOG4-RSA-SHA256\n20210429T153232Z\n20210429/auto/storage/goog4_request\n1e2a0ed22c06dac793d44cc09b10935d002b14116e606ea8c40f20129c343b22"}

HTTP request body (Update, from a later test run):

{"payload":"R09PRzQtUlNBLVNIQTI1NgoyMDIxMDQyOVQyMTU4MjlaCjIwMjEwNDI5L2F1dG8vc3RvcmFnZS9nb29nNF9yZXF1ZXN0CmVkMzA1ODJjZTU3Njg4NzUwMjdlNzE3NTg1NWJhNzNkMGU2ZmFjZDI3ZmY1OWI2ZWIzZjhjYWJmNGEyOTdjNjk="}

Result: Works in Ruby 3.0. Causes ArgumentError: unknown keyword: payload in Ruby 2.x.

#<Google::Apis::Core::ApiCommand:0x00007fbd7b2966f8 @options=#<struct Google::Apis::RequestOptions authorization=#<Signet::OAuth2::Client:0x00007fbd7c17e698 @authorization_uri=nil, @token_credential_uri=#<Addressable::URI:0xb7c URI:https://oauth2.googleapis.com/token>, @client_id=nil, @client_secret=nil, @code=nil, @expires_at=2021-04-29 10:32:29.584605 -0600, @issued_at=2021-04-29 09:32:30.584605 -0600, @issuer="542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com", @password=nil, @principal=nil, @redirect_uri=nil, @scope=["https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/iam"], @target_audience=nil, @state=nil, @username=nil, @access_type=:offline, @expiry=60, @audience="https://oauth2.googleapis.com/token", @signing_key=#<OpenSSL::PKey::RSA:0x00007fbd7c17e788 oid=rsaEncryption>, @extension_parameters={}, @additional_parameters={}, @connection_info=nil, @grant_type=nil, @refresh_token=nil, @access_token="ya29.c.KtUB-wf94v8nv0vQnQ5RQDZYw76KqNMWU48u6U4uDlYo_zqwuQdoLKeRYeD1jYimRZBW_s-r_efcaB3LuccRu62b6InEZCbfMoGExT5sHZPMEDp8ib7og3CGyPGCKTpwRmPDJ055FDqAZKnfsKJHisqdppzsVM1EgxRhfAeR2BVX1grrAlBTw0ZL78vTgoWmIGF26bVPQ2gRwnndFKuKL8tynUeL8WpBQyIf3OxU8sVcMhCTqOoq4espDyFgkvekThRY3mlGT2mocHG-0Q-YDpQpUOWszlEx">, retries=0, header=nil, normalize_unicode=false, skip_serialization=false, skip_deserialization=false, api_format_version=nil, use_opencensus=true, quota_project=nil, query=nil>, @url=#<Addressable::Template:0xb90 PATTERN:https://iamcredentials.googleapis.com/v1/{+name}:signBlob>, @method=:post, @header={}, @body=nil, @query={}, @params={}, @opencensus_span=nil, @client_version="0.3.0", @request_representation=Google::Apis::IamcredentialsV1::SignBlobRequest::Representation, @request_object={:payload=>"GOOG4-RSA-SHA256\n20210429T153232Z\n20210429/auto/storage/goog4_request\n1e2a0ed22c06dac793d44cc09b10935d002b14116e606ea8c40f20129c343b22"}>
D, [2021-04-29T08:51:46.295629 #89655] DEBUG -- : Sending HTTP post https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?
D, [2021-04-29T08:51:46.579239 #89655] DEBUG -- : 200
D, [2021-04-29T08:51:46.579488 #89655] DEBUG -- : #<HTTP::Message:0x00007fa92021bc60 @http_header=#<HTTP::Message::Headers:0x00007fa92021bb20 @http_version="1.1", @body_size=0, @chunked=false, @request_method="POST", @request_uri=#<Addressable::URI:0xb90 URI:https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?>, @request_query=nil, @request_absolute_uri=nil, @status_code=200, @reason_phrase="OK", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Content-Type", "application/json; charset=UTF-8"], ["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Encoding", "gzip"], ["Date", "Thu, 29 Apr 2021 14:51:46 GMT"], ["Server", "scaffolding on HTTPServer2"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x00007fa924260410>, not_before=2021-04-13 10:16:07 UTC, not_after=2021-07-06 10:16:06 UTC>, @http_body=#<HTTP::Message::Body:0x00007fa92021b8f0 @body="{\n  \"keyId\": \"153f4805d285f5df11d46b88c6c8d53006e7a6c3\",\n  \"signedBlob\": \"ljejz/RkCSuwZboDqokVdl/hu88RYPoz9FNRTI+k6QAWvFqVN66NZQGncQx5m8qnRrQ12LlbDuK8lIazF7XtZq8HC9m3yTvNkzHOwylWXSARGVSFGosRwCbj9hh+531aLRiE/jC2kTnOQV1qACiRWSHIZ6VkR0QccmelH89j5eZPVioafAGQgU0fpnJoDY6EbIIxZsyR3mLEH0KpSudH0iUYcyO+SptWpdKRQ8pgSL0sLVuA4Ai5uAiN5Im3f0OFSda3Z2tnR6iITKcucw4dmoFqJm8RkNeuVekZPP3QjX9GZ+1ipY3IgAI5fUpErV4kmx2Eo75zHFUX//1ncRu84Q==\"\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>
D, [2021-04-29T08:51:46.579884 #89655] DEBUG -- : Success - #<Google::Apis::IamcredentialsV1::SignBlobResponse:0x00007fa9231fddc0
 @key_id="153f4805d285f5df11d46b88c6c8d53006e7a6c3",
 @signed_blob=
  "\x967\xA3\xCF\xF4d\t+\xB0e\xBA\x03\xAA\x89\x15v_\xE1\xBB\xCF\x11`\xFA3\xF4SQL\x8F\xA4\xE9\x00\x16\xBCZ\x957\xAE\x8De\x01\xA7q\fy\x9B\xCA\xA7F\xB45\xD8\xB9[\x0E\xE2\xBC\x94\x86\xB3\x17\xB5\xEDf\xAF\a\v\xD9\xB7\xC9;\xCD\x931\xCE\xC3)V] \x11\x19T\x85\x1A\x8B\x11\xC0&\xE3\xF6\x18~\xE7}Z-\x18\x84\xFE0\xB6\x919\xCEA]j\x00(\x91Y!\xC8g\xA5dGD\x1Crg\xA5\x1F\xCFc\xE5\xE6OV*\x1A|\x01\x90\x81M\x1F\xA6rh\r\x8E\x84l\x821f\xCC\x91\xDEb\xC4\x1FB\xA9J\xE7G\xD2%\x18s#\xBEJ\x9BV\xA5\xD2\x91C\xCA`H\xBD,-[\x80\xE0\b\xB9\xB8\b\x8D\xE4\x89\xB7\x7FC\x85I\xD6\xB7gkgG\xA8\x88L\xA7.s\x0E\x1D\x9A\x81j&o\x11\x90\xD7\xAEU\xE9\x19<\xFD\xD0\x8D\x7FFg\xEDb\xA5\x8D\xC8\x80\x029}JD\xAD^$\x9B\x1D\x84\xA3\xBEs\x1CU\x17\xFF\xFDgq\e\xBC\xE1">

Style 2

request = { "payload" => string_to_sign }

request object in Google::Apis::Core::ApiCommand:

{"payload"=>"GOOG4-RSA-SHA256\n20210429T153000Z\n20210429/auto/storage/goog4_request\n0cb37904dfe5281443f3be44e31ba639c037164be0bc88850f48f9e6c600efc0"}

HTTP request body (Update, from a later test run):

{}

Result: Causes Google::Apis::ClientError: Invalid request in all Ruby versions.

 #<Google::Apis::Core::ApiCommand:0x00007f83073cf460 @options=#<struct Google::Apis::RequestOptions authorization=#<Signet::OAuth2::Client:0x00007f8306242918 @authorization_uri=nil, @token_credential_uri=#<Addressable::URI:0xb7c URI:https://oauth2.googleapis.com/token>, @client_id=nil, @client_secret=nil, @code=nil, @expires_at=2021-04-29 10:29:58.01145 -0600, @issued_at=2021-04-29 09:29:59.01145 -0600, @issuer="542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com", @password=nil, @principal=nil, @redirect_uri=nil, @scope=["https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/iam"], @target_audience=nil, @state=nil, @username=nil, @access_type=:offline, @expiry=60, @audience="https://oauth2.googleapis.com/token", @signing_key=#<OpenSSL::PKey::RSA:0x00007f8306242a08 oid=rsaEncryption>, @extension_parameters={}, @additional_parameters={}, @connection_info=nil, @grant_type=nil, @refresh_token=nil, @access_token="ya29.c.KtUB-wdfdrg6zZuRmfEfAUo_jGphTLmXGGab94fv4mAHL-kUSJ22eMfCdaYrK_26kPRw-tW8WUJ6vmAyJ0xMO8Hw5na32C1lyn9Zcu_ZU5-8ijFhM8LFExi1df8u3zdOMSKMC5SkX35dcJuPuK07EQrDY2iWOjPouaLje2P5WD08oxOMaJGBxpF_I2hE6EkyKIkjIx3vBvsQKHsbQL0-d8MHoyf6LTdw3Q9VTwiE5X7uALY2R6U5jrq3hvLIavaaNywh80wVWoKmuIOrTcMlCuOndbYrdcAF">, retries=0, header=nil, normalize_unicode=false, skip_serialization=false, skip_deserialization=false, api_format_version=nil, use_opencensus=true, quota_project=nil, query=nil>, @url=#<Addressable::Template:0xb90 PATTERN:https://iamcredentials.googleapis.com/v1/{+name}:signBlob>, @method=:post, @header={}, @body=nil, @query={}, @params={}, @opencensus_span=nil, @client_version="0.3.0", @request_representation=Google::Apis::IamcredentialsV1::SignBlobRequest::Representation, @request_object={"payload"=>"GOOG4-RSA-SHA256\n20210429T153000Z\n20210429/auto/storage/goog4_request\n0cb37904dfe5281443f3be44e31ba639c037164be0bc88850f48f9e6c600efc0"}>
D, [2021-04-29T08:53:11.085236 #90013] DEBUG -- : Sending HTTP post https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?
D, [2021-04-29T08:53:11.345776 #90013] DEBUG -- : 400
D, [2021-04-29T08:53:11.346028 #90013] DEBUG -- : #<HTTP::Message:0x00007fe06f166140 @http_header=#<HTTP::Message::Headers:0x00007fe06f165ce0 @http_version="1.1", @body_size=0, @chunked=false, @request_method="POST", @request_uri=#<Addressable::URI:0xb90 URI:https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?>, @request_query=nil, @request_absolute_uri=nil, @status_code=400, @reason_phrase="Bad Request", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Type", "application/json; charset=UTF-8"], ["Content-Encoding", "gzip"], ["Date", "Thu, 29 Apr 2021 14:53:11 GMT"], ["Server", "scaffolding on HTTPServer2"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x00007fe06fa5b688>, not_before=2021-04-13 10:16:07 UTC, not_after=2021-07-06 10:16:06 UTC>, @http_body=#<HTTP::Message::Body:0x00007fe06f164a48 @body="{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"Request contains an invalid argument.\",\n    \"status\": \"INVALID_ARGUMENT\"\n  }\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>

Style 3

request = Google::Apis::IamcredentialsV1::SignBlobRequest.new(payload: string_to_sign)

request object in Google::Apis::Core::ApiCommand:

#<Google::Apis::IamcredentialsV1::SignBlobRequest:0x00007ff8fda7bce8 @payload="GOOG4-RSA-SHA256\n20210429T153408Z\n20210429/auto/storage/goog4_request\n502e22058701e6935ea79fb232270d7e37e1e40a1ab1bb001c384c739856471c">

HTTP request body (Update, from a later test run):

{"payload":"R09PRzQtUlNBLVNIQTI1NgoyMDIxMDQyOVQyMjA0MTdaCjIwMjEwNDI5L2F1dG8vc3RvcmFnZS9nb29nNF9yZXF1ZXN0CjdmOWUwYTIyM2RiMGRjMmIwMGU3ZDgwNWQzMGM1YTEzODI4ODJlODNiZjc2MzUxOWZlMmUxNTI4NWRiNjk2MTQ="}

Result: Works in all Ruby versions.

 #<Google::Apis::Core::ApiCommand:0x00007ff8fda7ad20 @options=#<struct Google::Apis::RequestOptions authorization=#<Signet::OAuth2::Client:0x00007ff8fd9b9ff8 @authorization_uri=nil, @token_credential_uri=#<Addressable::URI:0xb7c URI:https://oauth2.googleapis.com/token>, @client_id=nil, @client_secret=nil, @code=nil, @expires_at=2021-04-29 10:34:05.785206 -0600, @issued_at=2021-04-29 09:34:06.785206 -0600, @issuer="542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com", @password=nil, @principal=nil, @redirect_uri=nil, @scope=["https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/iam"], @target_audience=nil, @state=nil, @username=nil, @access_type=:offline, @expiry=60, @audience="https://oauth2.googleapis.com/token", @signing_key=#<OpenSSL::PKey::RSA:0x00007ff8fd9ba0c0 oid=rsaEncryption>, @extension_parameters={}, @additional_parameters={}, @connection_info=nil, @grant_type=nil, @refresh_token=nil, @access_token="ya29.c.KtUB-wee4j4_EyLtaW5pxwbfZ4_Kf5jHJ-E8p-DXGq4-V_lqKnyRDc0mi51_KxfLnk_uE5c7A8K3EzomYjIeCHfnavMX807F8GtkKhObo9dQ5GIEyU-pZ8tNCU4j6NAAGsqnOE3AEvy9uihyk1fdOBQM8bAGbWMMw43p300ueFUvBwHNw94qXc_QMgWXU0bASqFgJugesePrHo5EXaJmuuYLg1_v3ndQsbGIoWBGJnG4dCOxVC_AtlSsKIdMiojrLU7e8tpL2Ozz5Vtbfu_Vl7e9shHuNDXo">, retries=0, header=nil, normalize_unicode=false, skip_serialization=false, skip_deserialization=false, api_format_version=nil, use_opencensus=true, quota_project=nil, query=nil>, @url=#<Addressable::Template:0xb90 PATTERN:https://iamcredentials.googleapis.com/v1/{+name}:signBlob>, @method=:post, @header={}, @body=nil, @query={}, @params={}, @opencensus_span=nil, @client_version="0.3.0", @request_representation=Google::Apis::IamcredentialsV1::SignBlobRequest::Representation, @request_object=#<Google::Apis::IamcredentialsV1::SignBlobRequest:0x00007ff8fda7bce8 @payload="GOOG4-RSA-SHA256\n20210429T153408Z\n20210429/auto/storage/goog4_request\n502e22058701e6935ea79fb232270d7e37e1e40a1ab1bb001c384c739856471c">>
D, [2021-04-29T08:48:20.236387 #89235] DEBUG -- : Sending HTTP post https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?
D, [2021-04-29T08:48:20.508003 #89235] DEBUG -- : 200
D, [2021-04-29T08:48:20.508295 #89235] DEBUG -- : #<HTTP::Message:0x00007fdbbc168c68 @http_header=#<HTTP::Message::Headers:0x00007fdbbc168b78 @http_version="1.1", @body_size=0, @chunked=false, @request_method="POST", @request_uri=#<Addressable::URI:0xb90 URI:https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?>, @request_query=nil, @request_absolute_uri=nil, @status_code=200, @reason_phrase="OK", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Content-Type", "application/json; charset=UTF-8"], ["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Encoding", "gzip"], ["Date", "Thu, 29 Apr 2021 14:48:20 GMT"], ["Server", "scaffolding on HTTPServer2"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x00007fdbbd29f518>, not_before=2021-04-13 10:16:07 UTC, not_after=2021-07-06 10:16:06 UTC>, @http_body=#<HTTP::Message::Body:0x00007fdbbc168628 @body="{\n  \"keyId\": \"153f4805d285f5df11d46b88c6c8d53006e7a6c3\",\n  \"signedBlob\": \"HTG1qu8fq0WZUL8/JT8uBJRP0LYh8TDEkoEW9+5T7qaq7hMEsgbB5dnrDMW8Dz+LlcjCeY/JuSlCLWdyW1X+AbKDeKwS1MQ9xw3X3oAXtNU+P/eQaESYRZGy8YhT8iSWaR7oIMc/095fMaAsQuu+7tk2tBIKaydjpacjriCFUQl/C7RKCAdXqvcuy9XInmAGARol+RdgnvRqiOvS9yhGZyM58cuu1G2L552DeldCMbx+B3mZ2M1zGCzXyAe1XOkqPKVB18Ah+QYqKrPVFORcAWbK8X4Ia7zSbtT6hcxmRO2ntXEt2bLvTag+t38SJUiBxLGWhtl9/jcO4O9MjRDmxg==\"\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>
quartzmo picture quartzmo on 29 Apr 2021

Oh interesting, TIL that the Ruby gem google-api-client was split into multiple gems and mono deprecated!

The PRs are very lengthy, but didn't pick up on anything that might be relevant. I referring to the following PRs:

  1. https://github.com/googleapis/google-api-ruby-client/pull/2134
  2. https://github.com/googleapis/google-api-ruby-client/pull/2143

@dazuma i'm assuming no major logic changes were made, but wondering if you may have more context.

frankyn picture frankyn on 29 Apr 2021

@frankyn I've been trying to find a change that caused the ArgumentError: unknown keyword: payload errors, but so far I haven't.

quartzmo picture quartzmo on 29 Apr 2021

I also got the ArgumentError: unknown keyword: payload error on Ruby 2.7.2.

quartzmo picture quartzmo on 29 Apr 2021

@quartzmo for the logs, I see that we only have response JSON body, is there an equivalent request JSON structure?

BTW: Thanks for getting the additional logging!!

frankyn picture frankyn on 29 Apr 2021

@frankyn I wasn't able to figure out how to log the request. I can try again this evening.

quartzmo picture quartzmo on 29 Apr 2021
👍1

@frankyn I updated the logs comment above with the request bodies.

It turns out that in Style 2, with the { "payload" => string_to_sign } request object argument, the body is {}.

quartzmo picture quartzmo on 30 Apr 2021

Thanks for the additional context @quartzmo!

IIUC the tests and rubydoc samples should use the class data model representation instead of a Ruby Hash to construct the request payload.

The request bodies are equivalent, so wondering if there's something going on the URL because it contains special characters in the URI with the Google Service account email.

BTW: I'm okay with shelving this and moving forward with the PR you have open at this point to address the broken tests.

frankyn picture frankyn on 30 Apr 2021

I reproduced what's happening inside ApiCommand:

irb(main):005:0> request_object = {"payload" => "123"}
=> {"payload"=>"123"}
irb(main):006:0> request_representation = Google::Apis::IamcredentialsV1::SignBlobRequest::Representation
=> Google::Apis::IamcredentialsV1::SignBlobRequest::Representation
irb(main):007:0> body = request_representation.new(request_object).to_json(user_options: { skip_undefined: true })
=> "{}"
quartzmo picture quartzmo on 30 Apr 2021
👀1

https://github.com/googleapis/google-api-ruby-client/blob/master/google-apis-core/lib/google/apis/core/api_command.rb#L81

quartzmo picture quartzmo on 30 Apr 2021

@frankyn Keep in mind that we didn't use Style 2 originally, we used Style 1. Which still works if you can get past the ArgumentError: unknown keyword: payload errors:

irb(main):021:0> body = request_representation.new({"payload": "123"}).to_json(user_options: { skip_undefined: true })
=> "{\"payload\":\"MTIz\"}"
quartzmo picture quartzmo on 30 Apr 2021

Thank you @frankyn for finding the cause of the ArgumentError: unknown keyword: payload errors in the Ruby 2.x versions: In #10575, I removed the trailing empty hash ({}) in test calls to sign_service_account_blob, in order to fix the Ruby 3.0 error ArgumentError: wrong number of arguments (given 3, expected 1..2). I must have forgotten to regression-test this change against the Ruby 2.x versions, since it introduced the unknown keyword error in those versions.

quartzmo picture quartzmo on 30 Apr 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

danicuki picture danicuki  路  4Comments
bendillinger picture bendillinger  路  3Comments
jeremywadsack picture jeremywadsack  路  3Comments
bkoski picture bkoski  路  3Comments
mackeee-orange picture mackeee-orange  路  3Comments