Google-cloud-ruby: POST Object : Getting issues with V4

[chvreddy]

Thanks for stopping by to let us know something could be better!

Environment details

Environment details
OS: MAC
Ruby version: ruby 2.4.9p362 (2019-10-02 revision 67824) [x86_64-darwin19]
Gem name and version: gem 'google-cloud-storage', '1.29.1'

Steps to reproduce

Code example

ruby

V4 process using Post Object throws error:

STEPS TO PRODUCE:

conditions = [ ["starts-with", "$key", ""] ]

result = self.class.bucket.generate_signed_post_policy_v4("#{inbox_prefix}#{Time.current.to_i}_${filename}", signer: gcs_signed_url(nil, nil, "POST"), issuer: ENV["GCS_SERVICE_ACCOUNT"], expires: 10, conditions: conditions )

result.fields.merge!(success_action_redirect: success_action_redirect)

error:

<Error>
<Code>InvalidPolicyDocument</Code>
<Message>The content of the form does not meet the conditions specified in the policy document.</Message>
<Details>Invalid condition: [["starts-with","$key",""]]</Details>
</Error>

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[chvreddy]

updated the post, please review.

[quartzmo]

@chvreddy Thanks for reporting this. I see that the project has acceptance tests for Bucket#post_object (V2) that set the condition ["starts-with", "$key", ""] just like your example. But I do not see a condition in the acceptance tests for Bucket#generate_signed_post_policy_v4 in bucket_generate_signed_post_policy_v4_test.rb. I'll try to add V4 test coverage for that condition this week. I'll post a link here.

[chvreddy]

Thanks @quartzmo Few thing want to get to your notice

• I don't mention any conditions it's throwing below error

InvalidPolicyDocument
The content of the form does not meet the conditions specified in the policy document.

Policy did not reference these fields: commit success_action_redirect

My question is in Bucket#post_object (V2) we use policy, but here in Bucket#generate_signed_post_policy_v4 we use condition right?

• Also in the error what is the field commit refers to didn't find anything about that in doc, because even when I use Bucket#post_object (V2) I am getting the error regarding commit field. I am not seeing any hidden input field type commit in my form.

• Also If i provide success_action_redirect condition its throwing below error

InvalidPolicyDocument
The content of the form does not meet the conditions specified in the policy document.

Invalid condition: [{"success_action_redirect":"http://localhost:3000/banker/final_qc_loan_tapes/files_processing"}]

conditions = [ {"success_action_redirect":success_action_redirect} ]
result = self.class.bucket.generate_signed_post_policy_v4("#{inbox_prefix}#{Time.current.to_i}_${filename}", signer: gcs_signed_url(nil, nil, "POST"), issuer: ENV["GCS_SERVICE_ACCOUNT"], expires: 10, conditions: conditions )
fields = result.fields.merge!(success_action_redirect: success_action_redirect)
result
browser inspected:
<input type="hidden" name="success_action_redirect" value="http://localhost:3000/banker/final_qc_loan_tapes/files_processing">

[quartzmo]

@chvreddy Can you try to create a full, end-to-end code example that I can run? We have end-to-end acceptance tests in the project that use Net::HTTP::Post to successfully use the POST Object created with Bucket#generate_signed_post_policy_v4. If possible, can you please try the following to adapt one of these tests to your use case?

1. Fork the google-cloud-ruby GitHub repo.
2. Set environment variables with your project ID and service account credentials.
3. Follow the Setup instructions to install dependencies for google-cloud-ruby/google-cloud-storage.
4. In acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb, choose a test that best fits what you are trying to do. It might be the last one, "generates a signed post object v4 with success_action_status". Add focus to the blank line above the test (above the call to it). This will run just the one test.
5. Run: bundle exec rake acceptance. Confirm that the unmodified test passes using your project.
6. If the unmodified test passes with your project, modify the test to match your use case, and re-run. It's best of course to do this in small steps.
7. When you can demonstrate the problem with the service or client with a failing test, commit your changes in a branch and push the branch to your fork. Post the link to your branch here, I'll pull it, and try for myself.

[chvreddy]

@quartzmo none of your tests in acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb covered conditions options i.e

      conditions = [["starts-with","$acl","public"]]
      post = bucket.generate_signed_post_policy_v4 "avatars/heidi/400x400.png", expires: 10, conditions: conditions

used link https://googleapis.dev/ruby/google-cloud-storage/latest/Google/Cloud/Storage/PostObject.html

since the bucket we have is not granted with WRITE or FULL_CONTROL permission to anonymous users need to set policy(so added conditions to generate_signed_post_policy_v4 as shown above correct me if that's wrong)
from: https://cloud.google.com/storage/docs/xml-api/post-object

forked the repo and wrote the additional specs using conditions and they are failing
https://github.com/chvreddy/google-cloud-ruby/commit/23200842b9d15d8734903ffa492e0f123c385b79

[quartzmo]

@chvreddy Thank you! I'll try out your new tests today.

[chvreddy]

Hi @quartzmo,
one more help, I also implemented post object with V2

policy = {
      expiration: (Time.now + 3600).iso8601,
      conditions: [
        ["starts-with", "$key", ""],
        {"success_action_redirect":success_action_redirect}
      ]
    }
    result = self.class.bucket.post_object("#{inbox_prefix}#{Time.current.to_i}_${filename}", policy: policy, signer: gcs_signed_url(nil, nil, "POST"), issuer: ENV["GCS_SERVICE_ACCOUNT"] )
    fields = result.fields.merge!(success_action_redirect: success_action_redirect)
    result

weird thing is for some CSV files its working fine but for some CSV's we are getting below error

<Error>
<Code>InvalidPolicyDocument</Code>
<Message>The content of the form does not meet the conditions specified in the policy document.</Message>
<Details>Policy did not reference these fields: commit</Details>
</Error>

Can you also please look in to it if possible

[quartzmo]

weird thing is for some CSV files its working fine but for some CSV's we are getting below error

This sounds hard to reproduce. In your branch, can you add a test to bucket_post_object_v2_test.rb that consistently reproduces it?

[chvreddy]

weird thing is for some CSV files its working fine but for some CSV's we are getting below error

This sounds hard to reproduce. In your branch, can you add a test to bucket_post_object_v2_test.rb that consistently reproduces it?

That's difficult because even I don't know why its failing i uploaded a file in dev env and it worked fine, but when QA tested the same file in dev env it failed with above error. So really i am confused and have no idea why its failing.
any idea about the policy field names commit(didn't find any doc with that reference)

[quartzmo]

any idea about the policy field names commit(didn't find any doc with that reference)

Are you using Rails? Do you see this in your form? <input name="commit" type="submit" ...

[quartzmo]

See the generated forms in https://guides.rubyonrails.org/v5.2/form_helpers.html

[chvreddy]

any idea about the policy field names commit(didn't find any doc with that reference)

Are you using Rails? Do you see this in your form? <input name="commit" type="submit" ...

Yes using rails and Yes I do see that. Understood where that commit tag is coming from, but then why is it failing sometimes and passing sometimes

[quartzmo]

why is it failing sometimes and passing sometimes

Unfortunately I don't know either, but the first thing I would do is to try to port this HTML form out of the complex Rails environment to the simpler environment of the acceptance tests, thus reproducing it using Net::HTTP::Post in a way that you can easily share it via GitHub. I understand that it might be hard to do.

[chvreddy]

@quartzmo I am not sure if its helpful for you to solve the issue but if the file size that I am uploading is less than 65KB then the upload is failing with below error

<Error>
<Code>InvalidPolicyDocument</Code>
<Message>The content of the form does not meet the conditions specified in the policy document.</Message>
<Details>Policy did not reference these fields: commit</Details>
</Error>

will also look from my side if there are any restrictions of file size(mostly no)

[joedicator]

@chvreddy

but if the file size that I am uploading is less than 65KB then the upload is failing with below error

Does it fail if it is _less_ than or _greater_ than 65KB?

[chvreddy]

@chvreddy

but if the file size that I am uploading is less than 65KB then the upload is failing with below error

Does it fail if it is _less_ than or _greater_ than 65KB?

@joedicator it fails with less_than 65KB

[quartzmo]

@chvreddy I spend a few hours today trying to add a test for Post Object V4 using conditions, but no success. See PR linked above.

[frankyn]

Following up internally, I was also not able to reproduce this issue, I'll follow-up when I get input from the backend team.

Bug: 175159591

[chvreddy]

@quartzmo I am not sure if its helpful for you to solve the issue but if the file size that I am uploading is less than 65KB then the upload is failing with below error

<Error>
<Code>InvalidPolicyDocument</Code>
<Message>The content of the form does not meet the conditions specified in the policy document.</Message>
<Details>Policy did not reference these fields: commit</Details>
</Error>

will also look from my side if there are any restrictions of file size(mostly no)

@quartzmo the V4 post object is working fine but I am still getting this error
Also one more question does the policy should refer to only hidden input fields or will also look for input field with submit tag(because the commit field name is for the input type submit) that too its happening only for files with size less than 65KB
tried with both V2 and V4 same result

[quartzmo]

@chvreddy Can you please open a new issue for your Policy did not reference these fields: commit error, and include as much detail as possible? It would be very helpful to have the entire HTML form, the source code creating the PostObject, and any other relevant details. If you can reproduce the error in an acceptance test in a fork of this repo (see instructions above), that would be extremely helpful.

[quartzmo]

weird thing is for some CSV files its working fine but for some CSV's we are getting below error

if the file size that I am uploading is less than 65KB then the upload is failing with below error

I added a small (< 1KB) CSV file to the acceptance tests, which is used here in a new passing test added in #8373.

[chvreddy]

weird thing is for some CSV files its working fine but for some CSV's we are getting below error

if the file size that I am uploading is less than 65KB then the upload is failing with below error

I added a small (< 1KB) CSV file to the acceptance tests, which is used here in a new passing test added in #8373.

yeah I saw the test, I think its somehow related to the input field type commit of submit tag.

[chvreddy]

@chvreddy Can you please open a new issue for your Policy did not reference these fields: commit error, and include as much detail as possible? It would be very helpful to have the entire HTML form, the source code creating the PostObject, and any other relevant details. If you can reproduce the error in an acceptance test in a fork of this repo (see instructions above), that would be extremely helpful.

sure will open a new ticket, Thanks!

[joedicator]

@chvreddy please post new ticket link here when created.

[chvreddy]

created new issue for above mentioned error
https://github.com/googleapis/google-cloud-ruby/issues/8402