Moto: Metadata not available when using pre-signed urls

[sturmer]

Hi all! 👋

This is a better formulation (hopefully) than #2040. What I do:

• generate a pre-signed url adding Metadata when doing it
• upload data via PUT to that generated url
• retrieve the data just uploaded

Expected: I should be able to see the metadata.
Actual result: No metadata is present.

To reproduce, run this script:

# https://github.com/spulec/moto/issues/2040
import boto3
from moto import mock_s3
import requests


@mock_s3
def test_generate_presigned_url():
    conn = boto3.resource('s3', region_name='us-east-1')
    conn.create_bucket(Bucket='mybucket')
    s3 = boto3.client('s3', region_name='us-east-1')

    # Create a pre-signed url with some metadata.
    url = s3.generate_presigned_url(ClientMethod="put_object", Params={
        'Bucket': 'mybucket', 'Key': 'mykey', 'Metadata': {'venue': "123"}})

    # Use the url to upload data.
    requests.put(url, data={'Bucket': 'mybucket',
                            'Key': 'mykey', 'Body': 'myvalue'})

    # Retrieve metadata
    return s3.head_object(Bucket='mybucket', Key='mykey').get('Metadata')


# To show the bug, use test_my_model_head_object() to fill metadata. For the working example, use test_my_model_put_object().
if __name__ == '__main__':
    metadata = test_generate_presigned_url()
    assert 'venue' in metadata
    assert metadata['venue'] == '123'

• moto version: 1.3.7

I'll be back with a patch if time permits.

[sturmer]

Possibly causing: https://github.com/localstack/localstack/issues/544

[spulec]

Thanks for opening. I don't believe we've added the code so that this will work so I'm going to set this as a feature request.

[ArkadioG]

I've got the same issue as @sturmer, when using presigned url to upload file.

Here is example presigned url:

http://localstack:4572/trustportal-assets-local/key=d290f1ee-6c54-4b01-90e6-d701748f0852/f47c3706-52c9-4595-afb5-820b25931411/certifficate5.pdf&AWSAccessKeyId=foo&policy=eyJleHBpcmF0aW9uIjogIjIwMTktMTAtMjhUMTY6MDc6MzNaIiwgImNvbmRpdGlvbnMiOiBbeyJidWNrZXQiOiAidHJ1c3Rwb3J0YWwtYXNzZXRzLWxvY2FsIn0sIHsia2V5IjogImQyOTBmMWVlLTZjNTQtNGIwMS05MGU2LWQ3MDE3NDhmMDg1Mi9mNDdjMzcwNi01MmM5LTQ1OTUtYWZiNS04MjBiMjU5MzE0MTEvY2VydGlmZmljYXRlNS5wZGYifV19&signature=hoKKHWSsufHOL36b1u8liEO1Ex8%3D

After PUT file using presigned url i receive sqs message, and what i see, is that localstack is using whole presigned url with params as key for the file instead of parsing parameters to get data (here is excerpt from sqs message - for event fired by s3 after uploading file using above presigned url):

'object': {  
'key': 'key=d290f1ee-6c54-4b01-90e6-d701748f0852/f47c3706-52c9-4595-afb5-820b25931411/certifficate5.pdf&AWSAccessKeyId=foo&policy=eyJleHBpcmF0aW9uIjogIjIwMTktMTAtMjhUMTY6MDc6MzNaIiwgImNvbmRpdGlvbnMiOiBbeyJidWNrZXQiOiAidHJ1c3Rwb3J0YWwtYXNzZXRzLWxvY2FsIn0sIHsia2V5IjogImQyOTBmMWVlLTZjNTQtNGIwMS05MGU2LWQ3MDE3NDhmMDg1Mi9mNDdjMzcwNi01MmM5LTQ1OTUtYWZiNS04MjBiMjU5MzE0MTEvY2VydGlmZmljYXRlNS5wZGYifV19&signature=hoKKHWSsufHOL36b1u8liEO1Ex8%3D',  
'size': 1024,  
'eTag': 'd41d8cd98f00b204e9800998ecf8427e',

Other bugs with processing of that presigned url:

• size is wrong
• ETag value is wrong - looks like returned value after put is always the same: d41d8cd98f00b204e9800998ecf8427e
• returned status code is wrong - 204 should be returned like in aws but 200 is returned after PUT

@sturmer so if you try to get/head object with key you get error, becouse localstack has different key for your object.