Boto3: ValueError: Invalid timestamp "": Unknown string format

Created on 14 Sep 2016 · 17Comments · Source: boto/boto3

code

for obj_sum in bucket.objects.all():
        if obj_sum.storage_class == 'GLACIER':
            obj = s3.Object(obj_sum.bucket_name, obj_sum.key)
                if not obj.restore:
                    print('Submitting restoration request: %s' % obj.key)
                    obj.restore_object(RestoreRequest={"Days": 7})

stacktrace:

Traceback (most recent call last):
  File "/Users/user/app/app-api/restore_s3.py", line 21, in <module>
    temp_restore()
  File "/Users/user/app/app-api/restore_s3.py", line 11, in temp_restore
    if not obj.restore:
  File "/Users/user/app/venv2/lib/python2.7/site-packages/boto3/resources/factory.py", line 339, in property_loader
    self.load()
  File "/Users/user/app/venv2/lib/python2.7/site-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/client.py", line 159, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/client.py", line 483, in _make_api_call
    operation_model, request_dict)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/endpoint.py", line 168, in _send_request
    request, operation_model, attempts)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/endpoint.py", line 233, in _get_response
    response_dict, operation_model.output_shape)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/parsers.py", line 211, in parse
    parsed = self._do_parse(response, shape)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/parsers.py", line 602, in _do_parse
    member_shapes, final_parsed)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/parsers.py", line 658, in _parse_non_payload_attrs
    member_shape, headers[header_name])
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/parsers.py", line 258, in _parse_shape
    return handler(shape, node)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/parsers.py", line 149, in _get_text_content
    return func(self, shape, text)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/parsers.py", line 412, in _handle_timestamp
    return self._timestamp_parser(text)
  File "/Users/user/app/venv2/lib/python2.7/site-packages/botocore/utils.py", line 336, in parse_timestamp
    raise ValueError('Invalid timestamp "%s": %s' % (value, e))
ValueError: Invalid timestamp "Thu,%2031%20Dec%202099%2020:00:00%20GMT": Unknown string format

tested on:
python 2.7 and 3.5
python-dateutil==2.5.3 and python-dateutil==2.4.2
boto==2.39.0 and latest
boto3 1.4.0

closing-soon guidance

Source

piotrek-py

Most helpful comment

Problem

Any un-parseable Expires header will mean you cannot retrieve the object with boto3.

Proposed Solution

In botocore.parsers.ResponseParser, _handle_timestamp should return None or, just the original text if it cannot be converted to datetime. (Possibly wrapped in custom type, ie, Unparseable(data))

Or, the spec for "GetObjectResponse" should note a default value if "Expires" fails to parse.

Details

@kyleknap Here's a reproduction from an internal issue we had with migrating from boto to boto3.

For some versions of boto, it would urlencode any unicode types passed inheaders, resulting in objects that cannot be retrieved by boto3.

It's not present in 2.45.0, but I can confirm it is present in boto==2.38.0

import boto, six, datetime, os
from boto.s3.key import Key

bucket_name = 'my-bucket-to-test'
key_name = os.urandom(8).encode('hex')
data = os.urandom(16)
hdr_expires = u'Tue, 24 Jan 2017 07:52:32 GMT' # this being a unicode string causes the issue

conn = boto.connect_s3()
bucket = conn.get_bucket(bucket_name, validate=False)
key = bucket.new_key(key_name)
key.set_contents_from_file(
    six.BytesIO(data),
    headers={'Expires': hdr_expires})

# Wont expose the Expires header without subclass
class Key2(Key):
    def __init__(self, bucket=None, name=None):
        super(Key2, self).__init__(bucket=bucket, name=name)
        self._expires_header = {}
    def handle_addl_headers(self, headers):
        for k, v in headers:
            if k == 'expires':
                self._expires_header = v

out = six.BytesIO()
fetched = Key2(bucket, key_name)
fetched.get_contents_to_file(out)
print 'Boto2 Expires: %r' % fetched._expires_header

with boto==2.38.0 this prints

Boto2 Expires: 'Tue,%2024%20Jan%202017%2007:52:32%20GMT'

with boto==2.45.0 this prints

Boto2 Expires: 'Tue, 24 Jan 2017 07:52:32 GMT'

The result - objects affected by that bug cannot be read by boto3.

nathan-muir on 24 Jan 2017

👍3

All 17 comments

@Ananas- Is that all of the code needed to reproduce this? The reason you are running into this is that you are providing a url-encoded timestamp in somewhere: Thu,%2031%20Dec%202099%2020:00:00%20GMT based on the traceback, but you are not doing any timestamp manipulation in the code snippet that you provided.

kyleknap on 19 Sep 2016

Yes, I'm just run that code in shell

piotrek-py on 20 Sep 2016

pip freeze
amqp==1.4.9 anyjson==0.3.3 Babel==2.2.0 billiard==3.3.0.22 boto==2.39.0 boto3==1.4.0 botocore==1.4.58 celery==3.1.20 click==6.6 coverage==4.0.3 defusedxml==0.4.1 Django==1.9.4 django-admin-smoke-tests==0.2.1 django-airbrake==1.2.0 django-appconf==1.0.2 django-authtools==1.4.0 django-bootstrap3==7.0.1 django-braces==1.9.0 django-celery==3.1.17 django-celery-transactions==0.3.6 django-cleanup==0.4.2 django-compressor==2.0 django-debug-toolbar==1.4 django-nose==1.4.3 django-oauth-toolkit==0.10.0 django-paintstore==0.2 django-phonenumber-field==1.1.0 django-positions==0.5.3 django-push-notifications==1.4.1 django-rest-framework-social-oauth2==1.0.4 django-stdimage==2.3.2 django-storages==1.4 django-widget-tweaks==1.4.1 djangorestframework==3.3.2 djrill==2.0 docutils==0.12 Flask==0.11.1 googlemaps==2.4.4 gunicorn==19.4.5 httplib2==0.9.2 itsdangerous==0.24 Jinja2==2.8 jmespath==0.9.0 kombu==3.0.33 Markdown==2.6.7 MarkupSafe==0.23 nose==1.3.7 oauth2==1.9.0.post1 oauthlib==2.0.0 phonenumberslite==7.2.8 Pillow==3.1.0 progressbar2==3.6.0 psycopg2==2.6.1 PyJWT==1.4.2 python-dateutil==2.5.3 python-google-places==1.1.0 python-social-auth==0.2.21 python3-openid==3.0.10 pytz==2015.7 rcssmin==1.0.6 redis==2.10.5 requests==2.9.1 requests-oauthlib==0.7.0 rjsmin==1.0.12 s3transfer==0.1.5 self==0.0.2 six==1.10.0 sqlparse==0.1.19 Werkzeug==0.11.11

piotrek-py on 10 Oct 2016

Problem

Any un-parseable Expires header will mean you cannot retrieve the object with boto3.

Proposed Solution

Or, the spec for "GetObjectResponse" should note a default value if "Expires" fails to parse.

Details

@kyleknap Here's a reproduction from an internal issue we had with migrating from boto to boto3.

For some versions of boto, it would urlencode any unicode types passed inheaders, resulting in objects that cannot be retrieved by boto3.

It's not present in 2.45.0, but I can confirm it is present in boto==2.38.0

import boto, six, datetime, os
from boto.s3.key import Key

bucket_name = 'my-bucket-to-test'
key_name = os.urandom(8).encode('hex')
data = os.urandom(16)
hdr_expires = u'Tue, 24 Jan 2017 07:52:32 GMT' # this being a unicode string causes the issue

conn = boto.connect_s3()
bucket = conn.get_bucket(bucket_name, validate=False)
key = bucket.new_key(key_name)
key.set_contents_from_file(
    six.BytesIO(data),
    headers={'Expires': hdr_expires})

# Wont expose the Expires header without subclass
class Key2(Key):
    def __init__(self, bucket=None, name=None):
        super(Key2, self).__init__(bucket=bucket, name=name)
        self._expires_header = {}
    def handle_addl_headers(self, headers):
        for k, v in headers:
            if k == 'expires':
                self._expires_header = v

out = six.BytesIO()
fetched = Key2(bucket, key_name)
fetched.get_contents_to_file(out)
print 'Boto2 Expires: %r' % fetched._expires_header

with boto==2.38.0 this prints

Boto2 Expires: 'Tue,%2024%20Jan%202017%2007:52:32%20GMT'

with boto==2.45.0 this prints

Boto2 Expires: 'Tue, 24 Jan 2017 07:52:32 GMT'

The result - objects affected by that bug cannot be read by boto3.

nathan-muir on 24 Jan 2017

👍3

I have the same issue, I have images saved in S3, got the following stack traces when calling image.width.

`Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/botocore/utils.py", line 352, in parse_timestamp
return dateutil.parser.parse(value, tzinfos={'GMT': tzutc()})
File "/usr/lib/python3/dist-packages/dateutil/parser.py", line 1008, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/lib/python3/dist-packages/dateutil/parser.py", line 395, in parse
raise ValueError("Unknown string format")
ValueError: Unknown string format

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.5/dist-packages/django/core/files/images.py", line 17, in _get_width
return self._get_image_dimensions()[0]
File "/usr/local/lib/python3.5/dist-packages/django/core/files/images.py", line 27, in _get_image_dimensions
self.open()
File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/files.py", line 81, in open
self.file.open(mode)
File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/files.py", line 51, in _get_file
self._file = self.storage.open(self.name, 'rb')
File "/usr/local/lib/python3.5/dist-packages/django/core/files/storage.py", line 37, in open
return self._open(name, mode)
File "/usr/local/lib/python3.5/dist-packages/storages/backends/s3boto3.py", line 424, in _open
f = self.file_class(name, mode, self)
File "/usr/local/lib/python3.5/dist-packages/storages/backends/s3boto3.py", line 98, in __init__
self.obj.load()
File "/usr/local/lib/python3.5/dist-packages/boto3/resources/factory.py", line 505, in do_action
response = action(self, *args, kwargs)
File "/usr/local/lib/python3.5/dist-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(params)
File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 531, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py", line 168, in _send_request
request, operation_model, attempts)
File "/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py", line 233, in _get_response
response_dict, operation_model.output_shape)
File "/usr/local/lib/python3.5/dist-packages/botocore/parsers.py", line 212, in parse
parsed = self._do_parse(response, shape)
File "/usr/local/lib/python3.5/dist-packages/botocore/parsers.py", line 613, in _do_parse
member_shapes, final_parsed)
File "/usr/local/lib/python3.5/dist-packages/botocore/parsers.py", line 669, in _parse_non_payload_attrs
member_shape, headers[header_name])
File "/usr/local/lib/python3.5/dist-packages/botocore/parsers.py", line 261, in _parse_shape
return handler(shape, node)
File "/usr/local/lib/python3.5/dist-packages/botocore/parsers.py", line 150, in _get_text_content
return func(self, shape, text)
File "/usr/local/lib/python3.5/dist-packages/botocore/parsers.py", line 415, in _handle_timestamp
return self._timestamp_parser(text)
File "/usr/local/lib/python3.5/dist-packages/botocore/utils.py", line 354, in parse_timestamp
raise ValueError('Invalid timestamp "%s": %s' % (value, e))
ValueError: Invalid timestamp "Mon,%2006%20Mar%202017": Unknown string format`

dongkangli on 18 Apr 2017

I get with local DynamoDB and python 3
if isinstance(value, (int, float)):
# Possibly an epoch time.
return datetime.datetime.fromtimestamp(value, tzlocal())

here invalid timestamp
value = 1506000184.758
tzlocal() is None
....
def _naive_is_dst(self, dt):
timestamp = _datetime_to_timestamp(dt)
return time.localtime(timestamp + time.timezone).tm_isdst
timestamp=0,0
time.timezone = -7200

eino-makitalo on 21 Sep 2017

See this is probably my problem https://github.com/dateutil/dateutil/issues/197

eino-makitalo on 21 Sep 2017

I have the same issue when migrating from boto 2.39.0 to boto3 1.5.26

CommandError: An error occurred during rendering /tmp/build_81f78184cae55162a7bdaada282baf4d/wehike_app/templates/legal/canc_policy.html: Invalid timestamp "Thu,%2031%20Dec%202099%2020:00:00%20GMT": Unknown string format

I am using django-compressor and django-storages, this happens when compressing+uploading the files to S3 but I am not sure what is exactly the underlying process.

@nathan-muir may be hitting the issue.

danielskun on 12 Feb 2018

This is blocking me from upgrading my boto based project to using boto3

danielskun on 20 Jun 2018

Any solutions or updates? running into this same issue.

Dansong00 on 9 Apr 2019

@Dansong00 You can override the timestamp_parser, so it won't fail (but you wont be able to read the value).. I used something like this:

import botocore.session
from boto3.session import Session
from botocore.parsers import ResponseParserFactory
from botocore.utils import parse_timestamp

def _parse_timestamp(value):
    try:
        return parse_timestamp(value)
    except ValueError:
        return None

def get_session(**kwargs):
    response_parser_factory = ResponseParserFactory()
    response_parser_factory.set_parser_defaults(
        timestamp_parser=_parse_timestamp
    )
    botocore_session = botocore.session.get_session()
    botocore_session.register_component('response_parser_factory', response_parser_factory)
    return Session(botocore_session=botocore_session, **kwargs)

You can use the module as follows:

from .mymodule import get_session

s = get_session()
client = s.client('s3')
resource =s.resource('s3')

nathan-muir on 12 Apr 2019

👍1

@nathan-muir the method that makes the call to S3 is also part of a third party library, how do i patch this without directly making a call to get_session. what is the best approach to this patch in a global way.

Dansong00 on 29 Apr 2019

@Dansong00 Yes, you can call boto3.setup_default_session(botocore_session=botocore_session)

nathan-muir on 30 Apr 2019

@nathan-muir added another line that resolved this issue for me. In the code patch i added

botocore.parsers.DEFAULT_TIMESTAMP_PARSER = _parse_timestamp

import botocore.session
from boto3 import setup_default_session
from boto3.session import Session
from botocore import parsers
from botocore.utils import parse_timestamp


def _parse_timestamp(value):
    try:
        return parse_timestamp(value)
    except ValueError:
        return None


def get_session(**kwargs):
    response_parser_factory = parsers.ResponseParserFactory()
    response_parser_factory.set_parser_defaults(
        timestamp_parser=_parse_timestamp
    )
    botocore_session = botocore.session.get_session()
    botocore_session.register_component('response_parser_factory', response_parser_factory)
    setup_default_session(botocore_session=botocore_session)

    parsers.DEFAULT_TIMESTAMP_PARSER = _parse_timestamp

    return Session(botocore_session=botocore_session, **kwargs)

I then called get_sessions in the __init__.py of my django project. This patches the timestamp parser across the entire project. Thanks for all the help 👍

Dansong00 on 1 May 2019

👍1

Following up on this issue. Is anyone still getting the error with latest version of botocore ? If yes, please reopen a new issue. I would be happy to help.

Recently We've merged #1987 as an interim solution while dateutil/dateutil#197 remains unresolved. We'll now perform a best effort fallback to tzwinlocal if we fail to parse the timestamp on Windows. This will maintain backwards compatibility but provide a potential route to recovery for the failure users were previously experiencing.

swetashre on 27 Mar 2020

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

no-response[bot] on 3 Apr 2020

Hi @swetashre, I'm still facing the same problem (Invalid timestamp "": Unknown string format) with the boto3 version 1.14.48 and botocore version 1.17.48, that acording to https://pypi.org/ are the latest versions, when an image has the metadata Expires: 30d. Anyway I will now try to override the _parse_timestamp method to get things working and see how it goes.