Boto3: DynamoDB: boto3 doesn't work with the local server: timestamp parsing problems

[chisholm]

Software versions:

Python 2.7.15
boto3 1.7.41
botocore 1.10.41
Windows 10

Using the local server, as obtained from here.

The problem appears to be that the local server returns zero timestamps for the "LastIncreaseDateTime" and "LastDecreaseDateTime" properties of the "ProvisionedThroughput" property in the response, and boto3 chokes on it. I have tracked the problem as far as the botocore parse_timestamp function. It wants to convert to the local time zone. I am on the US eastern time zone. I think those AWS timestamps are to be interpreted as epoch seconds, i.e. seconds since 1/1/1970 00:00:00 UTC. That means a zero timestamp converts to a local date/time before the epoch. I.e. at 1/1/1970 00:00:00 UTC, in my timezone, it's still 1969, and some internal values in the dateutil implementation go negative. Some time-related functions don't like being passed a negative number.

So here's a proposed fix: don't try to convert to the local time zone. Leave them as UTC. I tried changing those tzlocal() calls to tzutc(), and voila, problem solved.

Here's a simple demo to show the problem:

import boto3
import logging

logging.basicConfig(
    level="DEBUG"
)

aws_auth = {
    "aws_access_key_id": "a_secret_key_id",
    "aws_secret_access_key": "a_secret_access_key",
    "endpoint_url": "http://localhost:8000",
    "region_name": "elbonia-1"
}

ddb = boto3.resource(
    "dynamodb",
    **aws_auth
)

my_table = ddb.create_table(
    TableName="MyTable",
    AttributeDefinitions=[
        {
            "AttributeName": "id",
            "AttributeType": "S"
        }
    ],
    KeySchema=[
        {
            "AttributeName": "id",
            "KeyType": "HASH"
        }
    ],
    ProvisionedThroughput={
        "ReadCapacityUnits": 10,
        "WriteCapacityUnits": 10
    }
)

The stack trace:

Traceback (most recent call last):
  File "C:/Programming/python/test/boto_bug/demo_bug.py", line 36, in <module>
    "WriteCapacityUnits": 10
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\boto3\resources\factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\boto3\resources\action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\client.py", line 599, in _make_api_call
    operation_model, request_dict)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\endpoint.py", line 148, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\endpoint.py", line 175, in _send_request
    request, operation_model, attempts)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\endpoint.py", line 256, in _get_response
    response_dict, operation_model.output_shape)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 242, in parse
    parsed = self._do_parse(response, shape)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 633, in _do_parse
    parsed = self._parse_shape(shape, original_parsed)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 295, in _parse_shape
    return handler(shape, node)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 560, in _handle_structure
    raw_value)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 295, in _parse_shape
    return handler(shape, node)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 560, in _handle_structure
    raw_value)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 295, in _parse_shape
    return handler(shape, node)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 560, in _handle_structure
    raw_value)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 295, in _parse_shape
    return handler(shape, node)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\parsers.py", line 577, in _handle_timestamp
    return self._timestamp_parser(value)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\utils.py", line 362, in parse_timestamp
    return datetime.datetime.fromtimestamp(value, tzlocal())
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 140, in fromutc
    return f(self, dt)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 254, in fromutc
    dt_wall = self._fromutc(dt)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 234, in _fromutc
    dtdst = enfold(dt, fold=1).dst()
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 119, in enfold
    args = dt.timetuple()[:6]
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 219, in dst
    if self._isdst(dt):
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 282, in _isdst
    if self.is_ambiguous(dt):
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 244, in is_ambiguous
    (naive_dst != self._naive_is_dst(dt - self._dst_saved)))
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
ValueError: (22, 'Invalid argument')

Here is the actual response I got from the local Dynamo server

{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            }
        ],
        "TableName": "MyTable",
        "KeySchema": [
            {
                "AttributeName": "id",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": 1529528737.279,
        "ProvisionedThroughput": {
            "LastIncreaseDateTime": 0.000,
            "LastDecreaseDateTime": 0.000,
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 10,
            "WriteCapacityUnits": 10
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/MyTable"
    }
}

[JordonPhillips]

I'm not able to reproduce this issue. What is your system locale set to?

[chisholm]

I can give some details of the Win10 OS configuration. It warns me that "Some settings are hidden or managed by your organization". I'm a member of a windows domain.

"Date & Time" settings:

• Set time zone automatically: Off
• Time zone: (UTC-05:00) Eastern Time (US & Canada)
• Adjust for daylight saving time automatically: On

"Region & language" settings:

• Country or region: United States
• Languages: English (United States)

From within Python:

>>> import locale
>>> locale.getdefaultlocale()
('en_US', 'cp1252')
>>> locale.getlocale()
(None, None)
>>> import time
>>> time.timezone
18000
>>> time.tzname
('Eastern Standard Time', 'Eastern Daylight Time')

(Anything else that would be helpful?)

The root problem is easy to reproduce though:

(boto_bug_venv_py2) C:\Programming\python\test>python
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import dateutil.tz
>>> import datetime
>>> datetime.datetime.fromtimestamp(0, dateutil.tz.tzlocal())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 140, in fromutc
    return f(self, dt)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 254, in fromutc
    dt_wall = self._fromutc(dt)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 234, in _fromutc
    dtdst = enfold(dt, fold=1).dst()
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 119, in enfold
    args = dt.timetuple()[:6]
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 219, in dst
    if self._isdst(dt):
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 282, in _isdst
    if self.is_ambiguous(dt):
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 244, in is_ambiguous
    (naive_dst != self._naive_is_dst(dt - self._dst_saved)))
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
ValueError: (22, 'Invalid argument')

Those stack frames match up with the last 8 in the above stacktrace.

Here is an example invoking botocore directly:

>>> import botocore.utils
>>> botocore.utils.parse_timestamp(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\botocore\utils.py", line 362, in parse_timestamp
    return datetime.datetime.fromtimestamp(value, tzlocal())
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 140, in fromutc
    return f(self, dt)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 254, in fromutc
    dt_wall = self._fromutc(dt)
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 234, in _fromutc
    dtdst = enfold(dt, fold=1).dst()
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\_common.py", line 119, in enfold
    args = dt.timetuple()[:6]
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 219, in dst
    if self._isdst(dt):
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 282, in _isdst
    if self.is_ambiguous(dt):
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 244, in is_ambiguous
    (naive_dst != self._naive_is_dst(dt - self._dst_saved)))
  File "C:\Programming\python\test\boto_bug_venv_py2\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
ValueError: (22, 'Invalid argument')

Same stack trace again (except for the top frame).

[chisholm]

Here's what I think the OS-level underlying problem is. So this is in software layers beneath Python now. The OS' localtime() function behaves differently on Windows and Linux. Maybe you couldn't reproduce the error because you weren't trying to reproduce it on Windows? Microsoft's localtime() documentation says:

These functions validate their parameters. If timer is a null pointer, or if the timer value
is negative, these functions invoke an invalid parameter handler, as described in Parameter
Validation. If execution is allowed to continue, the functions return NULL and set errno to
EINVAL.

I'm guessing EINVAL's value is 22. I verified the error behavior with the following short C program:

#include <errno.h>
#include <stdio.h>
#include <time.h>

int main(void)
{
    struct tm *result;
    time_t t = -18000;

    result = localtime(&t);
    if (result == NULL)
        printf("Got errno: %d\n", errno);
    else
        printf("Success!\n");

    return 0;
}

If you compile and run this on Linux (I used a CentOS7 VM) and Windows (again, I am on Windows 10, and I used MinGW-w64 as the compiler since I don't have Visual Studio), you will get different results. You'll get the "Success" message on Linux, and Got errno: 22 on Windows.

I can verify via my Python debugger, that the code at the bottom frame of the above stack traces:

return time.localtime(timestamp + time.timezone).tm_isdst

Is calling time.localtime() with a negative value: timestamp equals -21600, and as mentioned above, time.timezone for me equals 18000. Their sum is -3600, which is negative.

dateutil's tzlocal() implementation depends on Python's time module, and the behavior of Python's time module most likely depends on the underlying OS's behavior with regard to certain time-related functions. Differences in OS behavior will therefore bubble up to the time module, to the dateutil tzlocal implementation, and therefore up to boto3. So it seems like boto3 is currently unusable with the local Dynamo server on Windows.

Again, seems like the simplest fix is to not try to convert these timestamps to local time. Just stick with UTC.

[joguSD]

Unfortunately, we can't remove converting to local time as that would be a breaking change.

As a workaround you can either set your localtime to UTC or add the following before importing boto3 in your script:

import os
os.environ["TZ"] = "UTC"

Let me know if that helps.

[chisholm]

Yeah, adding the Python snip seems to work around the crash problem. At the cost of the timestamps being wrong relative to my actual local time zone. The parsed provisioned throughput settings on the table returned from the create_table action are:

{u'LastDecreaseDateTime': datetime.datetime(1970, 1, 1, 0, 0, tzinfo=tzlocal()),
 u'LastIncreaseDateTime': datetime.datetime(1970, 1, 1, 0, 0, tzinfo=tzlocal()),
 u'NumberOfDecreasesToday': 0,
 u'ReadCapacityUnits': 10,
 u'WriteCapacityUnits': 10}

So there's still a problem, but at least it's usable!

[joguSD]

The python snippet just signifies that UTC is your localtime. So I think the timestamps are actually correct (they're the epoch in UTC).

[chisholm]

Yeah, but it's not my real local time zone :) That output says tzinfo=tzlocal() which would lead the casual reader to think that date and time is in their local time zone, and cause confusion. It's a sneaky change to make "tzlocal" mean the same as "tzutc" (as far as I can tell). It's basically implementing my suggested fix in an uglier way which doesn't require any boto3 code changes, but affects all software components which work with dates and times. A better fix would specifically target boto3. As I'm sure you're aware. People shouldn't have to do this. Anyway, it will be valuable for people to at least stop the crashing and be able to use the library.

[r4nyc]

To fix this make the following change to parse_timestamp in botocore\utils.py

def parse_timestamp(value):
"""Parse a timestamp into a datetime object.
Supported formats:
* iso8601
* rfc822
* epoch (value is an integer)
This will return a datetime.datetime object.
"""
if isinstance(value, (int, float)):
# Possibly an epoch time.
#NEW CODE
**min_time = time.timezone * 2
value = min_time if value < min_time else value

END NEW CODE**

    return datetime.datetime.fromtimestamp(value, tzlocal())

[chisholm]

I suspect your approach is to ensure epoch seconds values are sufficiently large to prevent variables from going negative inside the tzlocal implementation? My reaction:

• I think parse_timestamp() should either return an accurate datetime instance or raise an exception. I don't think it should silently change the timestamp (which will happen if you "clamp" it like that).
• I think parse_timestamp() should not raise an exception for any timestamp obtained from an AWS service, including the local dynamo service (barring possible service bugs). I'm no AWS expert, so maybe there are conditions under which a service can return some crazy values. I am assuming here, possibly incorrectly, that AWS services return "reasonable" values for timestamps. I suspect that means it should at least succeed for any non-negative epoch seconds value.
• The above two bullets should hold no matter which time zone a user is in, for all supported operating systems. Note that time.timezone is negative in many parts of the world (I checked the Python docs), so the above fix doesn't ensure a sufficiently large value in all parts of the world. It could still go negative. So I don't think it works.
• If it did work, hypothetically speaking, I think it would be more concise and understandable to use Python's built-in max() function, e.g. value = max(time.timezone * 2, value). My 2¢ :)

[r4nyc]

AWS libraries often pass in the int 0 when the value isn't set. That may not seem like a crazy value but it is enough to cause an exception on Windows if your timezone causes time.timezone tor return a positive value.

Internally fromtimestamp tries subtracting the time.timezone from the time when it is determining if daylight savings time is in effect. The library python calls on windows can't handle an epoch value less than 0. Setting the min_time to time.timezone * 2 guarantees the value will not go negative as it tries variations,

You can see the error easily on a Windows machine with the following code
import datetime
from dateutil.tz import tzlocal
datetime.datetime.fromtimestamp(0, tzlocal())

site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument

Without a patch this makes many boto scripts unrunnable on windows machines unless you change yoru environment to UTC.

[jrbrodie77]

I ran into this same bug on Win 10 with Python 3.7. Problem happens within tz.py which appears consistent with discussion above. I'm internal to amazon if anyone wants to contact me. brodiej@

[AndrewCi]

Any thoughts on r4nyc's solution? I am also running into this issue on Windows 10 and none of the fixes worked except for r4nyc, but I an weary of making changes to the botocore.

[SimonFrost-Arm]

r4nyc's solution doesn't work in the UK (timezone 0). The catch all soln is a change in tz.py (I found returning False if time is the epoch in is_ambiguous() works well).
However, for a botocore workaround, the following works:
in Utils.py, parse_timestamp()

if isinstance(value, (int, float)):
# Possibly an epoch time.
if value + time.altzone < 0:
return datetime.datetime.utcfromtimestamp(int(value))
else:
return datetime.datetime.fromtimestamp(value, tzlocal())

[Loboterreno]

Been experiencing the same issue following a local Dynamodb tutorial on my windows 10 machine. SimonFrost solution corrected the issue.

[indexfingers]

Been experiencing the same issue following a local Dynamodb tutorial on my windows 10 machine. SimonFrost solution corrected the issue.

Likewise

[phylroy]

The fix that @r4nyc seems to work. Why not add it to the repo?

[dmitry-weirdo]

Neither of the fixes worked for me (see my problem description here).
Should I perform some python files recompilation after changes in C:\Python\Python38\Lib\site-packages\botocore\utils.py?

[swetashre]

We've merged #1987 as an interim solution while dateutil/dateutil#197 remains unresolved. We'll now perform a best effort fallback to tzwinlocal if we fail to parse the timestamp on Windows. This will maintain backwards compatibility but provide a potential route to recovery for the failure users were previously experiencing.

Is anyone still getting the error with the latest version of botocore ? If yes, please open a new issue. We would be happy to help.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.