Arrow: Negative timestamp parsing on windows: timestamp out of range for platform localtime()/gmtime() function

[bollwyvl]

Thanks for arrow!

Over on conda-forge, our build of 0.15.1 turned up this test fail on windows:

ERROR: test_parse_timestamp (tests.parser_tests.DateTimeParserParseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\bld\arrow_1568153654810\_test_env\lib\site-packages\chai\chai.py", line 63, in wrapper
    func(self, *args, **kwargs)
  File "D:\bld\arrow_1568153654810\test_tmp\tests\parser_tests.py", line 229, in test_parse_timestamp
    self.expected = datetime.fromtimestamp(negative_int_timestamp, tz=tz_utc)
ValueError: timestamp out of range for platform localtime()/gmtime() function

----------------------------------------------------------------------
Ran 385 tests in 1.109s

Excluding this on windows got things going again, but is this a cause for concern?

[jadchaar]

@bollwyvl thanks for reporting this. This is due to the new regression tests we added as part of the fix for https://github.com/crsmithdev/arrow/issues/662. It seems that datetime on Windows does not play nicely with negative timestamps (ref: https://stackoverflow.com/questions/36179914).

I will need to do some more investigating, but this should not be an issue if you are not using negative time stamps :).

[bollwyvl]

Thanks for the clarification! Merged -1 minutes from now!

On Tue, Sep 10, 2019 at 9:47 PM Jad Chaar [e-mail] wrote:

@bollwyvl https://github.com/bollwyvl thanks for reporting this. This
is due to the new regression tests we added as part of the fix for #662
https://github.com/crsmithdev/arrow/issues/662. It seems that datetime
on Windows does not play nicely with negative timestamps (ref:
https://stackoverflow.com/questions/36179914).

I will need to do some more investigating, but this should not be an issue
if you are not using negative time stamps :).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/crsmithdev/arrow/issues/668?email_source=notifications&email_token=AAALCRADHDDS7MYQE35GOA3QJBE3ZA5CNFSM4IVNUWQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6M7UKI#issuecomment-530184745,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAALCRHNHBYQSSOCOWMHWN3QJBE3ZANCNFSM4IVNUWQA
.

[jadchaar]

@systemcatch I don't think Arrow ever properly supported negative timestamps on Windows. I was able to configure Travis CI to execute a test run Windows, so we can ignore these tests on Windows for now until we find a more permanent solution.

We may need to replace all the datetime.fromtimestamp() and datetime.utcfromtimestamp() calls with this if the timestamp is negative:

        if timestamp < 0:
            sec = 0
            microsec = 0
            if isinstance(timestamp, int):
                sec = timestamp
            else:
                whole, frac = str(timestamp).split(".")
                sec = int(whole)
                microsec = int(frac) * -1
            dt = datetime(1970, 1, 1) + timedelta(seconds=sec, microseconds=microsec)
        else:
            dt = datetime.fromtimestamp(timestamp, tzinfo)

For those that may be unaware, negative timestamps are useful for getting times before the epoch (1970-01-01).

[systemcatch]

@jadchaar I just tried it on my Windows work machine and you're right it probably never worked.

(arrow) C:\Users\Chris>python
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import arrow
>>> arrow.__version__
'0.14.2'
>>> arrow.get(-15675657)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Chris\arrow\lib\site-packages\arrow\api.py", line 21, in get
    return _factory.get(*args, **kwargs)
  File "C:\Users\Chris\arrow\lib\site-packages\arrow\factory.py", line 155, in get
    return self.type.utcfromtimestamp(arg)
  File "C:\Users\Chris\arrow\lib\site-packages\arrow\arrow.py", line 166, in utcfromtimestamp
    dt = datetime.utcfromtimestamp(timestamp)
OSError: [Errno 22] Invalid argument

[jadchaar]

Yeah that is quite peculiar. I wonder why the functionality of fromtimestamp differs so drastically between operating systems.

[jadchaar]

Hey @bollwyvl, we have released 0.15.2, which includes the fix for tests on Windows :).