Arrow: Problem with ThreadPoolExecutor

[fbergero]

I seem to found an issue with concurrent calls to arrow.get.

I assumed that arrow is thread safe but this snippet fails:

from concurrent.futures import ThreadPoolExecutor
import arrow


def doIt():
    try:
        return arrow.get('2019-02-13T04:26:47-0500')
    except Exception as e:
        print(e.message)


if __name__ == "__main__":
    with ThreadPoolExecutor(max_workers=100) as executor:
        while True:
            executor.submit(doIt)
    time.sleep(100)

We are using 0.5.4 which throws:

Could not match input to any of ['YYYY-MM-DDTHH:mm:ssZ'] on '2019-02-13T04:26:47-0500'
Could not match input to any of ['YYYY-MM-DDTHH:mm:ssZ'] on '2019-02-13T04:26:47-0500'
Could not match input to any of ['YYYY-MM-DDTHH:mm:ssZ'] on '2019-02-13T04:26:47-0500'
Could not match input to any of ['YYYY-MM-DDTHH:mm:ssZ'] on '2019-02-13T04:26:47-0500'
Could not match input to any of ['YYYY-MM-DDTHH:mm:ssZ'] on '2019-02-13T04:26:47-0500'

The latest version of 0.13.1 throws an even more cryptic error:

(None, -18000)
(None, -18000)
(None, -18000)
(None, -18000)
(None, -18000)
(None, -18000)

This only happens on a very low percentage of the calls (~0.05%).

Is this a known issue? Is it arrow not thread safe?
I'm running python 2.7

[fbergero]

Any news on this?

[systemcatch]

Hi @fbergero I'm not particularly familiar with thread safety in python, but I think it would be wise to assume that arrow isn't thread safe for now.

The output from 0.13.1 doesn't look like arrow error messages, maybe they're from something else?

[Frank-Krick]

The problem is related to the timezone handling. Arrow is using tz.tzoffset which internally caches values in a dictionary. Access to the dictionary is not thread safe so in certain cases, the entry is removed and then accessed by another thread which leads to the KeyError. Example stack trace below:

  Traceback (most recent call last):
  File "/usr/src/app/metrics_results_writer/message_schema.py", line 43, in _extract_content
    timestamp = arrow.get(string_timestamp)
  File "/usr/local/lib/python2.7/site-packages/arrow/api.py", line 22, in get
    return _factory.get(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/arrow/factory.py", line 174, in get
    dt = parser.DateTimeParser(locale).parse_iso(arg)
  File "/usr/local/lib/python2.7/site-packages/arrow/parser.py", line 119, in parse_iso
    return self._parse_multiformat(string, formats)
  File "/usr/local/lib/python2.7/site-packages/arrow/parser.py", line 277, in _parse_multiformat
    _datetime = self.parse(string, fmt)
  File "/usr/local/lib/python2.7/site-packages/arrow/parser.py", line 182, in parse
    self._parse_token(token, value, parts)
  File "/usr/local/lib/python2.7/site-packages/arrow/parser.py", line 235, in _parse_token
    parts['tzinfo'] = TzinfoParser.parse(value)
  File "/usr/local/lib/python2.7/site-packages/arrow/parser.py", line 336, in parse
    tzinfo = tz.tzoffset(None, seconds)
  File "/usr/local/lib/python2.7/site-packages/dateutil/tz/_factories.py", line 40, in __call__
    cls.__strong_cache[key] = cls.__strong_cache.pop(key, instance)
  File "/usr/local/lib/python2.7/collections.py", line 157, in pop
    del self[key]
  File "/usr/local/lib/python2.7/collections.py", line 86, in __delitem__
    link_prev, link_next, _ = self.__map.pop(key)
KeyError: (None, 0)

[systemcatch]

Unfortunately since arrow is depending on dateutil's tz here, there isn't much we can do on our end.