Redash: OverflowError: date value out of range in should_schedule_next

[rueian]

Hi,

My redash never refresh all my queries. And I found that there was an "OverflowError: date value out of range" exception raised at:
https://github.com/getredash/redash/blob/v1.0.3/redash/models.py#L703

    if failures:
        next_iteration += datetime.timedelta(minutes=2**failures)

I think maybe it should check the date value range before applying the delta.

[hhschu]

Does the failure counts increase when I manually click the refresh button? If so, it might result in a big enough (somewhere around 30) to cause the datetime to overflow.

[kitsuyui]

Ya, In my office it happened too.
It attempts to create scheduling 1.5 * 10 ^ 23 years future! :)
And it causes stopping all scheduled query job.

I suppose some fixes:

1. If schedule_failures is huge, cut off it like this: XXX

max_retry_after = 60  # 1 hour
if failures:
    retry_after = min(max_retry_after, 2 ** failures)
    next_iteration += datetime.timedelta(minutes=retry_after)

1. When found huge schedule_failures, abandon its scheduling.
2. Show queries that have huge schedule_failures for admin dashboard.

[arikfr]

What I fail to understand is how this gets reproduced. Considering we're using exponential back off, to get to value of failures = 30 it should pass 1073741822 minutes. Unless you all traveled back in time from the year 4059, something here doesn't make sense :-)

I don't mind applying the fix in #2057, but would really love to understand how this happens to make sure we don't cover a deeper issue.

[kitsuyui]

@arikfr
https://github.com/getredash/redash/blob/master/redash/tasks/queries.py
ya, as far as reading the source code, it seems not likely to be incremented just as you say.

In my case, schedule_failures was 96.
2 ** 96 [minutes] / 60[minutes/hours] / 24[hours/days] / 365 [days/years] = 1.507e+23 [years] .

I guess the worker increments when it restarted after the error has occurred.

But I have no evidence to reproduce.

[guzhenping]

l also have this question。。。

  [2018-06-19 11:49:21,987][PID:11][ERROR][MainProcess] Task redash.tasks.refresh_queries[ca828db2-f9ca-4c7e-aee3-f38e2718ee63] raised unexpected: OverflowError('date value out of range',)

And it causes stopping all scheduled query job.

How to fix ?

[RichardLitt]

@guzhenping Is there any chance you can explain how to reproduce this effectively? That's what's keeping this issue open, at the moment.

[guzhenping]

Hi, @RichardLitt !

l upgraded redash from v4.0.0 to v4.0.1.Just change the model of postgresql tables: python manage.py db upgrade. And then, l build the npm package and start the new redash server.

l use the old connection of postgresql for new redash. Maybe my method of upgrade is wrong.

[guzhenping]

By now, l use the code writed by @kitsuyui to fix the bug. It works.

[captn3m0]

Facing this as well:

[2018-08-01 13:45:33,583][PID:1][ERROR][MainProcess] Task redash.tasks.refresh_queries[aed8e7a3-87c5-48e8-aeaa-7b650cc8e0cc] raised unexpected: OverflowError('date value out of range',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/app/redash/worker.py", line 71, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/redash/tasks/queries.py", line 275, in refresh_queries
    for query in models.Query.outdated_queries():
  File "/app/redash/models.py", line 990, in outdated_queries
    if should_schedule_next(retrieved_at, now, query.schedule, query.schedule_failures):
  File "/app/redash/models.py", line 838, in should_schedule_next
    next_iteration += datetime.timedelta(minutes=2**failures)
OverflowError: date value out of range

We did an upgrade recently from 1.0.1 to 4.0.1+b4038

[arikfr]

Fixed in #3899.