Prophet: TypeError: data type "datetime" not understood pandas==0.18.1

[TheLeanAcademic]

Sorry, newbie problem. I've printed the head of my dataframe and the dtype of the 'ds' column with the error. Any advice on how I can fix this would be much appreciated.

I've tried df['ds'] = pd.to_datetime(df['ds']). No idea how to solve it.
Many thanks.

[bletham]

I'm not sure what's happening here. Is it possible for you to attach your data csv so I can try to replicate?
Also what version of pandas do you have?

[mitbal]

I've encountered the same issue. It has been solved now by updating pandas into newer version (0.20.3 to be exact).

[bletham]

This seems to be resolved. I think the current requirement of 0.18.1 should be sufficient but if someone runs into this issue with that version or higher we can bump the requirement to 0.20.3.

[iskandarshabaev]

I have the same issue with pandas v0.21.0

[bletham]

Something else must be going on here. I'm on pandas v0.20.3 and have not yet run into this issue:

import pandas as pd
from fbprophet import Prophet
df = pd.DataFrame({
  'ds': ['2014-06-23', '2014-06-24', '2014-06-25', '2014-06-26', '2014-06-27'],
  'y': [4.39, 4.23, 3.78, 2.99, 3.04],
})
df['ds'] = pd.to_datetime(df['ds'])
m = Prophet()
m.fit(df)

@iskandarshabaev can you give some steps for reproducing?

[rtkaleta]

Could it be related to the version of numpy you have installed alongside pandas?

I am on pandas==0.18.1 and started to get this error when I bumped numpy from 1.21.1 to 1.13.3.

Steps to reproduce:

samples['days'] = (samples['today'] - samples['before'])\
    .apply(lambda delta: delta.days)

where today and before are both datetime.datetime like

from dateutil.parser import parse
before = parse('2015-11-16 21:57:24')

[kerenya]

Hi,
I had the same issue - when upgrading numpy from 1.12.1 to 1.13 and newer, I get the same error.
when I downgrade it is getting back to be working.
thanks!

[bletham]

Thanks for helping debug this, there seems to be a pandas error specific to combinations of pandas and numpy versions:
https://stackoverflow.com/questions/44593036/error-returned-when-subtracting-datetime-columns-in-pandas

I am trying to reproduce this with this code:

import pandas as pd
from fbprophet import Prophet
df = pd.DataFrame({
  'ds': ['2014-06-23', '2014-06-24', '2014-06-25', '2014-06-26', '2014-06-27'],
  'y': [4.39, 4.23, 3.78, 2.99, 3.04],
})
df['ds'] = pd.to_datetime(df['ds'])
m = Prophet()
m.fit(df)

I have tried in:
Python 2, pandas 0.20.3, numpy 1.13.3, fbprophet 0.2.1
Python 3, pandas 0.21.0, numpy 1.13.3, fbprophet 0.2.1
and oddly haven't run into it yet.

On the stackoverflow it was suggested this might be Windows-specific, is everyone here using Windows?

Could someone who has this issue verify that the above code reproduces it?
If not can you construct a dataframe that does?

Thanks!

[kerenya]

Hi,
I am working on Mac os :Sierra .
python 2.7
pandas 0.18.1 (I cannot upgrade unfortunately, note that this was the pandas version in the last two comments mentioning the bug )
numpy 1.13.3

I am running this code:
scale=0.05
model = Prophet(changepoint_prior_scale=scale)
model.fit(y_train)

here are the first 29 rows of the data which also satisfies the error I get:
ds y
0 2012-01-01 5.370638
1 2012-01-02 5.411646
2 2012-01-03 4.127134
3 2012-01-04 4.672829
4 2012-01-05 4.330733
5 2012-01-06 4.406719
6 2012-01-07 4.919981
7 2012-01-08 4.007333
8 2012-01-09 4.418841
9 2012-01-10 4.564348
10 2012-01-11 5.153292
11 2012-01-12 3.258097
12 2012-01-13 4.442651
13 2012-01-14 4.691348
14 2012-01-15 5.342334
15 2012-01-16 4.820282
16 2012-01-17 4.477337
17 2012-01-18 3.610918
18 2012-01-19 3.951244
19 2012-01-20 4.110874
20 2012-01-21 4.442651
21 2012-01-22 4.442651
22 2012-01-23 4.110874
23 2012-01-24 3.806662
24 2012-01-25 4.043051
25 2012-01-26 3.401197
26 2012-01-27 3.367296
27 2012-01-28 4.262680
28 2012-01-29 4.736198
29 2012-01-30 2.833213

Cheers,
Keren

[bletham]

@kerenya Thanks, let me try downgrading my pandas and see if I can reproduce then. In the meanwhile, can you verify if this code produces the error:

import pandas as pd
from fbprophet import Prophet
df = pd.DataFrame({
  'ds': ['2014-06-23', '2014-06-24', '2014-06-25', '2014-06-26', '2014-06-27'],
  'y': [4.39, 4.23, 3.78, 2.99, 3.04],
})
df['ds'] = pd.to_datetime(df['ds'])
m = Prophet()
m.fit(df)

[bletham]

OK I was able to replicate this issue with pandas 0.18.1 and numpy 1.13.3. It is an issue in those packages which has been fixed with later pandas versions. I'll try and find a workaround for those that can't upgrade pandas.

[renelikestacos]

@bletham hey thanks for your suggestions, i updated to 0.22 pandas, 1.9 and it seems to work.

[bletham]

Here is a simple repro for the issue:

import pandas as pd
df = pd.DataFrame({'ds': ['2014-06-23', '2014-06-24']})
df['ds'] = pd.to_datetime(df['ds'])
t = df['ds'] - df['ds'].min()

This errors in numpy 1.13.3 with pandas versions 0.18.1 through 0.19.2. It works with pandas 0.17.1, or with 0.20.1 and higher.

I wasn't able to find a clean workaround in 0.18.1. I'm going to go ahead and increase the requirement to 0.20.1 (in https://github.com/facebook/prophet/commit/7e170ffba5ab7b433f0b56ff5d7814e67312be2d). It's a year old now so I hope it will be not be an issue for most people to meet this requirement.

[bletham]

This should no longer be an issue in v0.3.

[dimoibiehg]

@bletham I have the same problem with the following config (On Linux):

Python 2.7
Name: numpy Version: 1.14.0
Name: pandas Version: 0.23.3
Name: fbprophet Version: 0.3.post2

[bletham]

@dimoibiehg I do not run into this same issue in Python 2.7.13, numpy 1.14.0, pandas 0.23.3. Can you verify that this code produces the issue for you?

import pandas as pd
df = pd.DataFrame({'ds': ['2014-06-23', '2014-06-24']})
df['ds'] = pd.to_datetime(df['ds'])
t = df['ds'] - df['ds'].min()

If not can you provide some code that does produce it?

[dimoibiehg]

The Code is as you've mentioned:

import pandas as pd
from fbprophet import Prophet
df = pd.DataFrame({
  'ds': ['2014-06-23', '2014-06-24', '2014-06-25', '2014-06-26', '2014-06-27'],
  'y': [4.39, 4.23, 3.78, 2.99, 3.04],
})
df['ds'] = pd.to_datetime(df['ds'])
m = Prophet()
m.fit(df)

[bletham]

Hm that code runs for me without error, in Linux, Py 2.7.13, numpy 1.14.0, pandas 0.23.3, fbprophet 0.3.post2.

Could you try a --no-cache reinstall of pandas perhaps? Otherwise I'm really not sure what could be causing this. Did the code that in my previous comment produce the error?

does pd.to_datetime(df['ds']) produce the error?