Prophet: Error metrics and visualizations for diagnostics

[Ic3fr0g]

First off, congratulations on creating a wonderful tool for forecasting. I was wondering if there could be a functionality within prophet that calculates the accuracy on the lines of Rob J Hyndman's forecast package. I for one would love to see this functionality.

[Ic3fr0g]

Follow up question, how does one go calculating metrics like MASE or MAPE using your tool ?
Here was my naive approach to reconcile the two packages and to calculate MASE :

m <- prophet(df,yearly.seasonality = T,weekly.seasonality = T)
# m$growth = "linear"
future <- make_future_dataframe(m, periods = 365)
fp <- predict(m, future)[,c('yhat')]
fp_fit <- ts(fp[1:(length(fp)-364)],frequency = 365)
fp_hist <- msts(m$history[,'y'],seasonal.periods = c(7,365.25))
fp_mean <- ts(fp[length(fp)-364:0],frequency = 365,start = end(fp_hist))
fprophet <- list(mean = fp_mean,fitted = fp_fit,x = fp_hist)
class(fprophet) <- "forecast"
accuracy(fprophet)
#                   ME     RMSE      MAE MPE MAPE      MASE      ACF1
#Training set 40.84288 252152.8 186680.3 NaN  Inf 0.3441319 0.4190127

# Which is the same thing as doing - 
# d  - An integer indicating the number of lag-1 differences to be used
# for the denominator in MASE calculation. Default value is 1 for non-seasonal
# series and 0 for seasonal series.
# D - An integer indicating the number of seasonal differences to be used
# for the denominator in MASE calculation. Default value is 0 for non-seasonal
# series and 1 for seasonal series.
D = 1
d = 0
nd <- diff(fp_hist,lag = round(tsp(fp_hist)[3L]), differences = D)
scale <- mean(abs(nd), na.rm = TRUE)
res <- fp_hist - fp_fit
mase <- mean(abs(res/scale), na.rm=TRUE)
# mase = 0.3441319

Is this valid ? Or is there another way to validate the accuracy metrics of the prophets models ? I could upload the data for df if it interests you. I suppose it is accurate only for linear growth?

[slavakurilyak]

Try sklearn's regression metrics (example: MAE) or keras' loss functions (example: MAPE)

[Ic3fr0g]

Hmm what about other metrics for model estimation like AIC, R squared and sigma

[slavakurilyak]

Consider benhamner/Metrics, which has both Python and R implementations for common evaluation metrics

[bletham]

This would be a great feature.

[bletham]

PR in #257 and #261

[bletham]

Functions for cross-validation to estimate forecast accuracy are now added to Py version in https://github.com/facebookincubator/prophet/commit/79d0793ce4ad5bdf3a3b0b37b468302f3ee6683e. A big thanks to @teramonagi for that!

[bletham]

Added to R in https://github.com/facebookincubator/prophet/commit/3c09448018494b2f20268a3c4a7d1130f75da2e2. These new functions (cross_validation and simulated_historical_forecasts) return a dataframe with forecasted (yhat) and actual (y) values with forecasts made from a bunch of historical cutoff dates. This could then be used to compute whatever error metric is desired.

[zippeurfou]

I think AIC would be worth to implement as it is not that straight forward and benhamner/Metrics do not have it.

[bletham]

Agreed. The cross-validation function is now available in v0.2 on CRAN and pypi, but we need to add now error metrics and visualization functions. I'm going to leave this issue open in the meantime.

[bletham]

Computing and visualizing error metrics is now a feature in v0.3, as documented here: https://facebook.github.io/prophet/docs/diagnostics.html

[sriramkraju]

https://facebook.github.io/prophet/docs/diagnostics.html

Getting an error: ImportError: cannot import name 'performance_metrics'

# Python
from fbprophet.diagnostics import performance_metrics
df_p = performance_metrics(df_cv)
df_p.head()

---

ImportError Traceback (most recent call last)
in ()
----> 1 from fbprophet.diagnostics import performance_metrics
2 df_p = diagnostics.performance_metrics(df_cv)
3 df_p.head()

ImportError: cannot import name 'performance_metrics'

[bletham]

@sriramkraju can you check that you have v0.3 installed?

import fbprophet
fbprophet.__version__

[sriramkraju]

Thanks for the reply. I am having v 0.2.

How do I upgrade it to v 0.3?

[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
09/24/18,
8:33:11 PM

--

Thanks and Regards, Sriram

On Mon, Sep 24, 2018 at 8:16 PM, Ben Letham [e-mail]
wrote:

@sriramkraju https://github.com/sriramkraju can you check that you have
v0.3 installed?

import fbprophet
fbprophet.__version__

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebook/prophet/issues/194#issuecomment-424175228,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Af2X50vLUNNqYesnVQa5XdYxYCNVKYJqks5ueYP2gaJpZM4NeCDr
.

[sriramkraju]

I did a pip upgrade to 0.3, it works thanks!

[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
09/24/18,
8:44:50 PM

--

Thanks and Regards, Sriram

On Mon, Sep 24, 2018 at 8:36 PM, Sriram Krishnaraju [e-mail]
wrote:

>

Thanks for the reply. I am having v 0.2.

How do I upgrade it to v 0.3?

[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5& Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5& 09/24/18,
8:33:11 PM

--

Thanks and Regards, Sriram

On Mon, Sep 24, 2018 at 8:16 PM, Ben Letham [e-mail]
wrote:

@sriramkraju https://github.com/sriramkraju can you check that you
have v0.3 installed?

import fbprophet
fbprophet.__version__

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebook/prophet/issues/194#issuecomment-424175228,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Af2X50vLUNNqYesnVQa5XdYxYCNVKYJqks5ueYP2gaJpZM4NeCDr
.

[AntonioBlago]

I did a pip upgrade to 0.3, it works thanks! [image: Mailtrack] https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5& Sender notified by Mailtrack https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5& 09/24/18, 8:44:50 PM
…
-- Thanks and Regards, Sriram On Mon, Sep 24, 2018 at 8:36 PM, Sriram Krishnaraju [e-mail] wrote:
Thanks for the reply. I am having v 0.2. How do I upgrade it to v 0.3? [image: Mailtrack] https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5& Sender notified by Mailtrack https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5& 09/24/18, 8:33:11 PM -- Thanks and Regards, Sriram On Mon, Sep 24, 2018 at 8:16 PM, Ben Letham @.*> wrote: > @sriramkraju https://github.com/sriramkraju can you check that you > have v0.3 installed? > > import fbprophet > fbprophet.__version__ > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#194 (comment)>, > or mute the thread > https://github.com/notifications/unsubscribe-auth/Af2X50vLUNNqYesnVQa5XdYxYCNVKYJqks5ueYP2gaJpZM4NeCDr > . >

How can you perform an pip upgrade upgrade to 0.3? I tried it but it doesnt work.

[bletham]

pip install fbprophet --upgrade should do it

[AntonioBlago]

pip install fbprophet --upgrade should do it

thanks @bletham

[ivonnics]

Hi!
I just installed Prophet, however, when I type:
from fbprophet import Prophet
I get this error:
ImportError: cannot import name 'MONDAY'
I runned: pip install fbprophet --upgrade before asking the question, and even with that I get the same error.
Any suggestion on how to overcome this error, would be extremelly welcome...

[bletham]

@ivonnics this issue is described in #796. In short, it is from a change that the holidays package made last week and can be fixed by downgrading that package as described in #796.

[ivonnics]

Great news @bletham ....
Tested and I can now give testimony and confirm that Prophet 0.4 is running well!!!!
:smile: :grin: :laughing: