Prophet: predicting over uncertain futures

[MaximeTuab]

Hi all,
assume I want to predict over multiple futures, each of them having its probability to happen.
What you be the best way to use fbprophet for the prediction?
Especially, how to manage an aggregated and reliable confidence interval over all these futures?
An other way to see that would be : how to set a certain probability for each future events (at the prediction step), and manage it thru fbprophet
Thanks a lot!!
M.

[bletham]

I'm not exactly sure what you mean about uncertain futures. I'm guessing you have some extra regressors, and there is some uncertainty in those extra regressors and you have some probability distribution over future regressor values. Is that right?

In this case, you can use m.predictive_samples(future) to draw samples from the model predictive posterior, conditioned on a particular sample of extra regressors that have been put into future. Normally, if you do m.predictive_samples(future) it will return 1000 samples of yhat. If you were to take the quantiles of these, those will be exactly the yhat_lower and yhat_upper that is produced by predict. It is 1000 samples because that is set by the kwarg uncertainty_samples when you create the model (which defaults to 1000). What you'll want to do here is do a 2-step process, where you first draw a sample for the extra regressors, and then draw a sample of yhat conditioned on that. This is the correct way to integrate over the uncertainty in the regressors. So the final process will look like:

# Create and fit a model
m = Prophet(uncertainty_samples=1)  # and any other kwargs you are using
m.fit(history)

# Sample future yhats
future = m.make_future_dataframe(365)
yhats = np.zeros((len(future), 1000))
for i in range(1000):
    # First sample the value of the extra regressors.
    regressor = sample_regressors()  # using known distribution
    # Add sample regressors to future dataframe
    future['regressor'] = regressor
    # Sample a prediction based on those
    sample = m.predictive_samples(future)
    # Store in yhats
    yhats[:, i] = sample['yhat']

# yhats has 1000 columns, and each is a sample of the future
# Get uncertainty intervals by computing quantile
yhat_lower, yhat_upper = np.quantile(yhats, q=[0.2, 0.8], axis=1)

```

[MaximeTuab]

Hey Ben, thank you for your clear answer, it fits perfectly to my needs. I play with that and give you some feedback soon. Regards,

[MaximeTuab]

After some testings on synthetics data, I can definitly say that your answer fits perfectly to my needs! Thank you again @bletham

[bletham]

Great, glad to hear it!

[ee15btech11014]

I'm not exactly sure what you mean about uncertain futures. I'm guessing you have some extra regressors, and there is some uncertainty in those extra regressors and you have some probability distribution over future regressor values. Is that right?

In this case, you can use m.predictive_samples(future) to draw samples from the model predictive posterior, conditioned on a particular sample of extra regressors that have been put into future. Normally, if you do m.predictive_samples(future) it will return 1000 samples of yhat. If you were to take the quantiles of these, those will be exactly the yhat_lower and yhat_upper that is produced by predict. It is 1000 samples because that is set by the kwarg uncertainty_samples when you create the model (which defaults to 1000). What you'll want to do here is do a 2-step process, where you first draw a sample for the extra regressors, and then draw a sample of yhat conditioned on that. This is the correct way to integrate over the uncertainty in the regressors. So the final process will look like:

# Create and fit a model
m = Prophet(uncertainty_samples=1)  # and any other kwargs you are using
m.fit(history)

# Sample future yhats
future = m.make_future_dataframe(365)
yhats = np.zeros((len(future), 1000))
for i in range(1000):
    # First sample the value of the extra regressors.
    regressor = sample_regressors()  # using known distribution
    # Add sample regressors to future dataframe
    future['regressor'] = regressor
    # Sample a prediction based on those
    sample = m.predictive_samples(future)
    # Store in yhats
    yhats[:, i] = sample['yhat']

# yhats has 1000 columns, and each is a sample of the future
# Get uncertainty intervals by computing quantile
yhat_lower, yhat_upper = np.quantile(yhats, q=[0.2, 0.8], axis=1)

@bletham Many thanks for your explanation.
Here in this case, I want to calculate exact yhat instead of yhat_lower,yhat_upper. What is the procedure I should follow?
Thanks in advance.

[bletham]

@ee15btech11014 You could replace np.quantile with np.mean to get the posterior mean.