Sktime: [BUG] Dynamic forecast using ExponentialSmoothing doesn't refit the forecaster

[pradithya]

Describe the bug

I am following dynamic forecast example but using ExponentialSmoothing instead of NaiveForecaster.
I noticed that the forecaster is not retrained in every step, thus instead of generating 36 1-step forecast it is creating 36-step forecast.

I.e.

y_pred = forecaster.update_predict(y_test, cv)

behaves like

y_pred = forecaster.predict(fh=[i for i in range(36)])

To Reproduce

forecaster = ExponentialSmoothing(trend="add", seasonal="multiplicative", sp=12)
forecaster.fit(y_train)
cv = SlidingWindowSplitter(fh=1)
y_pred = forecaster.update_predict(y_test, cv)

Expected behavior

forecaster is refitted every time step

Additional context

Versions
v0.4.2

System:
python: 3.7.3 (default, Mar 27 2019, 16:54:48) [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/aria/anaconda3/bin/python
machine: Darwin-19.4.0-x86_64-i386-64bit

Python dependencies:
pip: 19.3.1
setuptools: 41.6.0.post20191030
sklearn: 0.23.2
numpy: 1.19.0
scipy: 1.5.2
Cython: 0.29.21
pandas: 1.1.2
matplotlib: 3.3.0
joblib: 0.16.0
numba: None
pmdarima: None
tsfresh: None

[mloning]

Hi @pradithya, thanks for raising the issue!

• If you want to wholly refit the estimator, you have to iterate manually over the test data and repeatedly call fit().
• If you want to incrementally update the fitted parameters as new data comes in, you have use update_predict() with update_params=True. But note that will raise a NotImplementedError because it isn't implemented yet. Updating parameters is usually model specific. I'm not sure if there is a common way to update exponential smoothing models. For state-space models, Kalman filtering could be used, but I'm not sure if the underlying algorithms from statsmodels allows us to do this easily.
  
  
  
  • For fixed-cutoff multi-step predictions, the ExponentialSmoothing algorithms makes forecasts recursively using previously predicted values. For moving-cutoff predictions, it should instead use the new observed values. But again, not sure if we can easily implement this for the underlying statsmodel algorithm. The problem is that we do not update the self._fitted_forecaster in update() as update() is defined in the parent class here. I think we should raise a not implemented error here for now or fix it if possible.
  
  
• In addition, perhaps we should set the default of update_params to true, as this is what people may expect?
• We should update the documentation and example notebook too to explain this better. The NaiveForecaster may also be a somewhat confusing case, because it does update predictions without updating any parameters.

Let me know if you want to look into this, help would be very welcome!

[pradithya]

Hi @mloning thanks for promptly replying the issue.

If you want to incrementally update the fitted parameters as new data comes in, you have use update_predict() with update_params=True as below. But note that will raise a NotImplementedError because it isn't implemented yet. Updating parameters is usually model specific. I'm not sure if there is a common way to update exponential smoothing models. For state-space models, Kalman filtering could be used, but I'm not sure if the underlying algorithms from statsmodels allows us to do this easily.

Yeah, ExponentialSmoothing doesn't support incremental retraining with partial data. In such case is it good idea if update_predict should fallback to refit/retrain a completely new model in each step? Same goes for other model which doesn't support incremental retrain. So the promise of update_predict() is to return updated forecast as new data come and incremental retrain can be seen as a form of internal implementation / optimization of each model.

In addition, perhaps we should set the default of update_params to true, as this is what people may expect?

Yes, my expectation is the update_params in update_predict() is True by default, since the prediction without changing the model's param is already covered by predict() to some degree.

I can take a stab if you agree with above approach.

[mloning]

• I agree, the default of update_params should be set to True.
• I'm a little hesitant to fall back to refitting in update when no other method is implemented/available because this is what people expect fit is for (similar to what you argue about predict and update_predict which I agree with). Adding another kwargs like refit=False may be an option.
• The state-space formulation of ExponentialSmoothing does seem to support incremental updating, perhaps we should replace our implementation with that one.

@pradithya I'm happy if you go ahead and start changing the default of the update_params argument, let's see if that breaks any examples or unit tests!

@fkiraly I'm interested in hearing your thoughts here.

[fkiraly]

Yes, I already wanted to reply to this.

Regarding interface, @mloning :
I actually think that update should fall back to either "ignore the data", or "refit on all the data" (if available) - simply since it is a standard interface point that should be implemented by a templated class - composites may expect it, e.g., the hedge algorithm or other ensembles.

Similarly how you wouldn't return a NotImplementedError in sklearn fit_predict as long as you have a fit and predict.
Perhaps it can return a warning "update is not implemented, falling back to ..." though.

Regarding update for Holt-Winters specifically:
Isn´t Holt-Winters following as simple linear recurrence relation?
So, shouldn´t updating in-principle be somewhat straightforward if tedious to work out, at least not requiring to remember all the training data?

That's for the math/theory side - when it comes to off-shelf update functionality implemented, these do not seem to exist (in R or python). Which may hint at it not being that easy? Thoughts?

[fkiraly]

Maybe @big-o has some helpful insight on updating Holt-Winters.