Pandas: need help to clarify ewma() using adjust =TRUE

Created on 19 Nov 2014 · 6Comments · Source: pandas-dev/pandas

Hi!

Can someone please help me understand how adjust = True in the function ewma() works mathematically?
I understand the idea of simple exponential smoothing and it's basic formula when adjust = FALSE,
then alpha is constant, as in the equation:
daum_equation_1416434433349
When adjusted = True then I assume that, alpha does not stay constant and it is adjusted "automatically".But what is the mathematical representaiton for such case? I've looked into Pandas documentation
http://pandas.pydata.org/pandas-docs/stable/computation.html#exponentially-weighted-moment-functions
and there it says that when it is adjusted, then the function uses this weights:
daum_equation_1416434874359

It is still not clear to me how this adjustment is working.

The only simple exponential method using an adaptive/adjusting alpha that I know of, is from Rob Hyndman's book: "Forecasting Methods and Application" .There I found a variation of the simple exponential smoothing (p.155-157), where alpha is adjusted automatically, called "adaptive-respose single exponential smoothing" (ARRSES). Unfortunately I got different results from each, so I guess this is not the one.

Does anybody know how to explain this specific case mathematically? or knows a good reference?

Thanks!

Algos Docs Numeric

Source

aviPython

👍1

Most helpful comment

I never liked the term "adjustment" -- which predates me.

Suppose you had a series with an infinite history: x[t], x[t-1], x[t-2], etc.
Then, assuming convergence, you could define:

(1) y[t] = (x[t] + (1-a)*x[t-1] + (1-a)^2*x[t-2] + ...) / (1 + (1-a) + (1-a)^2 + ...)
         = (x[t] + (1-a)*x[t-1] + (1-a)^2*x[t-2] + ...) * a

(since the denominator is equal to 1 / (1 - (1-a)) = 1/a).

Now it's pretty easy to see that the y[t] satisfy:

(2) y[t] = a*x[t] + (1-a)*y[t-1].

So far so good. Now in reality of course we do not have series with infinite histories. So how do we define the exponentially weighted averages for finite series? There are (at least) two plausible definitions:

The first option, which in Pandas is specified using adjust=True, is to follow (1) above and define:

(1) y[t] = (x[t] + (1-a)*x[t-1] + (1-a)^2*x[t-2] + ... + (1-a)^n*x[t-n]) /
           (1 + (1-a) + (1-a)^2 + ... + (1-a)^n).

The second option, which in Pandas is specified using adjust=False, is to follow (2) above and define:

(2) (i)  y[0] = x[0]; and
    (ii) y[t] = a*x[t] + (1-a)*y[t-1] for t>0.

These two methods will in general produce different results (though the difference will decrease the farther out one goes). Personally I find (1) more intuitive -- and this is the default in Pandas.

Other than the unfortunate choice of the parameter name adjust, I think the docs are now reasonably clear about the two options. But by all means feel free to submit a PR adding clarifications...

I am not familiar with "adaptive-repose single exponential smoothing", but suspect that that is something different.

seth-p on 20 Nov 2014

👍12

All 6 comments

cc @seth-p

jreback on 19 Nov 2014

I never liked the term "adjustment" -- which predates me.

Suppose you had a series with an infinite history: x[t], x[t-1], x[t-2], etc.
Then, assuming convergence, you could define:

(1) y[t] = (x[t] + (1-a)*x[t-1] + (1-a)^2*x[t-2] + ...) / (1 + (1-a) + (1-a)^2 + ...)
         = (x[t] + (1-a)*x[t-1] + (1-a)^2*x[t-2] + ...) * a

(since the denominator is equal to 1 / (1 - (1-a)) = 1/a).

Now it's pretty easy to see that the y[t] satisfy:

(2) y[t] = a*x[t] + (1-a)*y[t-1].

The first option, which in Pandas is specified using adjust=True, is to follow (1) above and define:

(1) y[t] = (x[t] + (1-a)*x[t-1] + (1-a)^2*x[t-2] + ... + (1-a)^n*x[t-n]) /
           (1 + (1-a) + (1-a)^2 + ... + (1-a)^n).

The second option, which in Pandas is specified using adjust=False, is to follow (2) above and define:

(2) (i)  y[0] = x[0]; and
    (ii) y[t] = a*x[t] + (1-a)*y[t-1] for t>0.

Other than the unfortunate choice of the parameter name adjust, I think the docs are now reasonably clear about the two options. But by all means feel free to submit a PR adding clarifications...

I am not familiar with "adaptive-repose single exponential smoothing", but suspect that that is something different.

seth-p on 20 Nov 2014

👍12

Thank you very much for your effort in explaining this into details! It is now much more clear to me!
Yea I guess that the name "adjust" confused me a bit..

aviPython on 20 Nov 2014

@seth-p if you'd like to do a PR to add this somewhere in the docs.......

jreback on 20 Nov 2014

I'd go even further and say that the unadjusted formula is not only less intuitive, but incorrect for finite series. In order to use it you have to assume that "weight" series is already infinite-ish and the first element is not an ordinary value, but rather a prior knowledge of the whole infinite-ish series. Intuitively this gives the first value much more weight and that corresponds to the maths.

This is a valid scenario, for example, when you're calculating ewma for some time series data period-by-period and you are rolling over to the next period. This is when you'd prepend the weighted last element to the next period series changing it from

t[period+1, 0]: v0
t[period+1, 1]: v1
...

t[period, N-1]: ewma(t[i, N-1])
t[period+1, 0]: v0
t[period+1, 1]: v1

and use the "unadjusted" formula.

Maybe the kwarg could be renamed to "assume_infinite_history" or something alike. Or, going more radical, maybe we could use replace it with prior kwarg that, if not None, chooses the unadjusted formula with specified value as prior knowledge.