Arviz: Question: Mixed up on Neffective calculation (and Seffective)

[canyon289]

Short Description

These questions more stem from my confusion rather than a suggestion to change anything.

For the Effective N calculation, the docstring refers to Bayesian Data Analysis 3, equation 11.8. The book describes a split chain calculation, where M (chains) is 2x the number of chains created by the sampler, because each sampling chain is split in half. In the code it does not look like ArviZ implements any chain splitting.

However it seems like ArviZ implements Seffective which is detailed on avehtari's blog in section 2.2.1

Does ArviZ implement S_Effective or N_Effective?
Do ArviZ need to update the documentation to point to the blog post rather than BDA3?
Should the calculation or method be renamed to az.s_effective?

Code Example or link

ArviZ codebase at the commit below
https://github.com/arviz-devs/arviz/commit/0c173da1bec7bfcd0936b46c4fe38b9f7e0dfaf7

Relevant documentation or public examples

Provided inline above

[canyon289]

@aloctavodia Light ping on this one. Whenever you get a chance would be helpful to understand

[ahartikainen]

For r-hat we don't do chain splitting. We should do it.

Was this for n_eff or r-hat?

https://github.com/arviz-devs/arviz/issues/361

[ahartikainen]

Also, after Stan updates and starts to recommend the rank normalized method, we should change too.

I think we can create a PR but merge it after it is the recommended way. Or all of this depends how we are going implement these functions.

[canyon289]

This was for n_effective. I think our calculation matches s_effective on Ahi's blog, and not n_effective in BDA3

Is my understanding correct?

[aloctavodia]

Hi @canyon289 thanks for the reminder. I am on vacations right now. I will check this in a few days.

[junpenglao]

It is s_effective, See https://github.com/arviz-devs/arviz/issues/141#issuecomment-407516409

[canyon289]

Can we rename this method to s_effective and update the reference in the method documentation? I'm happy to do it, just asking to make sure its the right thing

[ahartikainen]

This is still the same algorithm as the current pystan n_eff?

[canyon289]

Hm, the most specific question I have then.

Per BDA3 and a message from Aki, n_effective splits chains. Quoting from BDA3 "suppose we simulate 4 chains, ... then m=8"

And note from Aki
https://discourse.mc-stan.org/t/question-about-effective-sample-size-formulation-from-bayesian-data-analysis-3rd-edition/6697

Give that, in Stan and ArviZ I can't see where any chain splitting is done in the n_effective implementations.
https://github.com/arviz-devs/arviz/blob/master/arviz/stats/diagnostics.py#L78
https://github.com/stan-dev/pystan/blob/develop/pystan/_chains.pyx#L111

The other part that doesn't match is the -1 in the denominator

In BDA3 the denominator is something like 1+ xyz. In both pystan and ArviZ the denominiator is -1 + abc
https://github.com/stan-dev/pystan/blob/develop/pystan/_chains.pyx#L177
https://github.com/arviz-devs/arviz/blob/master/arviz/stats/diagnostics.py#L115

For these reasons it seems to me that these are not n_effective (formula 11.8) in BDA3

But instead are s_effective

Plus @junpenglao says the ArviZ implemenations is s_effective and I think he's a smart guy :)

So my direct questions are

1. Is my above statements correct? (I might be mistaken so please let me know!)

And if so

1. Why are the methods name n_effective when the calculation is called s_effective on Aki's blog?
2. Why do both ArviZ and pystan mention BDA3 formula 11.8 as their source when the code doesn't match the formula cited?

If my understanding is correct I think we should rename the method in ArviZ to s_effective and refer to Aki's blog rather than BDA3.

If my understanding is not correct I'll have more questions :(

[junpenglao]

You are correct. For what is worth, the ArviZ implementation is the same as the pymc3 implementation, that I ported from https://github.com/stan-dev/pystan/pull/415. While it should be s_effective, in pymc3 we were still following Stan naming and ref conventions.

[canyon289]

Thanks @junpenglao. Made a PR

https://github.com/arviz-devs/arviz/pull/505

[canyon289]

Question answered by https://github.com/arviz-devs/arviz/pull/505