Libertem: tests/template/test_radial_fourier_template.py spurious failures

[sk1p]

I sometimes get a failure like this when running the tests locally:

>       assert np.allclose(results["dominant_0"], expected["dominant_0"].raw_data)
E       assert False
E        +  where False = <function allclose at 0x7f7633a078b0>(array([[10,  2,  2,  2, 22],\n       [ 2,  4,  8,  8, 22],\n       [ 2,  4,  8,  8,  2],\n       [ 2,  4,  0,  8,  0],\n       [ 0,  8,  2,  4,  0]]), array([[10,  2,  2,  2,  2],\n       [ 2,  4,  8,  8, 22],\n       [ 2,  4,  8,  8,  2],\n       [ 2,  4,  0,  8,  0],\n       [ 0,  8,  2,  4,  0]]))
E        +    where <function allclose at 0x7f7633a078b0> = np.allclose
E        +    and   array([[10,  2,  2,  2,  2],\n       [ 2,  4,  8,  8, 22],\n       [ 2,  4,  8,  8,  2],\n       [ 2,  4,  0,  8,  0],\n       [ 0,  8,  2,  4,  0]]) = <AnalysisResult: dominant_0>.raw_data

The test code here looks suspicious:

https://github.com/LiberTEM/LiberTEM/blob/bc74cd74d55702962b566eeb0b3cd78c6550c8c9/tests/template/test_radial_fourier_template.py#L36-L54

I think the analysis-creation should be outside of the loop, as the analysis doesn't depend on the channel. But that won't solve the problem at hand, I think. Can you comment on this, @twentyse7en ?

[twentyse7en]

I think the analysis-creation should be outside of the loop, as the analysis doesn't depend on the channel.

Yes, Thx for pointing out!

Can you comment on this

Test is working fine on my side. This is exactly the same structure we used in all template tests. I tried generating the notebook and it looks fine too.

[sk1p]

Test is working fine on my side. This is exactly the same structure we used in all template tests. I tried generating the notebook and it looks fine too.

Thanks for looking into it. Maybe this is something specific to my environment, as we also don't hit this problem in CI.

[sk1p]

Hmm, this now happens on two of my systems, with Python 3.7 and Python 3.8: for some reason, the notebook is generating a different result than the run inside of pytest. @twentyse7en did you try to re-create the python virtual environment, to install current versions of all dependencies?

@uellue are you also hitting this? Or am I somehow cursed? *_*

[uellue]

@sk1p I am getting the error on moellenstedt (my libertem37 env), but not on Windows. It smells a bit like a numerics issue, doesn't it? Since the code is tested on random data, the signal from Radial Fourier is pretty small and a tiny difference may determine which order is dominant.

I'd propose to use a dataset with a very clear signal. A folded linear gradient obtained with np.linspace(...).reshape(shape) should give a dominant one-fold symmetry if I am understanding correctly. :-)

[sk1p]

> /home/alex/source/LiberTEM/tests/template/test_radial_fourier_template.py(56)test_radial_fourier_default()
-> assert np.allclose(results["dominant_0"], expected["dominant_0"].raw_data)
(Pdb) results["dominant_0"]
array([[ 2,  4,  0,  4,  6],
       [ 6,  0,  4,  2, 14],
       [ 4,  8,  4,  2,  8],
       [ 8,  4,  0,  0,  8],
       [ 2,  8,  0,  0,  2]])
(Pdb) expected["dominant_0"].raw_data
array([[ 2,  4,  0,  4,  2],
       [ 6,  0,  4,  2, 14],
       [ 4,  8,  4,  2,  8],
       [ 8,  4,  0,  0,  8],
       [ 2,  8,  0,  0,  2]])
(Pdb) results["dominant_0"] - expected["dominant_0"].raw_data
array([[0, 0, 0, 0, 4],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])

And, crucially:

(Pdb) np.allclose(expected["absolute_0_2"].raw_data, expected["absolute_0_6"].raw_data)
True

If I'm not mistaken, this looks like orders 2 and 6 are close enough to be chosen based on numerical instability - is it possible that the "dominant order" logic needs to be updated to include some kind of tie breaker?

[sk1p]

Ah, or that: more deterministic test data should help, too!

[uellue]

"dominant order" logic needs to be updated to include some kind of tie breaker?

Hm, one could make it return "no dominant order", although it SHOULD already do that. Quite a few dominant orders are zero, which means these were already culled. Good point, we can turn this into an issue to re-examine that logic. Other option would be to use data with a very clear signal for testing, as described above.

[uellue]

although it SHOULD already do that.

Actually, probably we just shift the issue to "borderline cases" in the culling logic? :-D

[uellue]

This is the culling implementation:

https://github.com/LiberTEM/LiberTEM/blob/79d527c9df44a9ccab537eb68eda2a8255882ccf/src/libertem/analysis/radialfourier.py#L154-L160

[uellue]

What about a radical approach: We just don't test for the dominant order! If the other results are equal, we can be pretty sure that the parameters were translated correctly between web GUI and Python API, which is all we need to know here, right?

[sk1p]

Sure, that would work, too :grin: can this happen with real data? Is this then a case of "noise in/noise out"?

[uellue]

can this happen with real data? Is this then a case of "noise in/noise out"?

Well, in real data you may also have cases where the orders are very close. Since they are mapped to discrete values, you can get differences in borderline cases.

[twentyse7en]

did you try to re-create the python virtual environment, to install current versions of all dependencies?

tox -e py37 -r tests/template/test_radial_fourier_template.py is this enough?

[sk1p]

is this enough?

Looks good, yeah. It doesn't seem that this problem is triggered by a specific dependency, rather by a bit of non-deterministic behavior, which is more likely to be triggered if you have many cores / many workers.

What about a radical approach: We just don't test for the dominant order!

We should go with this solution - the order should be checked by a separate test anyways