Iris: Load is VERY slow for a NetCDF multi-variable file

[senesis]

📰 Custom Issue

When loading a single variable from a quite small NetCDF file which includes 300 variables, the load time is very large : around 100 seconds (while it is less than 0.1s for a similar single variable file).

This is a bottleneck for trying to use Iris (trough ESMValTool) for handling some climate model native data format.

The attached notebook load_time_histmth.pdf demonstrates the issue and includes a profiling, which shows that the most time consuming function is (by large) NetCDFDataProxy.__getitem__

The data file is available here

System info is :

uname -a
Linux ciclad-ng.private.ipsl.fr 2.6.32-754.35.1.el6.x86_64 #1 SMP Wed Oct 7 03:47:54 CDT 2020 x86_64 x86_64 x86_64 GNU/Linux

[bjlittle]

@senesis Thanks for taking the time to report this, much appreciated.

Could you just confirm the version of iris that you're using? I'm assuming v3.0.1?

[senesis]

Yes it is v3.0.1

[bjlittle]

@senesis Great thanks. And which version of Python? v3.8?

[bjlittle]

@senesis We have a patch in the pipeline that will go towards significantly alleviating this issue.

We're going to target this for the forthcoming iris v3.0.2 release :+1:

See the v3.0.x Release Discussion for further details.

[pp-mo]

@senesis great issue -- your account is excellent + should make the problem reproducible !

I was already looking into issues with slow netcdf loads, specifically with lots of variables (i.e. we also found something similar).
I think I may already have a solution that will at least "alleviate" it ... coming shortly I hope.

In the meantime though, I wonder if you could put up your 'load_time_histmth' notebook so I can test out with that, maybe in a Gist ??

[pp-mo]

Stop press: see #4135
I just tested this with your file 'Iris_multivar_data_file.nc'
For me, it speeds up loading that (300-odd cubes) from ~50 to ~5 secs.
Win ! :grinning:

[senesis]

In the meantime though, I wonder if you could put up your 'load_time_histmth' notebook so I can test out with that, maybe in a Gist ??

I am not familiar with Gist. The notebook is available here

[senesis]

For me, it speeds up loading that (300-odd cubes) from ~50 to ~5 secs.

Great , I am looking forward to get 3.0.2 (and that ESMValTool uses it)

[bjlittle]

@senesis GitHub gists are a fantastic way to easily share snippets of code and notebooks with your peers.

Checkout the GitHub document for further details :+1:

[pp-mo]

The notebook is available here

Well I tried the notebook, but I'm not sure if it delivers any more info really, as I am just using it with the same 'Iris_multivar_data_file.nc' file you mentioned above, which may not be the same.
Anyway, it has ~260 variables with dimensions "t, y, x".
For me the basic load is taking ~46 seconds, which reduces to 5.7 secs with the #4135 fix.

[senesis]

The notebook is available here

Well I tried the notebook, but I'm not sure if it delivers any more info really, as I am just using it with the same 'Iris_multivar_data_file.nc' file you mentioned above, which may not be the same.
Anyway, it has ~260 variables with dimensions "t, y, x".
For me the basic load is taking ~46 seconds, which reduces to 5.7 secs with the #4135 fix.

I actually used the same file in my notebook run.
Thanks again for the fix.