Esmvaltool: Use netcdf4-python to handle date and time in _multimodel.py

[mattiarighi]

To correctly handle time selections with different calendars in _multimodel.py, netcdf-specific time/date functions shall be used.

This is especially important when considering long time periods (~decades), where the error due to inconsistent handling of time units with different calendars can accumulate and lead to wrong results.

One option would be to use the num2date and date2num functions of the netcdf4-python package.

@jvegasbsc do you know whether a similar function exists in Iris?

[jvegasbsc]

Iris can give you datetime.datetime or netcdf.datetime directly in the cells of the time coordinate. It's its default behavior in version 2 but can be activated in version 1 with the following line

iris.FUTURE.cell_datetime_objects = True

The only thing we must be aware of is that we can not merge cubes with different calendars, as the time coordinate will never be the same.

See http://scitools.org.uk/iris/docs/latest/iris/iris/coords.html for more info

[valeriupredoi]

quick question, @jvegasbsc -- I have iris 2.0 with python 3.6 and when loading a cube I still get floats for the cube.coord('time').points, setting iris.FUTURE.cell_datetime_objects is True doesn't change it either (and iris is telling me this is deprecated and will be removed). Also to note is that iris.unit.Unit has now changed to cf_units.Unit

[jvegasbsc]

You should not access the points field. You must use the cell(index) method or the cells iterable to get the datetime instances

[valeriupredoi]

but I need the actual time points as list or array, preferably as datetime objects, what's the syntax to get them?

[jvegasbsc]

You can get a list with something like this

 datetimes = [cell.point for cell in time.coord.cells()]

[valeriupredoi]

sweet, that works! thanks, man!

[valeriupredoi]

@mattiarighi @jvegasbsc @bouweandela
Guys, the new multimodel module has completely got itself rid of the by-hand conversions between types of calendars and now uses the datetime objects straight from cube cells as Javier suggested here: https://github.com/ESMValGroup/ESMValTool/blob/REFACTORING_finalize_multimodel/esmvaltool/preprocessor/_multimodel.py

I am asking yous to have a look and test it and if you consider this has been solved, close the issue. One mention is that it's not using netcdf4 or any other external python module, so we keep things clean, as Mattia suggested.

Thanks to Javier for pointing this previously unknown to me capability of iris cubers :)
I tested it both with python 2.7 and 3.6, but I feel that testing with python 2.7 is slowly becoming redundant.
Cheers,
V

[bouweandela]

@valeriupredoi Please create a pull request and select us for code review, that way it is much easier to review.

[mattiarighi]

@bouweandela I'm still testing the numerical results in this branch and there are some issues. I was suggesting @valeriupredoi to fix them before we submit the pull request, so that a finalized version of the code can be reviewed.

[valeriupredoi]

yep, I managed to mangle some rather important indices in the full case of
multimodel; fixing this now and then hopefully we'll be ready after testing
is done :)

On Fri, Mar 23, 2018 at 9:57 AM, Mattia Righi [e-mail]
wrote:

@bouweandela https://github.com/bouweandela I'm still testing the
numerical results in this branch and there are some issues. I was
suggesting @valeriupredoi https://github.com/valeriupredoi to fix them
before we submit the pull request, so that a finalized version of the code
can be reviewed.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/issues/268#issuecomment-375600292,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo1urJHl1Q9XmAqH8Mp_FTb3oXQ8nks5thMcAgaJpZM4SspJ0
.

--
Dr. Valeriu Predoi
Computational Scientist for UKESM Core Team
Department of Meteorology, University of Reading
Earley Gate, Office 1U08
READING, RG6 6BB
United Kingdom
Mobile number: 07847416092

"If one day you be questioning your ability to come up with professional
results, think of this: Noah's ark was built by farmers whereas the Titanic
was crafted by skilled engineers"