Iris: Pandas bridge to support string dtype

[bjlittle]

Extend the capability of the iris.pandas functions to cope with string data

• as_cube
• as_series
• as_data_frame

[bjlittle]

@bascrezee Can you provide an self-contained example use case, in particular for iris.pandas.as_cube?

Many thanks! :smile:

[bascrezee]

I ran into this issue when reading station data into a dataframe and trying to convert it to a cube. Trying to convert the string data ('quality_flag') to a cube raises an error. See below for a self-contained example.

Thanks for looking into this!

Text file somedata.csv

time,value,quality_flag
2008-01-01,5,G
2008-01-02,6,G
2008-01-03,-23,F

import pandas as pd
import iris.pandas as ipd

df = pd.read_csv('./somedata.csv')

# This goes wrong (see error message below)
ipd.as_cube(df)

# The problem is in the string data, since after removing it from the 
# dataframe it works fine
df = df.drop(columns='quality_flag')
ipd.as_cube(df)

Error message

  File "mwe.py", line 7, in <module>
    ipd.as_cube(df)
  File "/net/exo/landclim/crezees/conda/envs/esmbas/lib/python3.6/site-packages/iris/pandas.py", line 114, in as_cube
    cube = Cube(np.ma.masked_invalid(data, copy=False))
  File "/net/exo/landclim/crezees/conda/envs/esmbas/lib/python3.6/site-packages/numpy/ma/core.py", line 2362, in masked_invalid
    condition = ~(np.isfinite(a))
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

[bjlittle]

Awesome thanks @bascrezee :+1:

[trexfeathers]

Hi @bascrezee, I've been looking at this today and I think there's more to it:

The pandas bridge currently expects all data under the column headers to be a single measurement (e.g. air pressure), with a single data type. So the example would not work even if strings were supported

Could you tell me what kind of cube you would expect to see from your example? That will help us understand what other steps need to be taken

[bascrezee]

Hi @trexfeathers, thanks for looking into this. In my use case, the string column would be the quality flag. I just tried out for another well-established (satellite) dataset which has a quality flag. When reading this dataset (NetCDF) from disk the variable and the quality flag are returned in separate cubes. Following that behaviour, it would make sense to make iris.pandas.as_cube return two cubes in the example given above. What do you think?

The two guiding principles for this problem can be:
a) from a data manager perspective: no data or metadata should be lost in the conversion from iris.pandas to iris.cube
b) from a user perspective: it should be possible to subset the data based on the quality flag information

[trexfeathers]

Once split into multiple cubes, you could subset the value cube with the quality flag cube, but I think users would find it more intuitive and easier to work with if quality flag was recognised as a coordinate of value

I think the best implementation would be to add a new parameter to as_cube - a tuple of column names allowing the user to specify which columns contain phenomena. A new cube will be generated for each phenomenon, treating every non-phenomenon column as a coordinate. If no columns are specified, all data in the dataframe is assumed to be one phenomenon - the current behaviour (so no current users of as_cube would need to change anything)

Please think about how this would work with your uses of as_cube, and let me know your thoughts. It should at least meet your points a) and b)

[bascrezee]

Once split into multiple cubes, you could subset the value cube with the quality flag cube, but I think users would find it more intuitive and easier to work with if quality flag was recognised as a coordinate of value

I agree that this is intuitive from a user perspective, although it is maybe not following the CF conventions? I am not a specialist there. Maybe @zklaus has some thoughts on this as well ?

I think the best implementation would be to add a new parameter to as_cube - a tuple of column names allowing the user to specify which columns contain phenomena. A new cube will be generated for each phenomenon, treating every non-phenomenon column as a coordinate. If no columns are specified, all data in the dataframe is assumed to be one phenomenon - the current behaviour (so no current users of as_cube would need to change anything)

That sounds good to me. Am I right, that for each phenomenon, the column containing the values of that phenomenon should be provided by the user as well? Or is there another way to determine this unambiguously?

Please think about how this would work with your uses of as_cube, and let me know your thoughts. It should at least meet your points a) and b)

:thumbsup:

[zklaus]

At the risk of complicating things a little bit, there are two concepts in the CF conventions that are usually associated with quality flags, namely ancillary data, and flags. Both have been in CF since at least 1.4.

I am not sure how this should be modeled in iris, but on the netcdf side of things it seems pretty clear:
The flag should be in a flag variable, probably of type byte and with standard_name status_flag, and the actual variable should have an ancillary_variables attribute linking it with the flag variable.

In terms of iris ancillary variables remind me of cell_measures and auxiliary coordinates, both of which are somehow additional information per cell. Maybe the cleanest way would be to add a new class AncillaryCube or similar to represent this kind of information.

This is all really only talking about ancillary data and qc flags, not about pandas at all.

For the pandas bridge, I think @trexfeathers suggestions is good: Add a parameter/parameters to as_cube that allow the specification of kinds for columns. If we go with a new representation of ancillary data, that could be one of the choices. Basically the user could decide for every column if it belongs to the data, is a cell_measure, an auxiliary coordinate, or ancillary data.
I would insist that only one phenomenon is allowed, possibly spread over several columns.
To allow for the easy conversion of dataframes that contain data from more than one phenomenon, another option for columns should be to ignore them.

What do you think?

[trexfeathers]

Thanks for the information about quality flags @zklaus. If there is a requirement to handle this specifically I recommend raising an issue with the necessary detail.

I'm not going to try and account for future implementation of ancillary data in this pull request, as there's no way to tell how that might look once fully planned out. For Iris right now, a dataframe column could either be read as a phenomenon or a coordinate, and since there are _typically_ more coordinates than phenomena we can efficiently handle this choice by only requiring a list of the phenomenon columns
EDIT: sorry, I forgot about cell measures. I'd still like to avoid the need to 'type' every single column, so while this would be possible I'd set it to assume coordinate unless told otherwise

If there is a chance of a phenomenon being spread over several columns, but not all of them, then you're right to raise it as it would be problematic. In any example I can think of it would be poorly structured data, which should instead have been un-pivoted into a single-column phenomenon. @bascrezee / @zklaus can help me with an example?

[trexfeathers]

I've spoken with team members this morning and I have been given examples of multi-column phenomena mixed with single-column information, so I'll be sure the solution supports that

I can also tell you that quality flag handling is noted for discussion in the near future 🙂

[zklaus]

Great, thanks for getting back to us so quickly!

Should I open a new issue to track quality flat and/or ancillary data issues?

[trexfeathers]

There is now an issue for this - https://github.com/SciTools/iris/issues/3358

[stephenworsley]

@zklaus To update you on where we are with the ancillary variable issue, NetCDF variables with ancillary variables attached will now be loaded as cubes with AncillaryVariable objects, which are similar to the AuxCoord and CellMeasure objects. This includes some support for unsuring quality flags load without units, as appears to be the convention.