Iris: Valid standard_name (present in STD_NAMES dict) is still rejected because of formatting maflormation

[valeriupredoi]

Summary
atmosphere_optical_thickness_due_to_pm1_ambient_aerosol is not recognized as a valid standard_name by iris although it really is, and present in the STD_NAMES dict too

Description and pinpoint of bug
Hey guys, apols if this is a duplicate but we have just seen this (these are code snippets from iris=2.3 but the codes are the same for 2.4, I just checked):

in iris/_cube_coord_common.py l/47:

        valid_name_pattern = re.compile(r'''^([a-zA-Z_]+)( *)([a-zA-Z_]*)$''')
        name_groups = valid_name_pattern.match(name)

name_groups is returned as None for name_groups = valid_name_pattern.match("atmosphere_optical_thickness_due_to_pm1_ambient_aerosol")
even if atmosphere_optical_thickness_due_to_pm1_ambient_aerosol is a valid standard name it's discarded right away. The name can be found in STD_NAMES in iris/std_names.py and is also a valid CF entry.

Solution
Reformatting the regexp call should do, I'll leave that to you :beer:

Point of contact: @bjlittle @lbdreyer (suggested)

[pp-mo]

I guess that this one fails because of the '1 in it.

I think we have a bit of a problem, as the CF rules on what "could" be a valid standard name don't seem to be stated anywhere ? Instead, they just rely on the huge list of accepted ones (AFAICT).

It also seems (to me, anyway) that we can't just use netcdf naming rules, as encoded by _TOKEN_PARSE , as standard names already go beyond that.

So, I think there are at least a few things wrong with the regexp:

• we should allow decimal digits ..
• .. but probably not as the first character
• we should probably ensure that the first character is alphabetic or '_' ..
• .. but we should also possibly disallow "_" as the first character,
  as "names commencing with underscore are reserved for system use" (according to the netcdf spec link in the code).

That would mean, possibly, replacing
r'''^([a-zA-Z_]+)( *)([a-zA-Z_]*)$'''
with
r'''^[a-zA-Z]([a-zA-Z0-9_]+)( *)([a-zA-Z_]*)$'''
@bjlittle @lbdreyer is this sounding like sense to you ?

ASIDE: Actually, I think _TOKEN_PARSE is also a bit wrong, as it also doesn't enforce any extra restrictions on the first character, which the link referenced in that code implies it should do

[valeriupredoi]

Cheers Patrick @pp-mo - whatever you guys decide to do sounds good to me; if it was by me I'd completely lift the constraint on standard_name being a CF-regulated standard name and use short_name instead as checked against CF/CMOR, but then again, that might be a bit too adventurous :grin: - anyways, when do you reckon this issue will be fixed - iris=3.0? Also pinging @zklaus since he is well versed wrt CF/CMOR standards, if he's got an opinion :beer:

[zklaus]

I would tend to side with @valeriupredoi here. If I understand correctly, iris requires the standard_name to come from the official standard_name table anyway, right? Then what is the purpose of also checking against such an expression?

[pp-mo]

what is the purpose of also checking against such an expression?

This came in with the change to support standard-name modifiers.
It is basically used to capture the 'base' part of the standard-name and separate it from an optional modifier, so it's not really used as a "check" as such -- which as you say is done by checking against the table -- but it must be capable of matching any actual standard names (!!)

If that's so, my suggestion above is overkill really + perhaps this can be simplified.
In fact, given that both non-whitespace parts are checked against a valid list, I'm not really clear why a simple .split() won't do the business here...

Bonus: I see there are some nice specific testcases for this in tests/unit/cube_coord_common/test_get_valid_standard_name.py
If we can make a simplified split()-based approach pass that, then I reckon that is the way to go.

However ... We're really busy here + I'm supposed to be doing something else.
Could one of you possibly take this on ?

[valeriupredoi]

cheers Patrick! yeah we can do this :+1: I nominate @jvegasbsc since he was the first one to point the finger to iris :grinning:

[jvegasbsc]

cheers Patrick! yeah we can do this :+1: I nominate @jvegasbsc since he was the first one to point the finger to iris :grinning:

I accept the nomination. Iris saved me a lot of work with the CMOR checker, so it feels right that I contribute a bit to it