Arctos: Datum: review needed, potential major change

[dustymc]

I need datum "translations" that I can DO STUFF with for https://github.com/ArctosDB/arctos/issues/2660

https://docs.google.com/spreadsheets/d/1hMxAUbHtLSbITLgnEllvmmgKfnWuCY9TPqkOhCf4VZM/edit#gid=551633880 is a first pass at linking existing datum values to a spatial reference system.

I need two things

1. Please review the SRID I've added, the "link" column should open a definition. I don't think I've mucked that up, but a review would be appreciated anyway.
2. For those without an SRID, we have to do something significant. Note that "unknown" (127,172 localities) is included in this.

We could:

• Figure out an SRID, which would involve finding or creating an appropriate spatial model and adding it to Arctos. Hopefully I've just missed finding a few....
• Convert them to something known, leave a remark noting the added (unknown amount of) error introduced by the blind conversion, then delete the untranslated datum values. This may be evil, but it's a one-time evil.
• Assign some arbitrary SRID to them, do arbitrary things with them forever.
• That clever thing I haven't thought of yet.

There is some urgency to this - if we are going to require spatial data to be spatial, we should at least stop introducing things that aren't - so I'm assigning high priorities.

Help!

Here's who's using the unknown non-"unknown" datums

select locality.datum,collection.guid_prefix,count(*) c from 
collection
inner join cataloged_item on collection.collection_id=cataloged_item.collection_id
inner join specimen_event on cataloged_item.collection_object_id=specimen_event.collection_object_id
inner join collecting_event on specimen_event.collecting_event_id=collecting_event.collecting_event_id
inner join locality on collecting_event.locality_id=locality.locality_id
where locality.datum in (
'Ordnance Survey of Great Britain 1936',
'European Terrestrial Reference System 1936',
'Geocentric Datum of Australia 2000',
'Fort Thomas 1955',
'GUX 1 Astro',
'Ocotepeque',
'Kusaie Astro 1951',
'Provisional La Conoa 1956 (Venezuela)',
'Montserrat Island Astro 1958',
'Antigua Island Astro 1943',
'Kertau 1948',
'Hu-Tzu-Shan',
'Ireland 1965',
'Indian Thailand',
'Taiwan Datum 1967'
)
group by locality.datum,collection.guid_prefix
order by locality.datum,collection.guid_prefix
;

                 datum                 | guid_prefix |  c   
---------------------------------------+-------------+------
 Antigua Island Astro 1943             | MSB:Para    |    4
 Antigua Island Astro 1943             | MVZ:Bird    |    3
 Fort Thomas 1955                      | MVZ:Herp    |    3
 Fort Thomas 1955                      | MVZ:Mamm    |    4
 Geocentric Datum of Australia 2000    | DMNS:Inv    |    1
 GUX 1 Astro                           | MVZ:Bird    |    6
 GUX 1 Astro                           | MVZ:Herp    |   18
 Hu-Tzu-Shan                           | MVZ:Bird    |   48
 Hu-Tzu-Shan                           | MVZ:Herp    |   79
 Hu-Tzu-Shan                           | MVZ:Mamm    | 1311
 Ireland 1965                          | MVZ:Mamm    |   52
 Kertau 1948                           | MVZ:Bird    |   10
 Kertau 1948                           | MVZ:Herp    |    1
 Kertau 1948                           | MVZ:Mamm    |   15
 Kusaie Astro 1951                     | DMNS:Mamm   |    1
 Kusaie Astro 1951                     | MVZ:Herp    |  331
 Kusaie Astro 1951                     | MVZ:Mamm    |   25
 Montserrat Island Astro 1958          | MVZ:Herp    |   18
 Montserrat Island Astro 1958          | MVZ:Mamm    |    4
 Ocotepeque                            | MVZ:Herp    |    5
 Ocotepeque                            | UTEP:Herp   |   14
 Ocotepeque                            | UTEP:HerpOS |   12
 Ordnance Survey of Great Britain 1936 | MVZ:Bird    |    2
 Ordnance Survey of Great Britain 1936 | MVZ:Egg     |   50
 Ordnance Survey of Great Britain 1936 | UAM:Mamm    |    5
 Provisional La Conoa 1956 (Venezuela) | MVZ:Bird    |   15
 Provisional La Conoa 1956 (Venezuela) | MVZ:Mamm    |   27
 Taiwan Datum 1967                     | MVZ:Herp    |   35

[Jegelewicz]

I clicked the links I know we use a lot and a few random ones as well and they all seemed good.

[Jegelewicz]

I am guessing that most of the unknown non-"unknown" datums came out of MVZ and that the reason other collections are using them is that they share localities?

[tucotuco]

I reconciled every value of geodeticDatum in GBIF to epsg codes to the extent possible. They are in https://raw.githubusercontent.com/VertNet/DwCVocabs/master/vocabs/geodeticDatum.csv. I hope that helps you resolve your unknown non-"unknown"s.

[dustymc]

Sweet, got a few more, thanks!

[sharpphyl]

Geocentric Datum of Australia 2000 | DMNS:Inv | 1

Fixed.

[dustymc]

Here's what I still can't deal with @mkoo @ccicero @atrox10

--------------+--------------+--------
 GUX 1 Astro  | MVZ:Bird     |      6
 GUX 1 Astro  | MVZ:Herp     |     18
 Ireland 1965 | MVZ:Mamm     |     52

TODO: add fkey relationship from ctdatum to spatial_ref_sys.

The elephant in the room is still "unknown" - I don't think there is a technical solution to missing data, this is a social problem. If nobody comes up with a better idea I'll probably just pretend it's WGS84?

[campmlc]

That's what Google does, correct? So at least we'd be consistent.

On Mon, Feb 8, 2021, 7:07 AM dustymc [e-mail] wrote:

• [EXTERNAL]*

Here's what I still can't deal with @mkoo https://github.com/mkoo
@ccicero https://github.com/ccicero @atrox10
https://github.com/atrox10

--------------+--------------+--------
GUX 1 Astro | MVZ:Bird | 6
GUX 1 Astro | MVZ:Herp | 18
Ireland 1965 | MVZ:Mamm | 52

TODO: add fkey relationship from ctdatum to spatial_ref_sys.

The elephant in the room is still "unknown" - I don't think there is a
technical solution to missing data, this is a social problem. If nobody
comes up with a better idea I'll probably just pretend it's WGS84?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3410#issuecomment-775174150,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBBSWTULZLJRSLJAE2TS57VYFANCNFSM4XFEYFMA
.

[dustymc]

what Google does

Sorta, but they don't accept anything else. This would leave us with sometimes converting and sometimes not, which is a little weird but maybe not across the line to evil since I've locked down editing "as-entered."

[tucotuco]

You are keeping the lack of datum information that would tell a
georeferencer that there is a major source of uncertainty, right?

On Mon, Feb 8, 2021 at 11:31 AM dustymc [e-mail] wrote:

what Google does

Sorta, but they don't accept anything else. This would leave us with
sometimes converting and sometimes not, which is a little weird but maybe
not across the line to evil since I've locked down editing "as-entered."

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3410#issuecomment-775190344,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADQ722UMUC6KN4NQGT5AH3S57YUVANCNFSM4XFEYFMA
.

[dustymc]

Yes, collecting_event.datum is part of "as-entered" coordinates. And from there I think I'm coming around to the idea of just using WGS84 for anything I can't convert; if we learn more at some point we can just update the code table and run the conversion again.

[campmlc]

I agree as long as we record the original lack of datum and clearly
document the source WGS84 as a default conversion for lack of more specific
info.

On Mon, Feb 8, 2021, 8:27 AM dustymc [e-mail] wrote:

• [EXTERNAL]*

Yes, collecting_event.datum is part of "as-entered" coordinates. And from
there I think I'm coming around to the idea of just using WGS84 for
anything I can't convert; if we learn more at some point we can just update
the code table and run the conversion again.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3410#issuecomment-775228195,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBB2NX2MLO7HIJ4TCVDS577HLANCNFSM4XFEYFMA
.

[tucotuco]

Anything in decimal coordinates, or anything at all? Seems a reasonable
plan. GBIF does the same, for what it's worth.

On Mon, Feb 8, 2021 at 12:27 PM dustymc [e-mail] wrote:

Yes, collecting_event.datum is part of "as-entered" coordinates. And from
there I think I'm coming around to the idea of just using WGS84 for
anything I can't convert; if we learn more at some point we can just update
the code table and run the conversion again.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3410#issuecomment-775228195,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADQ72YTT6OGYBTKWNA547TS577HLANCNFSM4XFEYFMA
.

[dustymc]

decimal coordinates

"Anything I can convert to decimal coordinates" - Arctos accepts and converts DD.dd and DMS and, in the very near future, UTM, and they can all come in with unknown datum.

document the source WGS84 as a default conversion

It's not that waffly; all datums (data?? Seems wrong...) will be required to have an SRID with a local definition, I'll just run with whatever lands in there. It'll be up to the CT managers to make sure it's something reasonable, that there's documentation if its weird, and to read the big red box. Like all CTs, there's a change log.

[dustymc]

This is in production, things without an SRID will just fail with an error when we get to postgis.

The remaining mysteries are all MVZ - @mkoo @ccicero @atrox10 you can update ctdatum or let me know if you need something moved.

Closing.