Arctos: rebuild locality access forms

[dustymc]

unify all private and public locality, event, geography forms

• they require about twice as much code as they really need and they're evolving independently; good ideas aren't being merged
• the include-form-thing should be replaced with a webservice call

needs prioritized

https://github.com/ArctosDB/arctos/issues/2498#issuecomment-658415658

[dustymc]

https://github.com/ArctosDB/arctos/issues/2820

[Nicole-Ridgwell-NMMNHS]

How will this fix #2963?

[dustymc]

How will this fix #2963?

I'm hoping you'll tell me, but big-picture I'm digging your idea

Results formatted the same way as specimen search results.

of a dynamic table, which also plays nicely with my idea of reusing code by building webservices.

[Nicole-Ridgwell-NMMNHS]

of a dynamic table

Yes, this is exactly what I need. What is the timeline for this? I get locality search requests pretty regularly and I still have to use our old Access database for those. That will be more difficult once we start adding new data.

[dustymc]

This has some potential security problems which need addressed, prioritizing.

Summary of linked issues

• add higher_geog "starts with/is/contains/ends with" search
• results in sort-able, downloadable tabular format
• ability to include the fields that I want (this is a limited scope, I think we can just include everything everywhere - OK @Nicole-Ridgwell-NMMNHS ?)
• spatial stuff continues to get ignored, unless I can have a few uninterrupted months to ponder spatial data...
• make sure everything's properly escaped and normalized

Did I miss any "core" "locality" (in the broadest sense) issues that need dealt with here?

[Jegelewicz]

Seems like a good start to me.

[Nicole-Ridgwell-NMMNHS]

this is a limited scope, I think we can just include everything everywhere

As long as there is enough room in the results table for all the locality attributes.

Did I miss any "core" "locality" . . . issues

Sounds like all the main things to me.

[dustymc]

enough room in the results table for all the locality attributes.

Elaborate please. I intend to pull them as JSON, so they'll just take one cell....

[Nicole-Ridgwell-NMMNHS]

so they'll just take one cell

I need to be able to sort a group of localities by individual attributes.

[dustymc]

Can you use eg https://arctos.database.museum/editLocality.cfm?locality_id=11104847 to demonstrate how you imagine that looking as a row in a table?

[Nicole-Ridgwell-NMMNHS]

If it could work like specimen attributes on the catalog search, where I can go to "Add or remove data fields" and pick whatever specimen attributes I want to show in the table, that would be great.

Say I'm looking for Miocene localities, and the Formation/Member data is also important. I search on attribute Miocene, I also add the data fields formation and member to my search. Now I can sort my results by the Formation/Member data.

Another example, a mitigation company needs me to find all locality records in R08E, T22N, Secs 1-6, 11, 12 & R08E, T23N, Secs 25, 35-36 (not a real example, but I've had real examples more complicated than this). I search on each township range, add the section data field and sort by the sections so I don't have to search for each section individually.

[dustymc]

I can bang together a getLocalityAttribute(id,type) function (that returns a bunch of concatenated text or something), but those come with a significant performance hit (https://github.com/ArctosDB/arctos/issues/2938), and I do not intend to cache here so there's no way to mitigate that without a serious re-think and a lot of added complexity.

I've been interpreting The Community's direction as "don't do things that don't perform" - lemme know if I'm not understanding that....

We currently have 22 attributes, I don't think making everyone sort through 22 extra (usually empty, probably) columns can work, MAYBE I could somehow hide those in certain forms or something. That wouldn't be very "sortable" anyway, and I certainly can't see a pathway to adding a column for each component of each of the 50 determinations for each of those 22 (and probably growing) attributes.

Adding the ability for a user to customize (esp. by adding not-locality things) would also be a significant re-think.

I could help you build SQL to get that sort of thing, but that's not a great solution for what sounds like a normal part of your job.

I can easily do this vertically (so 11104847 would return 18 rows) but I don't think that would serve the purpose very well either.

It's theoretically possible to pull bits out of the JSON, but I wasn't able to do so in a few minutes of exploration so I don't know how realistic that actually is at this point. https://github.com/ArctosDB/arctos/issues/2693 is the same idea.

The bigger picture is that there's a good reason these data aren't flat, and flattening them is going to be some flavor of strange, and/or expensive, and/or not what some other user will want, and/or ...

I just don't know how to deal with this, I suppose I have to kick it back to needs discussion.

Note to self: There's some code in component/api/getCollectingEvent.

[Jegelewicz]

For now, could we get what you originally planned? I could use the functionality of downloadable results from locality search....

What @Nicole-Ridgwell-NMMNHS is asking could also be possible if we were able to search using more than one locality attribute at a time. How hard is that?

[dustymc]

get what you originally planned

Technically sure, but I'm not crazy about the idea of building a form that doesn't work for us....

search using more than one locality attribute at a time. How hard is that?

trivial

[Jegelewicz]

I mean, that is what she asking for correct?

Say I'm looking for Miocene localities, and the Formation/Member data is also important. I search on attribute Miocene, I also add the data fields formation and member to my search. Now I can sort my results by the Formation/Member data.

Search Locality Attribute = Epoch/Series AND Attribute Value = Miocene
PLUS Locality Attribute = lithostratigraphic formation Attribute Value = Some Formation
PLUS Locality Attribute = lithostratigraphic member AND Attribute Value = Some Member

get results she needs in a table, download and do whatever. Locality Attributes can be in a single JSON column because she already knows she has what she wants...

[Jegelewicz]

This is no different from the part attributes in the main search results....right?

[Nicole-Ridgwell-NMMNHS]

If what I'm asking would be very difficult to do, then having something would be much, much better than what we have now.

[dustymc]

Search Locality Attribute = Epoch/Series AND Attribute Value = Miocene
PLUS Locality Attribute = lithostratigraphic formation Attribute Value = Some Formation
PLUS Locality Attribute = lithostratigraphic member AND Attribute Value = Some Member

Yes that's possible.

part attributes

Yep, same idea (but "parts" are natively flat here, so this is a bit simpler).

something

OK, I'll go back to work, maybe something useful will pop up.

[dustymc]

There's a basic functional form to play with at http://test.arctos.database.museum/demo.cfm

Big-picture, I might use this other places where tables are handy - not sure there's anything specific from that, but something to keep in mind.

Slightly smaller picture, the locality and geology forms will likely be this with some stuff commented out.

Does this meet basic needs - can we reformat some cells, change the style on the form, that sort of thing, and be happy with this, or do I need to be looking at some radically different UI or something?

Feedback greatly appreciated.

[Jegelewicz]

change the style on the form

If this could look more like the catalog record search results, I think that would be nice. So:

1. once you have requested search results, the search stuff is hidden, unless you unhide it.
   

2. You get a summary of the number of localities found (see above)

3. After you expand attributes, we need a way to un-expand them
   

4. You have ability to add or remove fields, including attributes
   

5. You have the ability to see the localities found on a map
   

6. Need to be able to download results

7. Need a way to highlight a row so that you can follow it across the screen as you scroll left and right

8. Need to be able to edit the localities (or view them individually assuming you are not logged in).

OK that's a lot to ask, but consistency with the main catalog record search results (in appearance at least) would make Arctos way more user friendly. No idea if I am asking for the moon or if this can be done fairly easily....

[Nicole-Ridgwell-NMMNHS]

After testing it a bit, I agree with all of Teresa's comments. Also, need to be able to search on more than one locality attribute.

I'd like to see what the locality attributes look like when the data is downloaded so I can see how easy/difficult it will be to parse the attributes out separately in excel.

[dustymc]

I'm very tempted to suggest we throw this back for more discussion - I feel like we really need AWG focus on this.

1. I can do that, but it's a very different THING going on here. SpecimenSearch is a traditional form-->submit-->results setup, this thing is entirely dynamic and API based. This seemed to me like a good opportunity to explore a more dynamic/interactive - well, interaction - with the data, rather than "here it is, start over if that's not what you want" approach in SS.

2. I'm sure I can pull that out and do weird things with it, but it's build into the library....

... so this is the difference between no additional code, and something that will have to be maintained, made to work in different environments, etc., etc., etc. (Also something users have encountered on most every site that uses DT vs. something we've rolled up.)

1. Possible, and I'm already adding code there, or just search again

2. That's a LOT more development, some more junk in the DB, probably a different architecture, etc.

3. I think that's just another app/view, unless we somehow find a way to create cacheable thumbnails or something. (I can create them via google, but their TOS won't let me cache them and generating them is expensive in terms of money and resources.)

4. On my plate for after everything else works/is accepted, should be fairly trivial (yay api back end).

5. That's probably possible in DT (or more accurately something DT can talk to), almost certainly makes more sense as part of https://github.com/ArctosDB/arctos/issues/2893

6. "edit" is a link (if you're logged in), but this is an event form, not locality. "view" is another app/view (which will be needed - and might get all of the 'edit' links for "us." along with a map).

consistency with the main catalog record search results

I'm actually thinking about going the other way. SS is based on some now-barely-supported library, DT seems to be pretty mainstream, has modern features, and I'm paying for some extensions so we can use the editor capabilities for data entry, it might make sense to do a LOT more with it.

Maybe that means we should spend some time here, making this a sort of proof of concept that DT can do everything we do in SS (and more), but given the state of the issues repo I'm also a little hesitant to spend all the time required to do that on a less-than-mainstream form without the entire AWG being a little more involved.

I don't think you're asking for anything that's not possible, just things that take time and will require additional maintenance.

[Jegelewicz]

OK

Added Priority Critical so that we can get it on AWG Agenda

I like the idea of a "proof of concept" if this seems like a good tool to use for all kinds of search results, but agree that maybe the proof should really be on the main search results as that is what most people use?...

but this is an event form, not locality.

Which is kinda a problem because @Nicole-Ridgwell-NMMNHS is looking for localities. That is why this is an issue.

[dustymc]

maybe the proof should really be on the main search results as that is what most people use?...

I'm not that iffy about this - I'm growing confident that DT can do everything we need there, it's a matter of whether if can do it in a way that ya'll like. (If it can't, we should find something that can ASAP - I'm DONE with a bunch of specialized libraries that all work slightly differently!) Starting "simple" should get at that with less investment, I think, and this lets me explore the various ways of doing things without having to deal with eleventy bajillion dynamic columns and etc.

looking for localities

The locality form will just be the event form, minus some stuff (and perhaps with different links). It's easier in this case to get the most complex use case solid, then remove what's not needed for localities and geography.

all kinds of search results

This also sets up API access, so it's really ALL. The back-end I'm using for this form should be able to feed any UI, local or otherwise. (There are some authentication issues to deal with for "otherwise" but that's relatively trivial, I hope...)

get it on AWG Agenda

Thanks. I'm probably flip-flopping even more, but I'm more or less certain that this is going to work in some form, so I'll keep going on the things that I know you're going to want (eg multiple attribute search) and we can deal with improvements, or perhaps just the scheduling thereof, via the AWG. (I _think_ this can be a phased thing - Phase 1 is a table - what I've got now, maybe with some polish - Phase 2 adds [whatever], Phase 3 adds.....)

Reasonable??

[Jegelewicz]

Seems reasonable to me.

[Nicole-Ridgwell-NMMNHS]

I think this can be a phased thing

That would be great - something that is functional at a basic level now with improvements & exploration of what is possible to come. I didn't know the main search was relying on barely supported stuff, so I'm all in favor of this being a proof of concept for specimen search.

[dustymc]

There are some changes, including multiple locality attributes and a download button.

There are also some internal changes that firm up my idea that this can work well for specimenresults - datatables is pretty cool!

Low-priority from my perspective since it's trivial to change, but I could use some help with column order and headers. There's a LOT of whitespace in the table - shorter headers will help with that, but probably make the columns more cryptic.

The list-of-lists that builds that is below.

• column order is list order
• before | is what I get from the API - don't mess with it
• after | is the display header - please do mess with it
• don't use | (pipe) or , (comma) in the data, they're reserved for list delimiters here (or let me know and I'll find other delimiters, whatever...)

higher_geog|Geography,
spec_locality|Specific Locality,
verbatim_locality|Verbatim Locality,
verbatim_date|Verbatim Date,
began_date|Began Date,
ended_date|Ended Date,
dec_lat|Latitude,
dec_long|Longitude,
source_authority|Source,
continent_ocean|Continent/Ocean,
country|Country,
STATE_PROV|State/Province,
COUNTY|County,
QUAD|Map Quadrangle,
FEATURE|Feature,
drainage|Drainage,
ISLAND|Island,
ISLAND_GROUP|IslandGroup,
SEA|Sea,
geog_remark|Geography Remark,
wkt_media_id|Geography Polygon,
max_error_distance|Maximum Error Distance,
max_error_units|Maximum Error Units,
georeference_source|Georeference Source,
georeference_protocol|Georeference Protocol,
locality_name|Locality Name,
DATUM|Datum,
LOCALITY_REMARKS|Locality Remarks,
MINIMUM_ELEVATION|Minumum Elevation,
MAXIMUM_ELEVATION|Maximum Elevation,
ORIG_ELEV_UNITS|Original Elevation Units,
MIN_DEPTH|Minimum Depth,
MAX_DEPTH|Maximum Depth,
DEPTH_UNITS|Depth Units,
localityAtts|Locality Attributes,
Verbatim_coordinates|Verbatim Coordinates,
collecting_event_name|Collecting Event Name,
evtAttrs|Collecting Event Attributes

@Nicole-Ridgwell-NMMNHS "see what the locality attributes look like when the data is downloaded " - some of that could be piped through https://github.com/ArctosDB/arctos/issues/3046 - I can help you write SQL, that might become some sort of "custom" form or get pushed back to the main form.

[Jegelewicz]

My stab at it

locality_name|Locality Name,
higher_geog|Geography,
source_authority|Geography Authority,
continent_ocean|Continent/Ocean,
country|Country,
STATE_PROV|State/Province,
COUNTY|County,
QUAD|Map Quad,
FEATURE|Feature,
drainage|Drainage,
ISLAND|Island,
ISLAND_GROUP|Is Group,
SEA|Sea,
geog_remark|Geography Remark,
spec_locality|Specific Locality,
MINIMUM_ELEVATION|Min Elevation,
MAXIMUM_ELEVATION|Max Elevation,
ORIG_ELEV_UNITS|Elevation Units,
MIN_DEPTH|Min Depth,
MAX_DEPTH|Max Depth,
DEPTH_UNITS|Depth Units,
LOCALITY_REMARKS|Locality Remarks,
wkt_media_id|Geography Polygon,
Verbatim_coordinates|Verbatim Coordinates,
dec_lat|Lat,
dec_long|Long,
max_error_distance|Error,
max_error_units|Error Units,
DATUM|Datum,
georeference_source|Georef Source,
georeference_protocol|Georef Protocol,
localityAtts|Locality Attributes,
collecting_event_name|Event Name,
verbatim_locality|Verbatim Locality,
verbatim_date|Verbatim Date,
began_date|Began Date,
ended_date|Ended Date,
evtAttrs|Event Attributes

Now here are some questions and comments.

1. Elevation - Are the numbers that appear in Max and Min elevation always in meters or are we getting numbers that go with the term in ORIG_ELEV_UNITS?

2. I took "collecting" off of the event column headers because in truth, these events may represent manufacture, use, etc.

3. I would say that we could leave off WKT - I know there would be cases when someone (probably @mkoo ) would want this, but probably 99% of the time, it is just something extra to deal with. (my opinion - I could be wrong).

4. What @Nicole-Ridgwell-NMMNHS and I REALLY want is a LOCALITY search - no events to muck up the works. I know there is a need for event searches, but I don't need 190 rows of locality NMMNH:Paleo:L-5299 because each record has it's own event. When we say locality search we mean it! To me, the stuff above are the results of an EVENT search.

5. wkt_media_id|Geography Polygon - is this the higher geography polygon or specific locality polygon?

[dustymc]

Elevation

it's directly from the table, so the latter

leave off ...

At some level, whatever - at some other level this should serve a diverse group of users who are looking for a diverse set of data, it's easy(ish...) to ignore stuff, not so easy to find something that's not there.

That said, I don't think I'll remove anything from the download so those data ARE available, just not in a UI - and you can use them to build your own UI if you want.

REALLY want is a LOCALITY

Again, that's just this minus some columns. Get this happy and that's a couple hours of copy-paste. (Or get this wrong and that's changing a bunch of stuff in three places....)

results of an EVENT search.

Yes, exactly. " Find Collecting Events: " at the top....

wkt_media_id|Geography Polygon

Geography - should probably add the event as well, at least to the API

[Nicole-Ridgwell-NMMNHS]

Can we leave off higher_geog? Seems redundant to have both that and country/state/county/quad/feature/island/island group/sea?

Does Geography Authority need to be in there? I don't want to make assumptions about what people would be looking for, but if they were looking for that kind of info, it seems like they would just do a geography search.

I agree with Teresa, ideally a locality search wouldn't pull up each individual event.

Still trying to find an easy way to parse out the Json in the downloaded file. If that doesn't work then I might have to take you up on this:

I can help you write SQL, that might become some sort of "custom" form or get pushed back to the main form.

[dustymc]

leave off higher_geog?

It is redundant, but it's also the thing that gets displayed and passed around, and I don't have to scroll through a bunch of maybe-null (and possibly invisible if we add customization) columns to figure out what's going on. No compelling arguments, but I like it.

Geography Authority need to be in there?

I think so - there are a LOT of things named Florence in Italy, for example, the authority helps me understand which of them you might be referring to. (The ones that just say "some organization" don't, but those should be viewed as potentially-bad-data flags so I suppose they add - or subtract - some value as well...)

locality search wouldn't pull up each individual event.

It won't - this is an event search. https://github.com/ArctosDB/arctos/issues/2916#issuecomment-680204290, https://github.com/ArctosDB/arctos/issues/2916#issuecomment-681010571.

If there's nothing major - removing or reorganizing columns isn't, adding them might be - and general functionality is livable for the moment, I can extract locality and geography forms from this.

I think I'm also going to build a "view place" form - I was hoping for some merger with that and the edit forms, but I'm not (yet?) seeing how.

[dustymc]

@Nicole-Ridgwell-NMMNHS I wrapped some stuff into http://test.arctos.database.museum/place.cfm - break it, please.

[Nicole-Ridgwell-NMMNHS]

So far it seems to be working well. The search for localities option is great, as is the option to see locality details.

I can't seem to do anything useful with the Json in excel. I realize separating the locality attributes is a phase 2 thing, but it should be priority when we get there. I'll get back to you about writing some SQL for those for now.

[dustymc]

This is in production, but not linked. The intention is to replace all possible forms with this one. Further testing appreciated.

[dustymc]

@ArctosDB/arctos-working-group-officers @ewommack perhaps we should make some sort of announcement before replacing a bunch of locality forms?

[ewommack]

@dustymc We could add it to the Newsletter, but that doesn't come out until October. Do you have a time frame you want to do the release?

I like Newsletter articles for features, because they can include pictures and links, and we have a section already for this type of info: Arctos Feature Highlights.
But they are only quarterly, and we're moving right along with some new nifty things.

[dustymc]

I'm interpreting the silence as enthusiastic approval and will proceed with replacing all possible locality forms with https://arctos.database.museum/place.cfm

[Jegelewicz]

@mkoo @campmlc @ccicero have any of you looked at this?