Arctos: Consistency in Bulkloader Field Names

[Jegelewicz]

The specimen bulkloader and the Specimen Event Bulkload Tool use a lot of the same fields, but the field names are different. Can we make these more consistent? And can we also add LOCALITY_ID and COLLECTING_EVENT_ID to the Specimen Event Bulkload Tool?

Specimen Event Bulkload Tool | Specimen Bulkload | Consistent?
-- | -- | --
ASSIGNED_BY_AGENT | EVENT_ASSIGNED_BY_AGENT | no
ASSIGNED_DATE | EVENT_ASSIGNED_DATE | no
BEGAN_DATE | BEGAN_DATE | yes
COLL_EVENT_REMARKS | COLL_EVENT_REMARKS | yes
COLLECTING_EVENT_NAME | COLLECTING_EVENT_NAME | yes
  | COLLECTING_EVENT_ID | no
COLLECTING_METHOD | COLLECTING_METHOD | yes
COLLECTING_SOURCE | COLLECTING_SOURCE | yes
DATUM | DATUM | yes
DEC_LAT | DEC_LAT | yes
DEC_LAT_MIN | DEC_LAT_MIN | yes
DEC_LONG | DEC_LONG | yes
DEC_LONG_MIN | DEC_LONG_MIN | yes
DEPTH_UNITS | DEPTH_UNITS | yes
ENDED_DATE | ENDED_DATE | yes
GEOREFERENCE_PROTOCOL | GEOREFERENCE_PROTOCOL | yes
GEOREFERENCE_SOURCE | GEOREFERENCE_SOURCE | yes
guid |   | no
HABITAT | HABITAT | yes
HIGHER_GEOG | HIGHER_GEOG | yes
LAT_DEG | LATDEG | no
LAT_DIR | LATDIR | no
LAT_MIN | LATMIN | no
LAT_SEC | LATSEC | no
  | LOCALITY_ID | no
LOCALITY_NAME | LOCALITY_NAME | yes
LOCALITY_REMARKS | LOCALITY_REMARKS | yes
LONG_DEG | LONGDEG | no
LONG_DIR | LONGDIR | no
LONG_MIN | LONGMIN | no
LONG_SEC | LONGSEC | no
MAX_DEPTH | MAX_DEPTH | yes
MAX_ERROR_DISTANCE | MAX_ERROR_DISTANCE | yes
MAX_ERROR_UNITS | MAX_ERROR_UNITS | yes
MAXIMUM_ELEVATION | MAXIMUM_ELEVATION | yes
MIN_DEPTH | MIN_DEPTH | yes
MINIMUM_ELEVATION | MINIMUM_ELEVATION | yes
ORIG_ELEV_UNITS | ORIG_ELEV_UNITS | yes
ORIG_LAT_LONG_UNITS | ORIG_LAT_LONG_UNITS | yes
SPEC_LOCALITY | SPEC_LOCALITY | yes
SPECIMEN_EVENT_REMARK | SPECIMEN_EVENT_REMARK | yes
SPECIMEN_EVENT_TYPE | SPECIMEN_EVENT_TYPE | yes
UTM_EW | UTM_EW | yes
UTM_NS | UTM_NS | yes
UTM_ZONE | UTM_ZONE | yes
VERBATIM_DATE | VERBATIM_DATE | yes
VERBATIM_LOCALITY | VERBATIM_LOCALITY | yes
VERIFICATIONSTATUS | VERIFICATIONSTATUS | yes
wkt_polygon | WKT_POLYGON | yes

[Jegelewicz]

Any chance this is an easy fix? It is making me nuts as I load specimens and then new events for georeferencing.

[dustymc]

It's not - that's how it got this way!

Do please prioritize - it's REALLY helpful to me when ya'll assign labels and milestones, otherwise these all looks much the same to me. (And I think there's an issue about the specimen bulkloader that should probably come first if we're looking for consistency.)

[Jegelewicz]

I'm guessing no one cares but me since no one commented months ago. I'll try the please help, then bring to the AWG.

[campmlc]

I strongly agree with this. See #1938 also.
@DerekSikes

[dustymc]

Can we make these more consistent?

They can, but I'm not sure I understand the why and I think better understanding always leads to better UI. I see a lot of really scary errors involving folks typing headers. That's not something I'd do, ever - these things could be 10000 random characters and if I could understand the meaning I would absolutely never care. How do you use this?

add LOCALITY_ID and COLLECTING_EVENT_ID to the Specimen Event Bulkload Tool

https://github.com/ArctosDB/arctos/issues/1947

ANYWAY, all that aside, I'm all for consistency. Field Level Documentation exists to dynamically build code/forms and could be extended (maybe even just used) for more internal apps, I think.

[ccicero]

I am adding to this thread based on some related frustrations and a discussion at our AWG issues meeting last Thursday. I strongly agree about consistency in field names, and would like to extend this to the bulkload error file that is generated. Field names should be consistent there as well (with the bulkloader builder, maybe they are?), but more than that, it would avoid a lot of confusion and frustration if they were ordered in the same way as the bulkloader builder.

Issue is this:
1) Put the 'loaded' column with the errors first. If you don't know the column name and do a search for it by name, it's very hard to find.
2) Order the columns in the same order as the bulkloader builder, rather than in presumably default Oracle order which appears random. @dustymc

[dustymc]

There are lots of ways to see those data and I'm not sure who (if anyone) uses what - examples of problematic forms would be useful to me.

[ccicero]

When you get errors during bulkloader, you can download a file called 'bulkloader_stage' that has a 'LOADED' column with the errors. That column is buried among all the other columns, which are not in a logical order. This is what I'm talking about. What other ways are there to get the 'bulkloader_stage' file?

[dustymc]

Thanks! Browse and edit/CSV downloads data that's made it through that stage - I think the other stuff I was thinking of is the pre-bulkloader, which should also be made consistent.

[campmlc]

I would like to request that the order be in the same order as data fields
display in the single record data entry page, with the exception that
LOADED is first. This request also applies to the AJAX view of that same
data entry page. It is currently very frustrating to try to check data or
make edits in the AJAX table when, for example, OTHER_ID_1 and
OTHER_ID_REFERENCES_1 are at opposite ends of table. The order of the AJAX
table isn't bad, compared to the csv downloads, but we need to move
GUID_PREFIX to near the front, and put COLLECTION_SOURCE, LOCALITY_NAME,
COLLECTING_EVENT_NAME, SPECIMEN_EVENT_TYPE, SPECIMEN_EVENT_REMARKS etc to
be associated with the other fields related to specimen and collecting
events. Parts and attributes should be last, as they tend to have many
blank columns.

On Tue, Mar 12, 2019 at 4:00 PM dustymc [e-mail] wrote:

Thanks! Browse and edit/CSV downloads data that's made it through that
stage - I think the other stuff I was thinking of is the pre-bulkloader,
which should also be made consistent.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811#issuecomment-472198331,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hAE5mxpLGmu7ORSPGrTC83ifDYAuks5vWCODgaJpZM4Yx_fL
.

[dustymc]

order as data fields display in the single record data entry page,

That page is dynamic/customizable/order-able - do you mean the default order or have you rearranged?

[campmlc]

Default order works for me.

On Wed, Mar 13, 2019 at 9:18 AM dustymc [e-mail] wrote:

order as data fields display in the single record data entry page,

That page is dynamic/customizable/order-able - do you mean the default
order or have you rearranged?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811#issuecomment-472467162,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hNGnuvydWJfqg386QQbyYDUFspI_ks5vWRazgaJpZM4Yx_fL
.

[campmlc]

Any way for the fields to show up in the customized order by collection?
That would be even better.

On Wed, Mar 13, 2019 at 9:25 AM Mariel Campbell [e-mail]
wrote:

Default order works for me.

On Wed, Mar 13, 2019 at 9:18 AM dustymc [e-mail] wrote:

order as data fields display in the single record data entry page,

That page is dynamic/customizable/order-able - do you mean the default
order or have you rearranged?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811#issuecomment-472467162,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hNGnuvydWJfqg386QQbyYDUFspI_ks5vWRazgaJpZM4Yx_fL
.

[dustymc]

order by collection

Not really, it's customized by user. I definitely wouldn't want to invest that much work in to that form without a clear path forward - eg, we've talked about replacing that with something that would work offline from time to time. That would be a much higher initial investment, but would also be a much more powerful tools (if we can pull it off...).

[campmlc]

Absolutely agree that this should also work with Arctos Air. In fact, if
we'd had this functionality I could have made Arctos Air work. The problem
was the inability to preview and edit prior to load due to the lack of
coherent order of the columns.

On Wed, Mar 13, 2019 at 9:35 AM dustymc [e-mail] wrote:

order by collection

Not really, it's customized by user. I definitely wouldn't want to invest
that much work in to that form without a clear path forward - eg, we've
talked about replacing that with something that would work offline from
time to time. That would be a much higher initial investment, but would
also be a much more powerful tools (if we can pull it off...).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811#issuecomment-472475463,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hJdH7dKLzyXXaBqwgLZ7HWol-yknks5vWRq5gaJpZM4Yx_fL
.

[Jegelewicz]

It might help if field names were grouped, just as we group parts and attributes. If all fields related to Identification began with ID_, then sorting by field name (column header) would at least put all of the identification fields together.

Possibly a dumb idea, but it seems like a simple one.

[campmlc]

Bulkpendingdata.txt
Here is how the csv downloads currently from the AJAX view. I cannot review these data and edit them in this format.

[dustymc]

I'm suggesting something like the "Arctos Air" experiment could REPLACE the current data entry screen. The HTML app certainly doesn't do anything that couldn't be done in a stand-alone app.

FWIW the original "model" (how we thought things would work, not data arrangement) was that users would build their own data entry apps - we'd have many simple apps that do specific things rather than one giant complicated thing that does everything. "The data entry screen" just talks to an API so that's still possible, but it's never happened.

Another issue...

all fields related to Identification began with ID_,

I'm not opposed to renaming fields. I'll nominate collection_object_id - it (a local key) shares a name with several other unrelated things, which confuses everybody.

[dustymc]

There are two things which can be resolved while I'm figuring out HOW.

1) Renaming columns - should we, and if so to what?
2) Order

There's a spreadsheet currently ordered by the internal ID at https://docs.google.com/spreadsheets/d/1PriziHBDRE5d-vKX_n6VAjEfuM9hOeGhJAWlkF-eGYc/edit#gid=2001065259.

(1) needs resolved here, but I've started a place for it in case we do go in that direction.

For (2), you can just drag rows to reorder.

[sharpphyl]

• It is currently very frustrating to try to check data or
  make edits in the AJAX table when, for example, OTHER_ID_1 and
  OTHER_ID_REFERENCES_1 are at opposite ends of table. The order of the AJAX
  table isn't bad, compared to the csv downloads, but we need to move
  GUID_PREFIX to near the front, and put COLLECTION_SOURCE, LOCALITY_NAME,
  COLLECTING_EVENT_NAME, SPECIMEN_EVENT_TYPE, SPECIMEN_EVENT_REMARKS etc to
  be associated with the other fields related to specimen and collecting
  events. Parts and attributes should be last, as they tend to have many
  blank columns.

This may belong in a separate issue but I, too, would like to be able to reorder the AJAX columns and omit the ones we don't use or that don't have any data in them. I know I can sort and I can eliminate columns - a very time consuming process - but the minute I move to the next data entry person, all the changes are lost. If it could be set up like the data entry screen where the customization is stable from one entry to the next (and one session to the next) it would be very helpful.

[dustymc]

@sharpphyl I don't want to say that's impossible, but I don't know how it would be done - or how it would be done without the page having different columns every time you do anything, which I don't think is usable. The app has no idea what you're going to load next, so if we customize the form to the data when you initialize it then the next page might contain both empty columns and data in columns that aren't displayed. (You can probably do that now, but it's something you've just done rather than something that comes built-in.) If we rebuild the form around the data for every page then stuff will bounce all over the place as fields are used (so appear) or not (so don't), and we'll probably hit performance issues. I don't think there's a functional definition of "ones we don't use" either.

[campmlc]

I think she means that we match the order of the csv download to the data
entry page customized format for a given collection. So if you choose to
leave out a column in your customized display, that column would not appear
in the download from the AJAX table.

On Wed, Mar 13, 2019 at 5:19 PM dustymc [e-mail] wrote:

@sharpphyl https://github.com/sharpphyl I don't want to say that's
impossible, but I don't know how it would be done - or how it would be done
without the page having different columns every time you do anything, which
I don't think is usable. The app has no idea what you're going to load
next, so if we customize the form to the data when you initialize it then
the next page might contain both empty columns and data in columns that
aren't displayed. (You can probably do that now, but it's something you've
just done rather than something that comes built-in.) If we rebuild the
form around the data for every page then stuff will bounce all over the
place as fields are used (so appear) or not (so don't), and we'll probably
hit performance issues. I don't think there's a functional definition of
"ones we don't use" either.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811#issuecomment-472643835,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hOOpFYleAOJRpUUQC49tftKCX_Dxks5vWYeWgaJpZM4Yx_fL
.

[sharpphyl]

I'm referring to the approval/upload process in the AJAX table with entries from multiple data entry volunteers (not a bulk upload). There's just no way to eliminate the fields that we don't use to make the form easier to navigate and read. On the data entry screen, we can customize the form to select and position the fields we use and it stays that way permanently until we recustomize it. It would be nice to be able to do the same thing on the AJAX screen if it's possible. If not, we'll live with what we have.

[dustymc]

can we use table browser to map column names for various bulktools

[dustymc]

redo all downloads to use internal_column_order

[dustymc]

Specimen bulkloader and bulkloader-stage downloads should now be by internal_column_order

[dustymc]

This may be deserving of its own issue (and maybe there is one, but I can't find it) - someone please move if so.

locality.locality_name and collecting_event.collecting_event_name have been by request (of @DerekSikes maybe??) labeled '... nickname' in most forms. Both use "...name" in the specimen bulkloader (and likely other batch tools). Can we settle on terminology? (My vote is for 'name' but there's no technical reasoning behind that - I'm not very attached to anything.)

[campmlc]

I agree with name.

On Thu, May 16, 2019, 10:27 PM dustymc [e-mail] wrote:

This may be deserving of its own issue (and maybe there is one, but I
can't find it) - someone please move if so.

locality.locality_name and collecting_event.collecting_event_name have
been by request (of @DerekSikes https://github.com/DerekSikes maybe??)
labeled '... nickname' in most forms. Both use "...name" in the specimen
bulkloader (and likely other batch tools). Can we settle on terminology?
(My vote is for 'name' but there's no technical reasoning behind that - I'm
not very attached to anything.)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811?email_source=notifications&email_token=ADQ7JBGG4JOEN2T6ERYLCHLPVYX2FA5CNFSM4GGH67F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVTWHEI#issuecomment-493314961,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQ7JBCTSBZBFGZ4CZUUD7LPVYX2FANCNFSM4GGH67FQ
.

[campmlc]

I am posting here a map of the single record data entry fields against the main bulkload data entry fields as a related item where we need to maintain consistency of both field names and the order of the columns for both upload and download.
https://docs.google.com/spreadsheets/d/1HwD8-UWOi1wG7x71NZbaVdQz9ULscSLBVh7_vVBsIE4/edit?usp=sharing

[DerekSikes]

The only one that has tripped me up because unlike the other discrepancies,
it's not simply a different version of the same term, is Locality name vs
locality nickname.

They should both be Locality nickname. (or LOCALITY_NICKNAME) etc.

-Derek

On Sat, Jun 1, 2019 at 3:45 PM Mariel Campbell [e-mail]
wrote:

I am posting here a map of the single record data entry fields against the
main bulkload data entry fields as a related item where we need to maintain
consistency of both field names and the order of the columns for both
upload and download.

https://docs.google.com/spreadsheets/d/1HwD8-UWOi1wG7x71NZbaVdQz9ULscSLBVh7_vVBsIE4/edit?usp=sharing

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811?email_source=notifications&email_token=ACFNUM44GIIY36MCFGXVZYLPYMCZFA5CNFSM4GGH67F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXKUOQ#issuecomment-497986106,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACFNUM7Y4D3PMD6GOM2LCCLPYMCZFANCNFSM4GGH67FQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects
Professor of Entomology
University of Alaska Museum
1962 Yukon Drive
Fairbanks, AK 99775-6960

[email protected]

phone: [phone]
FAX: [phone]

University of Alaska Museum - search 400,276 digitized arthropod records
http://arctos.database.museum/uam_ento_all
http://www.uaf.edu/museum/collections/ento/
+++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us http://www.akentsoc.org/contact.php

[campmlc]

Derek uses this the most, so if nickname is preferable I agree with that.

On Mon, Jun 3, 2019 at 1:46 PM DerekSikes [e-mail] wrote:

The only one that has tripped me up because unlike the other discrepancies,
it's not simply a different version of the same term, is Locality name vs
locality nickname.

They should both be Locality nickname. (or LOCALITY_NICKNAME) etc.

-Derek

On Sat, Jun 1, 2019 at 3:45 PM Mariel Campbell [e-mail]
wrote:

I am posting here a map of the single record data entry fields against
the
main bulkload data entry fields as a related item where we need to
maintain
consistency of both field names and the order of the columns for both
upload and download.

https://docs.google.com/spreadsheets/d/1HwD8-UWOi1wG7x71NZbaVdQz9ULscSLBVh7_vVBsIE4/edit?usp=sharing

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/ArctosDB/arctos/issues/1811?email_source=notifications&email_token=ACFNUM44GIIY36MCFGXVZYLPYMCZFA5CNFSM4GGH67F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXKUOQ#issuecomment-497986106
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/ACFNUM7Y4D3PMD6GOM2LCCLPYMCZFANCNFSM4GGH67FQ

.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects
Professor of Entomology
University of Alaska Museum
1962 Yukon Drive
Fairbanks, AK 99775-6960

[email protected]

phone: [phone]
FAX: [phone]

University of Alaska Museum - search 400,276 digitized arthropod records
http://arctos.database.museum/uam_ento_all
http://www.uaf.edu/museum/collections/ento/
+++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us http://www.akentsoc.org/contact.php

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1811?email_source=notifications&email_token=ADQ7JBDGBTMQXTW64E5EKGLPYVRJTA5CNFSM4GGH67F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW2KP7A#issuecomment-498378748,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQ7JBBII7BTBXZ46N77BT3PYVRJTANCNFSM4GGH67FQ
.