Arctos: Agent Bulkload Tool(s)

[Jegelewicz]

Please add a "delete everything from the bulkloader staging table (or just load new data to delete)" option to the Agent Bulkloader.

[dustymc]

??

[Jegelewicz]

Yes, I know, but if someone loads stuff then decides dot to do anything it would be good to have them delete it so everyone else can see it is available.

[acdoll]

Is there a way we can get this to load with middle initials? I fully understand we are trying to prevent confusing abbreviations, but adding them actually makes the data better, not worse. Peter J. Smith is clearly better than Peter Smith. Something like: if OTHER_NAME_TYPE = "middle name", then "." is OK?
Or if on the validation page there was the option to force creation for people with a certain level of permissions?
I realize this is a rather minor pet peeve of mine, but it would be nice not to have to enter these individually.

[dustymc]

http://arctos.database.museum/info/ctDocumentation.cfm?table=CTAGENT_NAME_TYPE

initials are "middle name" - you can put them in any name_ and corresponding name_type column.

Anyone with manage_codetables can force-load things with warnings.

[campmlc]

The requirement to for someone with manage code tables to bulkload any
agents with initials is overly restrictive. We have new collections coming
in where there are many names with only initials. These collections have to
wait on someone who knows nothing about their data to force load.
I think we should ease the restrictions, but make some improvements to the
agent bulkloader form. For one, display the results with validation errors
to be corrected, rather than requiring download. Also, explain the steps,
starting with validation. The form is very confusing. When should I click
validation, and why? And what is supposed to happen next?
If we display the errors, then we can ask the user to please review and
correct them. But then we should allow " force load with errors".

On Fri, Jan 18, 2019, 4:27 PM dustymc [e-mail] wrote:

[image: screen shot 2019-01-18 at 3 24 13 pm]
https://user-images.githubusercontent.com/5720791/51417895-25a07480-1b35-11e9-89f5-c2f950e940de.png

http://arctos.database.museum/info/ctDocumentation.cfm?table=CTAGENT_NAME_TYPE

initials are "middle name" - you can put them in any name_ and
corresponding name_type column.

Anyone with manage_codetables can force-load things with warnings.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1612#issuecomment-455720106,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hOJ4CRc9cwnX29c-VeydcTdZ7ngFks5vElhsgaJpZM4VXQKA
.

[dustymc]

overly restrictive.

The Old Way involved cleaning up LOTS of duplicates. I'm not completely opposed to going back, but I think getting someone familiar with Arctos involved early is ultimately a lot less work for everyone.

For one, display the results with validation errors to be corrected, rather than requiring download.

There's a viewer which works with smallish datasets.

When should I click validation

When you have unvalidated data in the staging table.

why

So you can find and fix problems.

And what is supposed to happen next?

You'll get some path to load, or instructions to deal with problems, depending on your rights and errors.

" force load with errors".

That's not always possible, there are many kinds of problems.

[lin-fred]

Now that I've messed with it once or twice now I'm going to document my next upload and will post here anything that is confusing so we can clarify what exactly it is that we want to change or not change. I will also follow the directions in the Handbook to see if anything needs clarifying.

[Jegelewicz]

@lin-fred Thank you! I was trying to work through this and it was making me nuts!

[Jegelewicz]

display the results with validation errors to be corrected, rather than requiring download. If we display the errors, then we can ask the user to please review and correct them.

Yes - like the AJAX grid we can use in specimen bulkload

Also, explain the steps, starting with validation. The form is very confusing. When should I click validation, and why? And what is supposed to happen next?

@lin-fred if you can propose a better "look" for this page, that would be a start. Currently the instructions are just thrown up there.

[dustymc]

ike the AJAX grid

[Jegelewicz]

If Agent Preload Thingee is the AJAX grid tool, can we be consistent and call it "fix/view with Arctos tools" like we do in the main bulkloader?

[acdoll]

I don't know why we have access to this tool if half the time you need CT level access to use it. People abbreviate their middle names- I want that to show in the preferred name.

Anyone with manage_codetables can force-load things with warnings.

There should be a link right there to contact someone with codetable access. Better yet, maybe a button to save your data in a holding table and a request for upload will be sent directly to someone who oversees the agent list. If I have to volunteer myself to make something like that happen, I will. In the mean time - could someone please force load these agents:

UA_Paleo_agents.zip

Also, in the code table AGENT_NAME_TYPE, what does the "old person." mean (e.g. old person.first_name)?

[Jegelewicz]

I was gonna load these, but I see this - are these correct?

[Jegelewicz]

I mean it does look like there are a lot of crazy long names, but just want to make sure before I pull the trigger!

[Jegelewicz]

Also, in the code table AGENT_NAME_TYPE, what does the "old person." mean (e.g. old person.first_name)?

That term doesn't exist in the code table. @dustymc ?

[dustymc]

don't know why we have access to this tool if half the time you need CT level access to use it.

You can do lots without final access - match agents to Arctos agents, use the cleanup tools, etc., etc., etc. The CT-access is just an attempt to get an experienced Arctos user involved before you load 957 ways of saying "that guy" and we all spend the next decade stumbling over them while also failing to find stuff. I think the idea remains sound, but we obviously need a better proxy for "experienced Arctos user."

old person

There once was a table called "person" (which included some pigeonholes into which a lot of persons don't fit comfortably). Its been gone for years, the documentation could be cleaned up.

[campmlc]

I think Arctos officers should have code table access if that is required
to load agents. It does seem very onerous to limit loading agents with a
middle initial to only those with code table access.

On Thu, Jul 23, 2020 at 12:16 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

don't know why we have access to this tool if half the time you need CT
level access to use it.

You can do lots without final access - match agents to Arctos agents, use
the cleanup tools, etc., etc., etc. The CT-access is just an attempt to get
an experienced Arctos user involved before you load 957 ways of saying
"that guy" and we all spend the next decade stumbling over them while also
failing to find stuff. I think the idea remains sound, but we obviously
need a better proxy for "experienced Arctos user."

old person

There once was a table called "person" (which included some pigeonholes
into which a lot of persons don't fit comfortably). Its been gone for
years, the documentation could be cleaned up.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1612#issuecomment-663157458,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBGRRUX6XQHON5QUELDR5B47RANCNFSM4FK5AKAA
.

[Jegelewicz]

If we had an Agents Committee - those people could help with this too.

Honestly though, I have had to do this about 5 times since January - it really doesn't happen very often. Also, @acdoll is an exception - I usually end up loading for someone who is either new to Arctos or an occasional user. I think I side with Dusty here that this helps spot trouble before it starts.

[campmlc]

I mean onerous for those operators who have to disrupt what they are doing
to ask for permission. Again, at the very least, all officers should have
this capability.

On Thu, Jul 23, 2020 at 2:05 PM Teresa Mayfield-Meyer <
[email protected]> wrote:

• [EXTERNAL]*

If we had an Agents Committee - those people could help with this too.

Honestly though, I have had to do this about 5 times since January - it
really doesn't happen very often....

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1612#issuecomment-663206133,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBAY3VRKYPQ2KXEXYZDR5CJYXANCNFSM4FK5AKAA
.

[Jegelewicz]

I don't see how we could respond any faster - it was only 20 minutes after Andy posted this that I went to load it.

[campmlc]

No complaints about your turnaround time! But I think more than one person
should be able to do this, and be familiar with the process. I'm just now
trying to teach myself what is involved with loading agents. I still think
that adding a middle initial in preferred name should not force require a
code table admin to load. This is way too common a situation. Why not just
a pop up requesting that middle initials be replaced with names if such are
available? Even if they are, the operator doing the load will know this,
not the code table admin.

On Thu, Jul 23, 2020 at 2:12 PM Teresa Mayfield-Meyer <
[email protected]> wrote:

• [EXTERNAL]*

I don't see how we could respond any faster - it was only 20 minutes after
Andy posted this that I went to load it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1612#issuecomment-663208820,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBBM35PSZ6N2FLS2EPTR5CKQ7ANCNFSM4FK5AKAA
.

[dustymc]

more than one person should be able to do this,

achinn (open)
ajlinn1910 (open)
aren (open)
campmlc (open)
ccicero (open)
dgstewart (open)
dlm (open)
ebraker (open)
ekrimmel (locked)
fskbh1 (open)
gordon (locked)
jegelewicz (open)
jldunnum (open)
katherinelanderson (locked)
lam (open)
mkoo (open)
uam (locked)

adding a middle initial in preferred name should not force require a code table admin to load.

That pattern comes from two places

• that's all they know
• they're tossing out data because they don't adequately understand the Arctos Agents model

Scripts can't tell those situations apart. Experienced Arctos operators can. Cleaning up from the second is a tremendous amount of work. Asking some admin to push the button, hopefully after asking some relevant questions, is a few minutes.

Why not just

I presume it's because people don't read the documentation, but whatever the reason instructions have failed to prevent a significant problem a couple times, which has lead to here.

The Community is a vitally important part of Arctos. I don't understand the reluctance to utilize that resource.

[Jegelewicz]

@acdoll please confirm that these are OK - https://github.com/ArctosDB/arctos/issues/1612#issuecomment-663133098

[AWildAshAppeared]

Hi Teresa, I hope it's ok that I'm just jumping in here, but I can confirm that those names are correct. I put the list of names together and then had it checked for spelling by the curators, so they are good to go.

[Jegelewicz]

Thanks @AWildAshAppeared ! I thought that Jegelewicz was a long last name.....

The agents are loaded!

[AWildAshAppeared]

Thanks very much @Jegelewicz !

[acdoll]

Thanks @Jegelewicz ! And sorry @dustymc for letting my frustrations bubble over here - I do understand that the point of being restrictive here is to prevent a massive pile of ... bad data. This middle initial thing has just grated me for quite a while. It's not even much of a problem now that we've stopped uploading the names of people who bring animals to rehab centers (due to multiple complaints about posting publicly). I guess I just have to get over my reluctance towards asking for help on GitHub; and then pay better attention when I do.

[dustymc]

By all means, bubble away! I know lots about things that don't work, these conversations sometimes lead to things that do work.

[dustymc]

I'm rounding up and standardizing bulk tools. There's some good info in here, so I'll keep going instead of starting a new thread.

I don't think we should attempt to move the agent loader into the "component loader" format - it's closer to Code Tables/Authorities than adding toe length to dead rats and needs to stay interactive; it deserves a dedicated form (and perhaps a new position in the menus).

The current loader is single-user - I'm tempted to leave that alone, to really push new loads of agents into a community effort rather than a siloed "naive user with lots of jumbled strings" situation. The documentation should better reflect that.

I'm still fond of the approach of paranoid checks plus leveraging The Community for final loading, but perhaps that should be two tools - a cleanup/check accessible by most anyone, plus a more code-table-like final load? It would be more to maintain, perhaps more difficult to navigate, but still might help clarify the process. And the check could be multi-user, which would eliminate/minimize any waiting around for someone else to get done before loading.

The inline editing tools exist mostly because it was fairly painful to write to Oracle. PG is good at that, I sort of want to get rid of them and just stick to upload/download and editing in local tools. Alternatively they could be rebuilt - if we keep them, we need to do so in datatables.

Discuss/add, please - how do we make bulkloading agents better?

[Jegelewicz]

I feel like I would need to load some agents to make good comments here. The next time that happens, I'll try to remember to do that.

[acdoll]

I'm still fond of the approach of paranoid checks plus leveraging The Community for final loading, but perhaps that should be two tools - a cleanup/check accessible by most anyone, plus a more code-table-like final load?

That should work for us. If they load into a datatable that shows the status of your basic checks (e.g. fatal error, possible duplicate) and allows that user to do some clean up (or just delete and reload), that would be great. Then, could we have a 'mark to load' button that notifies our agent or code table committee that there are agents waiting to be loaded?