Arctos: Separate Part Name from Part Preservation

[campmlc]

Proposal to separate part name (e.g. "tissue") from part preservation (e.g. "frozen").
Posting this as new issue related to repeated earlier discussion.

From email string:
These are clearly part attributes - they apply to parts, there can be many of them, they're determinations, they're time-dependent, etc. You can add them with the "add more" widget on data entry. We'll need to set up code table values before you can use them - that's fairly trivial, you can do it under edit code tables.

They can be bulkloaded with the parts bulkloader, and are in partdetail in specimen results/downloads.

I agree that preservation should not be confounded with part name. Getting it out affects lots of things (labels, the "tissues" search, data entry, etc.) so will need some discussion. Issue please.

---

Current status: https://docs.google.com/spreadsheets/d/1AY1EXQzKUSAg9EvZFjayTXcCBciXd_syCxt0GSX2tww/edit#gid=485619714 will serve as a migration path on 2021-07-01

[campmlc]

Previous issues: #212, #319, #352, #991, #1020, #1084, #1119, #1131, #1203, #1384

[dustymc]

Thanks @campmlc !

This isn't (yet...) a technical issue, so I'm unassigning me. The structure to do this all exists, I think we just have to work out how to use it (and/or decide if we can use it).

I think a model in which attributes of a physical object are in a 1-->many relationship with that object is "correct" (more correct, anyway) from a data modeling standpoint, but I'm also not entirely sure it's usable. Given the choice between correct and usable, I'll always (try to!) take usable.

212, #991, #1020: maybe we should toss the whole part model and start blank-slate - the current model has known problems with no clear solutions.

Also for any developing wish-list: find all the ribs. They're currently under "rib" and "body" and "whole organism" and "embryo" and (sorta/maybe) "fossil" and ....

319 got revived for and implemented as https://github.com/ArctosDB/arctos/issues/800. It's clearly "more correct" as preservation history than what we're proposing here (eg, "frozen" is ambiguous, TEMPERATURE at TIME isn't) but I think we decided it wasn't usable as part of the GGBN work.

352 introduced the structure which allowed this.

1084: https://github.com/ArctosDB/arctos/issues/1084#issuecomment-334190059 is the summary (we can't articulate what a "tissue" is).

1119 is pretty much this and #319.

1131 and https://github.com/ArctosDB/arctos/issues/1203 are a bit like https://github.com/ArctosDB/arctos/issues/1084#issuecomment-334190059 - we did some stuff, things definitely got better for it, but "is this a part?" still has a very arbitrary answer.

https://github.com/ArctosDB/arctos/issues/1384 just looks like another symptom of #319 to me - qualitative labels (excellent, rotten) for what is essentially quantitative data (time at temperature) are weird (albeit perhaps necessary if anyone's going to interact with the data!).

[dustymc]

find/make example in test, post here

[Jegelewicz]

AWG agrees separation would be better. Use a part attribute to maintain history. Add part attributes to data entry and bulkloader. Controlled vocabulary unless someone has a good reason not to. @dustymc ? put an example here for us to see how it will look.

[dustymc]

http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187

[Jegelewicz]

This doesn't look any different from what we have now.

I think we were expecting
part=DNA
preparation=frozen

[campmlc]

What this issue refers to is splitting "DNA" from "frozen", or any physical
part from it's fixation or preservation medium. This means everything in
parentheses needs to be transferred to one or more controlled vocabulary
fields. In many cases, we will need more than one ("liver (95% ethanol,
frozen)" = part name "liver"; fixation "95% ethanol", storage: "frozen".

If we use part attributes for this, I'd like them to be in an expandable
dropdown so that we don't have to look at the entire list for each part
unless we click "expand".

On Wed, Aug 1, 2018 at 1:33 PM, Teresa Mayfield [e-mail]
wrote:

This doesn't look any different from what we have now.

[image: image]
https://user-images.githubusercontent.com/5725767/43544115-33fe1b4e-958f-11e8-9585-543e74540c20.png

I think we were expecting
part=DNA
preparation=frozen

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409694849,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hHA_ZUeQBJBXB7ng9WSbeIuIIt0Rks5uMgKSgaJpZM4SXYNg
.

[dustymc]

doesn't look any different

It's not, which is why I've been saying "go for it!" for some time now.

part=DNA
preparation=frozen

That's just data and doesn't change any functionality. I'm happy to help with updates and such, if you want to do something in your collection or can convince other collections to come along or whatever.

expandable dropdown so that we don't have to look at the entire list for each part unless we click "expand".

I'm not sure what this means.

[campmlc]

Great. I believe we have consensus to make this a global change. So we
would need you to split out everything in parentheses, and add a separate
part preparation field that allows multiple concatenated searchable values
into the parts view. This would need to be on same line as disposition,
condition, etc. as follows:
Part Name: "liver"
Part Preparation: "95% ethanol; frozen"
Part Disposition: "in collection"
Part Condition: "good"

The last comment refers to the part attributes display. I am requesting to
make this an expandable drop-down visible only when clicked, to reduce the
overwhelming visual clutter on the page.

On Aug 1, 2018 2:00 PM, "dustymc" [e-mail] wrote:

doesn't look any different

It's not, which is why I've been saying "go for it!" for some time now.

part=DNA
preparation=frozen

That's just data and doesn't change any functionality. I'm happy to help
with updates and such, if you want to do something in your collection or
can convince other collections to come along or whatever.

expandable dropdown so that we don't have to look at the entire list for
each part unless we click "expand".

I'm not sure what this means.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409702539,
or mute
the thread
https://github.com/notifications/unsubscribe-auth/AOH0hE1zZNCQ_DX0tCTAGF4bnOaG_TGrks5uMgjJgaJpZM4SXYNg
.

[dustymc]

consensus

I hope so, but this should definitely go to the group.

add a separate part preparation field

That is a part attribute; there's nothing to add.

multiple concatenated searchable values

Hu? http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187 has change over time.

You'd concatenate it into "postcranial skeleton (rotten, spiffy, rotten, spiffy)"??

part attributes display...visible only when clicked

I think that's a huge problem. The thing people actually look at just says "liver" so they request tissue samples or etc - the DATA are liver (in formalin, which you can't see). That seems like it's going to cause usability problems.

[campmlc]

The preservation is just as important as condition and disposition, so it
should display on the main line of the part field. I don't care if is an
attribute in the underlying data table. The data structure does not have to
determine the visual display. I'm suggesting, as others have previously
done, that having all the part attributes visible for all parts makes the
visual display virtually impossible. So, add preservation as one of the
cells in the top row, and if there is more than one value, concatentate
them, as we do with the json string in part location path. Keep it in the
attribute table too, if necessary, with the dates etc. But make the part
attribute table optionally visible with an expand function.
We could actually do the same with condition -and just show the most recent
attribute value . . .

Happy to discuss this with the AWG, but we already have had multiple people
complain about the messiness of the current attribute display.

On Wed, Aug 1, 2018 at 3:54 PM, dustymc [e-mail] wrote:

consensus

I hope so, but this should definitely go to the group.

add a separate part preparation field

That is a part attribute; there's nothing to add.

multiple concatenated searchable values

Hu? http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187 has change
over time.

[image: screen shot 2018-08-01 at 2 53 56 pm]
https://user-images.githubusercontent.com/5720791/43551126-c2a679c6-959a-11e8-87b6-e1ec184f6e1f.png

You'd concatenate it into "postcranial skeleton (rotten, spiffy, rotten,
spiffy)"??

part attributes display...visible only when clicked

I think that's a huge problem. The thing people actually look at just says
"liver" so they request tissue samples or etc - the DATA are liver (in
formalin, which you can't see). That seems like it's going to cause
usability problems.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409737178,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hMKA_dSDjpS8MnUIqtfmsiTA-PB_ks5uMiOpgaJpZM4SXYNg
.

[dustymc]

preservation is just as important as condition and disposition

Moreso maybe, but it's a different type of data.

concatentate them, as we do with the json string in part location path

Those are different things - JSON is capable of carrying the "this is not the current determination" data and would satisfy my concerns.

In any case I think that's details - we're all merged on storing the data as part attributes, correct?

[acdoll]

I think we are asking to separate 'preservation' out as a different type of data so that we can have something like this:

That will allow multiple concurrent preservation types (and also could be tracked over time).

[campmlc]

Yes, that is my understanding. Thanks for the demo, Andy.

On Wed, Aug 1, 2018 at 6:49 PM, Andrew Doll [e-mail]
wrote:

I think we are asking to separate 'preservation' out as a different type
of data so that we can have something like this:
[image: image]
https://user-images.githubusercontent.com/12507297/43556164-8e6b63a8-95bb-11e8-8dc5-f7187eb79083.png

That will allow multiple concurrent preservation types (and also could be
tracked over time).

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409770092,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hM06A2TYhbMYv1uLq1ln5MGOtKcMks5uMkyhgaJpZM4SXYNg
.

[dustymc]

AWG:

• we DO want to do this for all Arctos
• minimize default info - add expand button - remember last action??
• default part name (latest attribute inf) [clicky2expand]

[campmlc]

Separate part from preservation; use part attributes, but show preservation as most recent preservation in part attributes.
All/none expand/collapse of part attributes. Default is that they are not visible.
AWG 8-9-18

[KyndallH]

Where are we with this??

[dustymc]

If we're forcing this on everyone, we need to let them know and I need a lookup table to make changes.

If you just want to use it, go ahead - this changes nothing in the model and has always been possible.

[Jegelewicz]

this changes nothing in the model and has always been possible.

I disagree. Parts are currently named "part (preservation)". This would be a pretty big change and it is one we have been asking for since I think June or July. I don't think any of us believe we have seen a demonstration of the requested change in action and perhaps this needs a committee to get it done.

[dustymc]

That is not a model change, it's just data (and perhaps accompanying authorities).

This may end up requiring a new "part preservation" (and maybe other stuff) code table(s) under http://arctos.database.museum/info/ctDocumentation.cfm?table=CTSPECPART_ATTRIBUTE_TYPE.

This may lead to form changes, but as always I very strongly prefer to build that around existing data - I'll tackle that after someone's used this.

If this involves controlled vocabulary, then it will work exactly the same was as "tissue quality." Tissue quality uses http://arctos.database.museum/info/ctDocumentation.cfm?table=CTTISSUE_QUALITY because http://arctos.database.museum/info/ctDocumentation.cfm?table=CTSPEC_PART_ATT_ATT.

If it involves free text, it will work exactly the same as "part location."

http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187 contains many controlled and uncontrolled part attributes, and should adequately demonstrate any possible arrangement that this could take.

[KyndallH]

It would be controlled text.

Pretty much what currently is in () in the part code table would become preservation. So blood (flash-frozen) would be blood with flash-frozen in preservation. Blood would simply be blood.

[dustymc]

preservation

There's an eternal argument about what's preservation and what's storage and etc. in there, but I'm pretty happy to ignore that for now if ya'll are.

currently is in ()

create table temp_pn_has_parens as select distinct part_name,
regexp_substr(part_name, '\(([^\)]+)\)', 1,1,NULL,1) stuff_from_parens
from ctspecimen_part_name where part_name like '%(%';

temp_pn_has_parens.csv.zip

create table temp_pn_parendata as select distinct stuff_from_parens from temp_pn_has_parens;

temp_pn_parendata.csv.zip

If you want to add and populate a description column to that I can create a code table.

Some of these should (probably?) be multiple attributes - "formalin-fixed, 55% isopropanol," "70% ethanol, frozen," etc.

[campmlc]

We want to get away from the multiple attributes, as that causes the
exponential expansion of the code table. I had thought we'd use part
attributes to track history and multiple fixation/preservation categories,
and only have the most recent in the "preservation" display field. We are
trying to simplify something with a complex history. Or, can we concatenate
all the terms, so they can be chosen separately at data entry and added to
over time in part attributes, but display as a concatenated preservation
field when more than one term exists?

On Tue, Oct 30, 2018 at 4:44 PM dustymc [e-mail] wrote:

preservation

There's an eternal argument about what's preservation and what's storage
and etc. in there, but I'm pretty happy to ignore that for now if ya'll are.

currently is in ()

create table temp_pn_has_parens as select distinct part_name,
regexp_substr(part_name, '(([^)]+))', 1,1,NULL,1) stuff_from_parens
from ctspecimen_part_name where part_name like '%(%';

temp_pn_has_parens.csv.zip
https://github.com/ArctosDB/arctos/files/2531928/temp_pn_has_parens.csv.zip

create table temp_pn_parendata as select distinct stuff_from_parens from temp_pn_has_parens;

temp_pn_parendata.csv.zip
https://github.com/ArctosDB/arctos/files/2531931/temp_pn_parendata.csv.zip

If you want to add and populate a description column to that I can create
a code table.

Some of these should (probably?) be multiple attributes - "formalin-fixed,
55% isopropanol," "70% ethanol, frozen," etc.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-434496947,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hEAynCLvSiVaPypSjq6q_IAS-Wzfks5uqNZWgaJpZM4SXYNg
.

[dustymc]

multiple

If you mean "this and that" as one thing, agreed - that should be two attributes, "this" and "that" (and maybe some accompanying metadata that we can probably ignore most of the time).

most recent

I'm still not very comfortable with that - there'd be no obvious distinction between "55% isopropanol" and "55% isopropanol - but oh yea it was in formalin for years." It's not my data/this isn't my decision, but it still strikes me as wrong.

simplify

That's how we got to stuffing it all into a string!

concatenated preservation field

That seems better, although it's likely to be a little weird for some things. In any case, yes, I think we can abstract some stuff through interfaces.

(Oracle 12 speaks JSON natively and might be useful to simplify that sticking together and pulling apart, if we can figure out an upgrade.)

[Jegelewicz]

Started definitions in Google Drive: https://drive.google.com/open?id=1MyOtB1zrQHpO63pxMILDTUJgaTyZNEuC

[campmlc]

Here are my edits to the Part/Preservation split table. With a few exceptions as noted, I would agree with the recommendations to split multiple preservation types as separate part attributes, and to concatenate preservation types in the part preservation field in display field next to part condition and part disposition per Andy Doll's comment on Aug 1.
@acdoll @KyndallH @Jegelewicz

https://docs.google.com/spreadsheets/d/1d-5cNQzhh5o2H9UrSNI-yfOAcgAAdwIlKuRhC5cDmus/edit#gid=997912679

[Jegelewicz]

Looks good, now I'll throw in my wrench. Should we just call the attribute "conservation" or should we have two attributes: "preservation" and "fixation"? Just want to think through this all the way before we combine stuff under a label that doesn't represent everything we are talking about.

[dustymc]

That wrench has been flopping around for a decade or so - https://github.com/ArctosDB/arctos/issues/1460#issuecomment-434496947.

Something similar is part of what got us here. There's an incredibly fuzzy line between a few relevant components - I think the old was "part modifier" and "part preservation" or similar. Condition is always hanging around and hurling rusty wrenches as well, and now part attribute "tissue quality" is involved too. There are terms which can defensibly be placed in any of those "fields." Using something arbitrary results in a factorial of the number of fields ways of saying something. (And the number of fields is now dynamic, so we're not limited to ONLY 24 ways of saying the same thing across 4 fields - with a couple attributes on a couple of them we can easily get to 6!=720 ways of saying the same thing.)

"Correct" is almost certainly separating preservation, storage, conservation, part modifier, etc. "Usable" may mean not doing that, or at least not doing that without drawing some sort of uncrossable line between the values we'll store in the various components. Given the choice between "correct" and "usable," I'll sort of always vote for usable.

[campmlc]

Ideally all of this would be most "correctly" tracked through object
tracking/part history, where we'd have all the above - fixtation,
preservation, storage, location, refill/topoff/remount/reframe/check dates
and by whom, plus history of freezer temp logs or humidity levels etc etc.
And then we could have a reciprocal link to this info from the parts table,
rather than a full display.
Regardless of where the data are being added and to what field, it would be
most "usable" to have a single field on the parts table with some sort of
summary, eg from the current edited table, with concatentated info.

For future, maybe we should require all parts to have a label, and all
parts to feed through object tracking and check history?

On Fri, Nov 9, 2018 at 9:50 AM dustymc [e-mail] wrote:

That wrench has been flopping around for a decade or so - #1460 (comment)
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-434496947.

Something similar is part of what got us here. There's an incredibly fuzzy
line between a few relevant components - I think the old was "part
modifier" and "part preservation" or similar. Condition is always hanging
around and hurling rusty wrenches as well, and now part attribute "tissue
quality" is involved too. There are terms which can defensibly be placed in
any of those "fields." Using something arbitrary results in a factorial of
the number of fields ways of saying something. (And the number of fields is
now dynamic, so we're not limited to ONLY 24 ways of saying the same thing
across 4 fields - with a couple attributes on a couple of them we can
easily get to 6!=720 ways of saying the same thing.)

"Correct" is almost certainly separating preservation, storage,
conservation, part modifier, etc. "Usable" may mean not doing that, or at
least not doing that without drawing some sort of uncrossable line between
the values we'll store in the various components. Given the choice between
"correct" and "usable," I'll sort of always vote for usable.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-437421231,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hJU5KcGy827DhO4h4yDeYrvuB8cLks5utbJHgaJpZM4SXYNg
.

[dustymc]

most "correctly" tracked through object tracking

Absolutely, but there was a decision to denormalize to part attributes.

reciprocal link to this info f

That's always been there.

require all parts to have a label, and all parts to feed through object tracking and check history

I don't think that's realistic - it would just make Arctos unusable for a bunch of collections.

[Jegelewicz]

Ideally all of this would be most "correctly" tracked through object tracking/part history, where we'd have all the above - fixtation, preservation, storage, location, refill/topoff/remount/reframe/check dates and by whom, plus history of freezer temp logs or humidity levels etc etc.

This would still be problematic for legacy data where none of that was ever recorded. I am totally OK with what has been agreed to in the Google sheet, just wanted to make sure we discussed this now rather than later. Sounds like the most usable method for now and will accomplish what is needed.

[campmlc]

Yes, I think we go with what we have for now. I do like the idea of
incorporating more info via object tracking/history at some point - but
honestly don't have any idea how that can be better integrated to be more
usable.

On Fri, Nov 9, 2018 at 10:11 AM Teresa Mayfield-Meyer <
[email protected]> wrote:

Ideally all of this would be most "correctly" tracked through object
tracking/part history, where we'd have all the above - fixtation,
preservation, storage, location, refill/topoff/remount/reframe/check dates
and by whom, plus history of freezer temp logs or humidity levels etc etc.

This would still be problematic for legacy data where none of that was
ever recorded. I am totally OK with what has been agreed to in the Google
sheet, just wanted to make sure we discussed this now rather than later.
Sounds like the most usable method for now and will accomplish what is
needed.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-437427871,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hDfhPeOKmnVolbzzoMRAep6shxECks5utbckgaJpZM4SXYNg
.

[dustymc]

To Do (& please make my terminology better!):

1) New code table ctpart_preservation (part_preservation NOT NULL, description NOT NULL)
2) Populate with part_preservation=EDITED, description=Definition from https://docs.google.com/spreadsheets/d/1d-5cNQzhh5o2H9UrSNI-yfOAcgAAdwIlKuRhC5cDmus/edit#gid=997912679
3) New entry "preservation" in https://arctos.database.museum/info/ctDocumentation.cfm?table=CTSPECPART_ATTRIBUTE_TYPE
4) New entry in https://arctos.database.museum/info/ctDocumentation.cfm?table=CTSPEC_PART_ATT_ATT (to make the attribute value-controlled)
5) pull MSB parts, separate part and preservation, update

• Map should mostly be in https://docs.google.com/spreadsheets/d/1d-5cNQzhh5o2H9UrSNI-yfOAcgAAdwIlKuRhC5cDmus/edit#gid=997912679 but there will probably be some stuff to deal with manually, multiple preservation, etc.
  6) MSB will use, see if we need any additional forms/tools/whatever
  7) update for all collections

[Jegelewicz]

Populate with part_preservation=EDITED, description=Definition from https://docs.google.com/spreadsheets/d/1d-5cNQzhh5o2H9UrSNI-yfOAcgAAdwIlKuRhC5cDmus/edit#gid=997912679

See my edits.

[dustymc]

Can we get rid of "unknown"? I don't think we need to go out of our way to say we don't know....

[Jegelewicz]

Hmmmm. It seems odd, but in a way may be useful. If we really don't know then it might be good to say so, otherwise the lack of a preservation attribute might be a red flag that we need to enter one (low quality data).

[dustymc]

Certainly not stopping anything, and this is an incredibly common thing, but it's still denormalization. If I find a "widget" I can't know anything about how it's preserved (despite the various "as is traditional..." and "the normal thing..." and etc. documentation floating around). If I find a "widget (pres=unknown)" I can't know anything about how it's preserved (despite someone telling me they have nothing to tell me). Two ways of saying one thing==>bad.

Oh well...

[dustymc]

Do NOT edit the spreadsheet - edits from now on will be ignored.

[dustymc]

Is there some distinction between

desiccated
having all moisture removed using either heat or silica

and

mummified
embalmed and wrapped in cloth or allowed to shrivel or dry up

??

[dustymc]

1-4 in https://github.com/ArctosDB/arctos/issues/1460#issuecomment-453289966 are implemented; this is usable in production.

formalin, 5% buffered and formalin, 4% buffered fell out due to having no definition - they can be added to the code table manually.

Is "fluid preserved" somehow usefully different than unknown/NULL?

[Jegelewicz]

Is "fluid preserved" somehow usefully different than unknown/NULL?

Sure! you know it isn't desiccated or mummified....

Not sure about keeping those two as separate terms, but others may have good reasons....I know that there is at least one "mummy" that was made naturally in the UTEP collections so really neither of those definitions apply to it.

[dustymc]

isn't desiccated or mummified....

I suppose that depends if air fits in your definition of fluid or not...

I think @KyndallH @amgunderson have some naturally-desiccated stuff too.

[dustymc]

isn't desiccated or mummified....

I suppose that depends if air fits in your definition of fluid or not...

I think @KyndallH @amgunderson have some naturally-desiccated stuff too.

@campmlc @Jegelewicz there's a new spreadsheet of MSB parts at https://docs.google.com/spreadsheets/d/1jTgHlcdu_bptiHx7FTK17PYUFfpHsrixfX_favM1kGc/edit#gid=25221597

PAREN_BIT - ignore
PART_NAME - original part name - don't touch this, it's my path back
BARE_PART - new part name, I'll probably need to create some of these
PRES1 - preservation (from the last spreadsheet)
PRES2 - preservation (because some will need two - feel free to add more columns if necessary)

http://arctos.database.museum/info/ctDocumentation.cfm?table=CTPART_PRESERVATION should be complete for preservation, but we can add stuff if necessary.

Vague plan from here:

1. update MSB specimens to this vocabulary
2. throw up a temporary trigger disallowing parentheses in parts for MSB (so you'll have to separate preservation)
3. do whatever we have to do to make this work for you
4. push to everyone else
5. delete parts with parentheses (and disallow them in the code table)

[Jegelewicz]

Soooooo. Let me complicate things.

For anything that cuurently has "YYYY, XXXX-fixed" should we have two attributes:
"fixation" with the value "XXXX"
"preservation" with the value "YYYY"

or are we going with two preservation attributes:
"preservation" with the value "XXXX"
"preservation" with the value "YYYY"

[dustymc]

fixation/preservation

https://github.com/ArctosDB/arctos/issues/1460#issuecomment-437421231

That's probably "correct" but I'd rather avoid the eternal "which is THIS?" debate (which inevitable ends up as a little of this, a little of that, and nobody can find anything).

The second option works for me.

[campmlc]

We could do second option and put "fixed" in attribute remarks.

On Mon, Jan 14, 2019 at 4:13 PM dustymc [e-mail] wrote:

fixation/preservation

1460 (comment)

https://github.com/ArctosDB/arctos/issues/1460#issuecomment-437421231

That's probably "correct" but I'd rather avoid the eternal "which is
THIS?" debate (which inevitable ends up as a little of this, a little of
that, and nobody can find anything).

The second option works for me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454199894,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hPuMk-mk_NR8LcJLPAA_1oKeeo9Nks5vDQ8QgaJpZM4SXYNg
.

[dustymc]

remarks

Why would anyone care? If the answer could possibly involve searching, remarks (free-text in general) isn't the solution. If people only care after they've found the specimen, that might work (even when you've inadvertently and inevitably entered "fxied").

I still need the spreadsheet in https://github.com/ArctosDB/arctos/issues/1460#issuecomment-453707026 filled out.

[KyndallH]

Dessicated is more a plant term and it is different because they actively dry their samples. If you just let it dry out by itself, it actually reduces the quality of the tissue. And yes, I think Aren has some mummified stuff.

[KyndallH]

Though I wouldn't mind mummified going away. That is a horrible term.

[campmlc]

how about "air-dried" vs "freeze-dried"? vs just "dry" which is - we don't
know the exact method.

As for why anyone would care about remarks, yes, people may only care after
they've found the specimen. For collection managers, this is the info I
need to determine suitability of material for loans.

On Tue, Jan 15, 2019 at 12:37 PM Kyndall Hildebrandt <
[email protected]> wrote:

Though I wouldn't mind mummified going away. That is a horrible term.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454522731,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hB2856TTLMDl5lwjkcEDtwYk51uCks5vDi37gaJpZM4SXYNg
.

[dustymc]

plant term

The code table says "having all moisture removed using either heat or silica." "Silica" makes sense (to me...) as preservation - and "heat"==>"air-dried"??

info I need to determine suitability of material for loans

Sounds like a compelling reason to standardize.

Option One: You get a request for 7000 parts (biggest current loan that seems to actually examine specimens and not just data is 7054 parts). You need to find 7000 parts and read the remark for each and every one of them.

Option Two: You get a request for 7000 parts. You need filter for any that are "fixed."

[campmlc]

Happy to keep "fixed" in the part attribute name. As I've said before:
Ideally all of this would be most "correctly" tracked through object
tracking/part history, where we'd have all the above - fixtation,
preservation, storage, location, refill/topoff/remount/reframe/check dates
and by whom, plus history of freezer temp logs or humidity levels etc etc.
And then we could have a reciprocal link to this info from the parts table,
rather than a full display.

However, this what was was suggested yesterday:
"preservation" with the value "XXXX"
"preservation" with the value "YYYY" (for fixation).

I don't like that option unless we can specify "fixation" somewhere.
Otherwise, let's go with
For anything that cuurently has "YYYY, XXXX-fixed", have two attributes:
"fixation" with the value "XXXX"
"preservation" with the value "YYYY"

On Tue, Jan 15, 2019 at 2:43 PM dustymc [e-mail] wrote:

plant term

The code table says "having all moisture removed using either heat or
silica." "Silica" makes sense (to me...) as preservation - and
"heat"==>"air-dried"??

info I need to determine suitability of material for loans

Sounds like a compelling reason to standardize.

Option One: You get a request for 7000 parts (biggest current loan that
seems to actually examine specimens and not just data is 7054 parts). You
need to find 7000 parts and read the remark for each and every one of them.

Option Two: You get a request for 7000 parts. You need filter for any that
are "fixed."

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454563106,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hEk6feefuqSHsbUE4NYP-KQlv9X6ks5vDkuagaJpZM4SXYNg
.

[Jegelewicz]

"fixation" with the value "XXXX"
"preservation" with the value "YYYY"

I prefer the two attributes because I believe there is a difference between fixing something in formalin, then storing it in ethanol and storing something in formalin, then transferring it to storage in ethanol.

[dustymc]

two attributes

That's probably "correct" but...

1) I'm not completely convinced there's a clear separation, and if there's not then arbitrary usage will make it impossible to find specimens. If there is a clear separation - given the same thing the kiddos will have no choice but to enter things the same way - then this isn't a problem.

2) That's another weird code table that we'd have to look at to determine "tissue-ness." Not a deal-breaker, but it's some added complexity (and processors) that I'm not certain is necessary.

unless we can specify "fixation"

Is there some useful/functional difference between plunking something in formalin for a while and then moving it to ethanol, and fixing it in formalin and then moving it to ethanol? If not, then formalin + ethanol probably make sense. If so, then we probably need something like formalin (fixative) and formalin and ethanol - a "fixed" part would go through the first and last, something stored in formalin would use the middle (and maybe the last if it's transferred to ethanol).

[KyndallH]

Fixation is the same as preservation but at different times. Fixation is the initial preservation.

The code table says "having all moisture removed using either heat or silica." "Silica" makes sense (to me...) as preservation - and "heat"==>"air-dried"??
No -they actually have ovens that dry them at a low heat though higher than ambient temperature with circulating air.

[campmlc]

From
http://clade.ansp.org/malacology/people/rosenberg/archiving/method/fixation_preservation.html
:
"The words fixation and preservation are used casually and often
incorrectly in the collection care literature. There is a distinct
difference between a fixative and a preservative. In general, fixation is
more desirable for histological or detailed morphological studies, but
specimens should never be stored in any fixative for long (or even short)
periods of time, and indeed, with many crustaceans, fixation may not be
necessary at all (as opposed to preservation in ethanol or other
preservative).

A fixative actually "fixes" a specimen by stabilizing (cross-linking) the
proteins within its tissues such that long afterwards, the tissues will
still retain a semblance of their appearance in life. Additionally, fixing
usually raises the refractive index of the tissue making it more
susceptible to staining. By definition, then, a fixative is a toxic
chemical with a strongly adverse affect on tissue, in a sense chemically
"freezing" the tissues permanently in place. "

On Tue, Jan 15, 2019 at 5:27 PM Kyndall Hildebrandt <
[email protected]> wrote:

Fixation is the same as preservation but at different times. Fixation is
the initial preservation.

The code table says "having all moisture removed using either heat or
silica." "Silica" makes sense (to me...) as preservation - and
"heat"==>"air-dried"??
No -they actually have ovens that dry them at a low heat though higher
than ambient temperature with circulating air.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454604358,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hLs7XJwEMMfUoYeWsXm2fL-2uADWks5vDnHvgaJpZM4SXYNg
.

[campmlc]

Here's another:
" Fixation involves the preservation and stiffening of tissues so that
they do not decompose and so that the specimen remains in an easy-to-study
posture. Preservation involves the storage of the specimen in a medium that
allows for its safe study for many years. "
https://www2.clarku.edu/faculty/pbergmann/Resources/SOP%20Specimen%20Preservation.pdf

95% ethanol is a fixative in the sense that it dehydrates (dessicates!)
tissues such that they are hardened and inflexible and protected from
decomposition. But it can also be a preservative for DNA.

On Tue, Jan 15, 2019 at 5:47 PM Mariel Campbell [e-mail]
wrote:

From
http://clade.ansp.org/malacology/people/rosenberg/archiving/method/fixation_preservation.html
:
"The words fixation and preservation are used casually and often
incorrectly in the collection care literature. There is a distinct
difference between a fixative and a preservative. In general, fixation is
more desirable for histological or detailed morphological studies, but
specimens should never be stored in any fixative for long (or even short)
periods of time, and indeed, with many crustaceans, fixation may not be
necessary at all (as opposed to preservation in ethanol or other
preservative).

A fixative actually "fixes" a specimen by stabilizing (cross-linking) the
proteins within its tissues such that long afterwards, the tissues will
still retain a semblance of their appearance in life. Additionally, fixing
usually raises the refractive index of the tissue making it more
susceptible to staining. By definition, then, a fixative is a toxic
chemical with a strongly adverse affect on tissue, in a sense chemically
"freezing" the tissues permanently in place. "

On Tue, Jan 15, 2019 at 5:27 PM Kyndall Hildebrandt <
[email protected]> wrote:

Fixation is the same as preservation but at different times. Fixation is
the initial preservation.

The code table says "having all moisture removed using either heat or
silica." "Silica" makes sense (to me...) as preservation - and
"heat"==>"air-dried"??
No -they actually have ovens that dry them at a low heat though higher
than ambient temperature with circulating air.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454604358,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hLs7XJwEMMfUoYeWsXm2fL-2uADWks5vDnHvgaJpZM4SXYNg
.

[dustymc]

So that's talking about the substance - formalin is a fixative, ethanol (at some concentrations, for some purposes, except when nobody told the student it needs to be diluted....) is a preservative. Under that, "formalin fixed" is redundant. Since we can (by that definition) filter based on the substance (and not how it's used), I'm not seeing any advantage to the word "fix" in any context (if it is or was in a fixative that's presumable the only thing it could be there for), nor to separating fixatives into another table (we can identify them by what they are, so it doesn't matter where they are).

ovens

Would they use them if they were working in Albuquerque? Is the result different than freeze-drying? I'm guessing the time (at temperature??) it took to dry is what matters?? Eg, I think the goal here is (or should be) to determine suitability for use, and it's not clear how much technique we need to get that. I'm totally fine with having 900 ways to say "dry" if we need them, but they should be defined such that they're not arbitrarily used.

[Jegelewicz]

formalin is a fixative, ethanol (at some concentrations, for some purposes, except when nobody told the student it needs to be diluted....) is a preservative.

Not necessarily. Some specimens are preserved in formalin.....(amphibian eggs and larva)

[Jegelewicz]

I'm not seeing any advantage to the word "fix" in any context (if it is or was in a fixative that's presumable the only thing it could be there for), nor to separating fixatives into another table (we can identify them by what they are, so it doesn't matter where they are).

I think the attributes "fixation" and "preservation" could use the same code table, but I like the idea of having two different terms based upon Mariel's definitions above.

[Jegelewicz]

Would they use them if they were working in Albuquerque?

We didn't in El Paso.....

[dustymc]

preserved in formalin.

If I'm reading http://clade.ansp.org/malacology/people/rosenberg/archiving/method/fixation_preservation.html correctly, that's not possible - you (and most everyone else AFAIK) just left them in a fixative - eg, the difference is in the chemical, not in the technique.

two different terms

Elaborate please - are you suggesting "formalin (somehow used as a preservative)" and "formalin (somehow used as a fixative)"?

[campmlc]

Yes, 95% ethanol can also be both a fixative and a preservative. In this
case, if we had a specimen fixed in 95% ethanol and then transferred to 70%
ethanol as a preservative, and both were "preservative" according to
Arctos, we'd need a way to distinguish which came first. Fixative by
definition comes first before preservative. If we don't distinguish these
by terms or by remarks, we need a way to distinguish them based on order of
use.

We also need to consider data entry. Students now see "whole organism
(ethanol-fixed)". What will they see if we break this up into two part
attributes, and how will they indicate that it was fixed in 95% and then
transferred to 70% for long term storage?

On Wed, Jan 16, 2019 at 10:30 AM dustymc [e-mail] wrote:

preserved in formalin.

If I'm reading
http://clade.ansp.org/malacology/people/rosenberg/archiving/method/fixation_preservation.html
correctly, that's not possible - you (and most everyone else AFAIK) just
left them in a fixative - eg, the difference is in the chemical, not in the
technique.

two different terms

Elaborate please - are you suggesting "formalin (somehow used as a
preservative)" and "formalin (somehow used as a fixative)"?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454867486,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hJS-PD_TV3zJIcUhkmVKTeczLpj_ks5vD2GYgaJpZM4SXYNg
.

[dustymc]

[campmlc]

OK, that could work from a data standpoint. But from a data entry
standpoint, this is getting very complex. We don't want students to have to
use the add more tool for every part that is entered. We would need this to
be visually integrated into the standard data entry form. Possible?
Also, using attributes in this way should not require that CMs have to
approve "extra" data for every load. That doubles our workload and will
lead to staff resistance. How can we integrate the data entry and make it a
more seamless process that doesn't require multiple complex interrelated
steps?

On Wed, Jan 16, 2019 at 11:12 AM dustymc [e-mail] wrote:

[image: screen shot 2019-01-16 at 10 07 57 am]
https://user-images.githubusercontent.com/5720791/51269286-f512dc00-1976-11e9-8acd-4911f79fa92a.png

[image: screen shot 2019-01-16 at 10 10 13 am]
https://user-images.githubusercontent.com/5720791/51269291-f9d79000-1976-11e9-8b27-05d091b1a703.png

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-454882183,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hGUDQ87TuAhLS0MOdwBN_yKf7pcRks5vD2ttgaJpZM4SXYNg
.

[dustymc]

visually integrated

Certainly possible, but there are some things to work out. (Like screen real estate - there are a couple dozen possible "related" loaders, all of which can contain infinite "rows.")

not require that CMs have to approve "extra" data for every load

There are a bunch of old Issues about this - it's basically a matter of me not liking the idea of loading data that ya'll haven't seen. It's more-integrated than it was, but I don't really know how you approve stuff so I'm not sure if you see "extras" or not. In any case it's 100% social - there's no reason I can't auto-load "extras" instead of sending email (I suppose I'd still need to send email for the failures). Probably a new Issue if you want to revisit that - I think it's just a matter of making sure we're all on the same page as far as what the "load" button does.

If we get to Oracle 12 there might be some other possibilities - it allegedly speaks JSON (and has a datatype to hold it), so the form could write a "partstring " (many parts each with many attributes) into one field, and unroll it for display on the data entry form, when you're approving records to load, etc. I don't have the capability to move beyond theoretical at the moment, but that's a standard way of moving complex data around so from a very superficial viewpoint I don't see why it wouldn't work.

[Jegelewicz]

Elaborate please - are you suggesting "formalin (somehow used as a preservative)" and "formalin (somehow used as a fixative)"?

I am suggesting that in the image you created above, the first "preservation" would instead be "fixation".

[dustymc]

FYI that's just a screen shot of data entry in production.

the first "preservation" would instead be "fixation"

I think that's "arbitrary usage==nobody can find anything" territory.

• A specimen is stored in 95% ethanol because some student forgot to dilute.
• A specimen is being fixed in 95% ethanol and will be transferred to 70% tomorrow.
• A specimen is allegedly being fixed in 95% ethanol, but the transfer scheduled for tomorrow won't actually happen so it's really storage.
• A specimen is stored in 95% ethanol because whatever made http://arctos.database.museum/SpecimenResults.cfm?part_name=whole%20organism%20(95%25%20ethanol) (11249 specimens) exist happens.
• Etc. I don't see any reason a student would pick one or the other, and therefore I don't see any way a researcher (or collection manager) could find those specimens.

That's also one more weird code table, more expensive queries, etc.

I'd be more accepting of something that can ONLY appear in one table or the other, so it can't be arbitrarily used, but that gets us back to "95% ethanol is a fixative, not a preservative" which (1) seems to conflict with existing data, and (2) negates any benefit of the separation.

[campmlc]

Issues meeting 5-2-19
@campmlc OKs
@Dusty will proceed with MSB
Weirdness: tissueness comes from parts; but that comes from preservation; this might affect has tissues search
Use DGR:Mamm

[campmlc]

Set up separate tissue flag meeting to assign tissueness to preservation types
http://arctos.database.museum/Admin/CodeTableEditor.cfm?action=editCTPART_PRESERVATION&tbl=CTPART_PRESERVATION
Mariel
Carla
Andy
Others?

[dustymc]

go with DGR Mamm

[dustymc]

@campmlc @Jegelewicz DGR:Mamm data attached.

• CID - my identifier
• GUID - guid
• NEW_PART_NAME - part name will become
• NEW_PRES_1 - one preservation attribute will become
• NEW_PRES_2 - one preservation attribute will become
• PART_NAME - old/existing part name
• PID - my identifier

I added blood (EDTA) to https://docs.google.com/spreadsheets/d/1jTgHlcdu_bptiHx7FTK17PYUFfpHsrixfX_favM1kGc/edit#gid=25221597 - I don't think anything else of that nature slipped through the cracks, but a quick review of this before I proceed would be greatly appreciated - do you see anything wrong with the attached data?

I'm not sure what to do with this:

2279707 | DGR:Mamm:2094 | whole organism, dissected | DELETE, but any part currently with this preservation will need 2 attributes | whole organism, dissected (ethanol, formalin-fixed) | 24484329
-- | -- | -- | -- | -- | --

I suppose lacking better ideas I'll just ignore it for now??

Shall I proceed with normalizing DGR:Mamm parts?

temp_msbmamm_pts(1).csv.zip

[Jegelewicz]

Here is what I would do with that:

NEW_PART_NAME - whole organism
NEW_PRES_1 - formalin
NEW_PRES_2 - ethanol
Condition - dissected

[Jegelewicz]

Otherwise - I think the rest looks OK and will be a good test run, but let @campmlc decide!

[campmlc]

I agree with Teresa on that one. I made a couple of edits to the google
spreadsheet. Karyotype (dry, slide) should just be karyotype (slide).
K2CR2O7 should be potassium dichromate.
All else looks good to go.

On Mon, May 13, 2019 at 1:47 PM Teresa Mayfield-Meyer <
[email protected]> wrote:

Otherwise - I think the rest looks OK and will be a good test run, but let
@campmlc https://github.com/campmlc decide!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460?email_source=notifications&email_token=ADQ7JBG7UJBPL7CVSWN6RHDPVHAWVA5CNFSM4ES5QNQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJLRGY#issuecomment-491960475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQ7JBGJJ4MXFVSUUW72QTLPVHAWVANCNFSM4ES5QNQA
.

[dustymc]

@campmlc the header had been excel-sorted into the data - I recovered it back, I hope I didn't overwrite anything but please check.

Anything in the pres columns should also be in CTPART_PRESERVATION - please just put the values in there. Here are the problems:

UAM@ARCTOS>  select distinct PRES1 from temp_part_map where PRES1 not in (select PART_PRESERVATION from CTPART_PRESERVATION);

PRES1
------------------------------------------------------------------------------------------------------------------------
formalin, 5% buffered
DELETE, but any part currently with this preservation will need 2 attributes
use slide only; no need for dry

I added an append_to_condition column and can deal with that during the migration.

I think "whole organism, dissected (ethanol, formalin-fixed)" is where it needs to be now, but everything else with random stuff in the pres columns will need sorted out.

Here's the updated DGR data - I might have it pushed out before you see this....

temp_msbmamm_pts(2).csv.zip

[dustymc]

MSB:Mamm is changed.

[campmlc]

OK, I fixed the file - please check. DGR looks ok.

On Mon, May 13, 2019 at 3:28 PM dustymc [e-mail] wrote:

@campmlc https://github.com/campmlc the header had been excel-sorted
into the data - I recovered it back, I hope I didn't overwrite anything but
please check.

Anything in the pres columns should also be in CTPART_PRESERVATION -
please just put the values in there. Here are the problems:

UAM@ARCTOS> select distinct PRES1 from temp_part_map where PRES1 not in (select PART_PRESERVATION from CTPART_PRESERVATION);

PRES1

formalin, 5% buffered
DELETE, but any part currently with this preservation will need 2 attributes
use slide only; no need for dry

I added an append_to_condition column and can deal with that during the
migration.

I think "whole organism, dissected (ethanol, formalin-fixed)" is where it
needs to be now, but everything else with random stuff in the pres columns
will need sorted out.

Here's the updated DGR data - I might have it pushed out before you see
this....

temp_msbmamm_pts(2).csv.zip
https://github.com/ArctosDB/arctos/files/3174895/temp_msbmamm_pts.2.csv.zip

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460?email_source=notifications&email_token=ADQ7JBFRRFP7BVNEMWP2J7TPVHMONA5CNFSM4ES5QNQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJTUQI#issuecomment-491993665,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQ7JBAFWJCAU5VZ4GR4VMLPVHMONANCNFSM4ES5QNQA
.

[Jegelewicz]

Concatenate preservation methods for UI.

[dustymc]

add to bulkloader

• part_preservation_n_1
• part_preservation_date_n_1
• part_preservation_n_2
• part_preservation_date_n_2

EDIT: AWG says do it

[Jegelewicz]

Can we magic in "frozen" + date any time something is scanned into a freezer?

[dustymc]

Can we magic in "frozen" + date any time something is scanned into a freezer?

Yes, but I think that's another discussion.

• it's denormalization - who should we believe if there's a conflict?
• it's denormalization - do you really want 12000 'yup, still frozen' attributes? (You probably should...)
• it could get fairly complex - does scanning a rack into a freezer trigger this for all parts? (Probably yes, but not sure I have the processors to pull it off)
• etc.

[Jegelewicz]

Yeah, I thought about all that after I posted this - ideally the container history will suffice, it would just be nice not to also have to enter the preservation method as frozen, which kinda doubles the work. Maybe some brilliant way to do that will come to me in a dream....

[Jegelewicz]

No options when attempting to enter preservation part attribute.

I decided to try out the preservation part attribute, but when I select attribute type preservation, there are no choices in the value field.

[dustymc]

Stupid tissue thing...

Should be happy now - thanks!

[Jegelewicz]

Well it works for creating, but not for editing...

[dustymc]

blargh - reload....

[Jegelewicz]

@campmlc I used this on http://arctos.database.museum/guid/MSB:Mamm:129862

It should be easy to pull the "frozen" out and add an attribute, but the fact that this was moved from -20 to -80 (included in the specimen remarks) might take a little more effort. Now that you have this example - how would you like it to appear on the specimen record? Does it facilitate the kinds of searches you might do?

[Jegelewicz]

We now have the ability to create attributes for parts and thus separate preservation from the part name. Next up is to document, test, create a better UI (after PG), make recommendations for use.

See #2354

[dustymc]

This will have a major impact on essentially all collections, and therefore seems like a good test of https://github.com/ArctosDB/arctos/issues/2346.

[dustymc]

This seems to have floated to the top of my pile again, and we now have a functional test environment. I'm still concerned about how this might work (or fail to!) in the bulkloader. I think there are three obvious approaches.

1. Do nothing; use the 'extras' for attribute-having parts. This is the most robust solution (it'll deal with six of any attribute for any number of parts), but also has some separation in that it writes to multiple "files." https://github.com/ArctosDB/arctos/issues/2974 has the capacity to make this more integrated than it currently is, but I'm not sure that'll completely mitigate the disassociated nature of this approach.

2. Add to the specimen bulkloader

3. part_n_attribute_type_m
4. part_n_attribute_value_m
5. part_n_attribute_unit_m
6. part_n_attribute_date_m
7. part_n_attribute_determiner_m
8. part_n_attribute_remark_m

where n is the number of parts (currently 12) and m is the number of attributes per part (not sure what's realistic, 2 seems to be a minimum). That gets everything in one table, but it's a lot of numbers to keep straight and a lot of columns. part_12_attribute_6 (6 attributes is probably a "should handle most normal situations" number) would be (12 parts * 6 columns) + (6 attributes * 6 columns * 12 parts)==about a thousand part-columns. I hate it.

1. https://github.com/ArctosDB/arctos/issues/1460#issuecomment-517406899 would add (4 preservation-columns * 12 parts)==48 new columns, be fairly simple to understand, BUT only work for preservation (which I believe would exclude a bunch of recent paleo-data that's been pushed to attributes) and would come with no ability to record who/when/remarks/etc.

My vote is for (1); I think it's the most usable and it's certainly the most capable.

https://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecpart_attribute_type has grown significantly since this Issue was started; I'm bumping this back to needs discussion.

[Jegelewicz]

My vote is for (1); I think it's the most usable and it's certainly the most capable.

Agree - although I would actually suggest more than six attributes per part...maybe?

[dustymc]

more than six attributes per part

That would just involve rebuilding the parts bulkloader, which doesn't seem too crazy and wouldn't (much) affect any "core" tools. Separate Issue I think - a use case would be helpful, that seems like a lot, analyzing might lead somewhere useful.

[campmlc]

If #1, this must be done in conjunction with improvements in the single
record data entry and catalog record parts UI, otherwise it is going to
cause extreme upset and confusion. Any way to add a "preservation" column
to single record data entry parts table and catalog record parts table, but
the preservation data that get entered or are viewed there are actually
populating/populated by the part attribute loader/part attributes?

On Tue, Aug 4, 2020 at 12:14 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

more than six attributes per part

That would just involve rebuilding the parts bulkloader, which doesn't
seem too crazy and wouldn't (much) affect any "core" tools. Separate Issue
I think - a use case would be helpful, that seems like a lot, analyzing
might lead somewhere useful.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-668748442,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBDKDJXFBHDVNTADVFDR7BFWJANCNFSM4ES5QNQA
.

[dustymc]

improvements in the single record data entry

New Issue. I'll need details.

single record data entry and catalog record parts UI

I'm not sure what this means - those handle very different kinds of data.

Any way to add a "preservation" column
to single record data entry parts table and catalog record parts table, but
the preservation data that get entered or are viewed there are actually
populating/populated by the part attribute loader/part attributes?

Not that I can see - you'd end up with a big pile of things that need to be linked but have nothing capable of linking them, nor preventing them from being manipulated independently.

We could perhaps get rid of parts in the bulkloader altogether and display from the linked table in that grid, but that's https://github.com/ArctosDB/arctos/issues/2178 and not something that I'd want to approach incrementally.

[campmlc]

We need a better UI for adding parts as "extras" than the tiny drop down
box at the bottom of the page. If all parts have to be entered that way in
order to capture preservation, there will be a lot of very unhappy users.
Am I understanding correctly?

On Tue, Aug 4, 2020, 12:32 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

improvements in the single record data entry

New Issue. I'll need details.

single record data entry and catalog record parts UI

I'm not sure what this means - those handle very different kinds of data.

Any way to add a "preservation" column
to single record data entry parts table and catalog record parts table, but
the preservation data that get entered or are viewed there are actually
populating/populated by the part attribute loader/part attributes?

Not that I can see - you'd end up with a big pile of things that need to
be linked but have nothing capable of linking them, nor preventing them
from being manipulated independently.

We could perhaps get rid of parts in the bulkloader altogether and display
from the linked table in that grid, but that's #2178
https://github.com/ArctosDB/arctos/issues/2178 and not something that
I'd want to approach incrementally.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-668756866,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBFGVTNAVNMBYLXJXPLR7BH3LANCNFSM4ES5QNQA
.

[Jegelewicz]

Hopefully that is what we are working toward with the Locality Bulkload Tool. I'm pretty sure Dusty has a global plan....

[dustymc]

global plan

Ish, but there are plenty of usability details to work out - eg stuff that works in a UI may not work for ya'll tabular form and vice-versa. Displaying a component loader in the DE screen is trivial, writing to it is trivial-ish, displaying that in a single table, or avoiding melting that UI (and the software that feeds it) with the 8 terabytes of parts you've bulkloaded separately, maybe not so much.

tiny drop down box at the bottom of the page.

That's just a UI object capable of spawning a javascript function. I can do a full-screen flashing red button if you want....

a lot of very unhappy users.

That's what I'm trying to avoid by throwing this back for more discussion. I need ya'll to tell me what (a) satisfies this Issue, and (b) works for you and your techs in whatever format you intend to use it.

[campmlc]

What would work best is to do one of two things:
1) keep the current user interface for single record data entry as is for
now, except for adding a separate column for preservation next to
disposition and condition. Make that a code table controlled field that
will eventually populate a part attribute. But all the user sees and enters
is a value for preservation in it's own column.
2) In the current user interface, replace the part data entry with a
modified version of the extras table as a popup. But substantial
modifications are necessary. For one, it needs to be at least visible.
Right now, if I click extras, I see nothing until I scroll down and finally
hit the table. It isn't visible initially in my browser view in Firefox. If
I add a part, it takes several clicks to say, yes, I accept it, and then it
disappears from view permanently. No idea what I just did, and no ability
to make any corrections from the same data entry view.

If we switch to pop up boxes for each table in data entry, that may
actually make data entry a lot easier. Allow scrolling back or continue to
next. Prompt entry of one type of data at a time.

On Wed, Aug 5, 2020 at 9:58 AM dustymc [e-mail] wrote:

• [EXTERNAL]*

global plan

Ish, but there are plenty of usability details to work out - eg stuff that
works in a UI may not work for ya'll tabular form and vice-versa.
Displaying a component loader in the DE screen is trivial, writing to it is
trivial-ish, displaying that in a single table, or avoiding melting that UI
(and the software that feeds it) with the 8 terabytes of parts you've
bulkloaded separately, maybe not so much.

tiny drop down box at the bottom of the page.

That's just a UI object capable of spawning a javascript function. I can
do a full-screen flashing red button if you want....

a lot of very unhappy users.

That's what I'm trying to avoid by throwing this back for more discussion.
I need ya'll to tell me what (a) satisfies this Issue, and (b) works for
you and your techs in whatever format you intend to use it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-669276741,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBDMSXREKR6MO3V4ULLR7F6TFANCNFSM4ES5QNQA
.

[dustymc]

Your (1) is my (3) in https://github.com/ArctosDB/arctos/issues/1460#issuecomment-668743745

Your (2) is somewhere between my (1) (if we do some confusing semi-redundant thing with the part-columns in table bulkloader or ???) and https://github.com/ArctosDB/arctos/issues/2178.

switch to pop up boxes for each table in data entry, that may actually make data entry a lot easier.

Been there, done that, twice - once in Versata, once in Access. Users struggle with multiple views.

Might be that we just can't do this without #2178

[campmlc]

I disagree with this: "Users struggle with multiple views."

Every form I fill out online allows this type of prompt to enter one type
of data at a time. If we combined this with an editable view of the entire
table at the end, e.g. what we see now, it would actually help with data
entry. Right now we have visual data overload. I have to mentally train
myself to go table by table, changing what needs to be changed,
leaving what doesn't, verifying all carryover, when my original data may be
in a completely different order than what I see on the screen. It requires
full concentration, and I still make mistakes. Students listening to
podcasts make mistakes. Smaller chunks of info at a time, prompted, with an
opportunity to review and correct, seem to be the way most webpages are
designed, presumably by UI developers.

On Wed, Aug 5, 2020 at 10:22 AM dustymc [e-mail] wrote:

• [EXTERNAL]*

Your (1) is my (3) in #1460 (comment)
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-668743745

Your (2) is somewhere between my (1) (if we do some confusing
semi-redundant thing with the part-columns in table bulkloader or ???) and

2178 https://github.com/ArctosDB/arctos/issues/2178.

switch to pop up boxes for each table in data entry, that may actually
make data entry a lot easier.

Been there, done that, twice - once in Versata, once in Access. Users
struggle with multiple views.

Might be that we just can't do this without #2178
https://github.com/ArctosDB/arctos/issues/2178

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-669292233,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBE7Y5GW67MZJGUQIOLR7GBLRANCNFSM4ES5QNQA
.

[Jegelewicz]

Discussed by Code Table Admins today. Things to accomplish to make this a viable solution.

1. Ability to load ONE part attribute per part in the bulkloader so that part/preservation can be loaded with the entire record. (Anyone wishing to load multiple part attributes per part will need to load parts and/or part attributes separately)

2. Create a new field for parts in the catalog record = "preservation" that will present a comma separated list of all preservation attribute values (only the values) for the part. (See example below)

3. Collapse all part attributes in the catalog record and provide a "Details" option to expand them.

Collapsed

Expanded (what you see after clicking "Details")

1. The "extras" for part preservation should be entered along with the part. This form should just BE how parts are entered:
   

BUT the option to leave all part attributes NULL should exist.

Anything I have left out? @ArctosDB/arctos-code-table-administrators

[dustymc]

ONE part attribute per part i

Elaborate - we've discussed having a highly-abstracted single column labeled "preservation," is this a request for 6 new columns corresponding to every part or ???

This form should just BE how parts are entered:

So why do you need the above??

[Jegelewicz]

We probably need to discuss in person, but here goes.

Elaborate - we've discussed having a highly-abstracted single column labeled "preservation," is this a request for 6 new columns corresponding to every part or ???

6 new columns corresponding to every part (all of the fields required for 1 part preservation attribute)

So why do you need the above??

The above is related only to bulkloading. We recognize that bulkloading records with 12 parts, each with 6 attributes just doesn't seem manageable in a single spreadsheet, but we also want bulkloading to facilitate recording the preservation (which would no longer be part of the part name). I tried to advocate for bulkloading in stages (part-less object record, then parts) but everyone (and I mean EVERYONE) said that was never going to fly. This was the compromise.

[dustymc]

There are currently 284 columns in the table, this would add 72 more. Is nearly 400 columns really usable? Or from the other direction, is it really that different than ~800 columns (=6 full part-attributes with 12 parts, plus the other stuff)?

Perhaps this just isn't practical without something like https://github.com/ArctosDB/arctos/issues/2178? Or maybe we can just keep adding columns?? Looks like PG supports 1600 columns per table....

Maybe we need to sort that out in a dedicated issue - how wide is too wide, if anything is? - and then return here.

[campmlc]

We discussed this in the issues meeting; no one has to use all the columns. This should be implemented as soon as possible to avoid adding a billion new part names with various random combinations of preservation types, in my opinion.

[dustymc]

The above is related only to bulkloading.

I think I finally figured out what you mean - there are new columns in the "core" bulkloader table, but they're not available in the data entry screen - correct?

The data entry screen serves as a UI for the bulkloader; if I'm interpreting this correctly, that's a request for things you can't see or edit (but can orphan) in certain UIs.

no one has to use all the columns

They kind of do; there's a UI that just shows everything (but see above, there may be complications, like where we'd display this in edit mode), and there's a UI that allows hiding some stuff in a view, but if these data exist then you're going to have to deal with them from time to time.

• The data entry screen is a crowded mess; we're constantly asking to put less stuff on it, and also to put more stuff on it.
• More values lead to users scrolling past things; we're constantly asking for fewer things in lists, and also to add more stuff to them. (This is sort of a horizontal list...)

I agree with the urgency behind this, but I don't think we can make a bunch of things significantly less usable in the process either.

Having two functionally simpler data entry screens - one that only enters, one that only edits - might allow columns that most users don't have to see. The entry screen could be customized, and I'd not have to worry about it being used to edit things which aren't visible (which is a fair chunk of the code behind the current screen) because it would not edit at all. The (new) edit screen would always show everything, and would only issue UPDATE commands against existing records. I'm not sure what would happen when a 'save' from the entry screen fails - something more confusing than what happens now, probably.

I'm not sure that I have any remotely realistic solutions, but I am fairly confident that we shouldn't just keep adding on until everything collapses in some spectacular fashion.

[campmlc]

Agree with " we shouldn't just keep adding on until everything collapses in
some spectacular fashion" - including millions of obscure part +
preservation combinations.
Happy to experiment with the separate enter and edit screens if that would
be a potential solution.

On Wed, Sep 30, 2020 at 9:34 AM dustymc [e-mail] wrote:

• [EXTERNAL]*

The above is related only to bulkloading.

I think I finally figured out what you mean - there are new columns in the
"core" bulkloader table, but they're not available in the data entry screen

• correct?

The data entry screen serves as a UI for the bulkloader; if I'm
interpreting this correctly, that's a request for things you can't see or
edit (but can orphan) in certain UIs.

no one has to use all the columns

They kind of do; there's a UI that just shows everything (but see above,
there may be complications, like where we'd display this in edit mode), and
there's a UI that allows hiding some stuff in a view, but if these data
exist then you're going to have to deal with them from time to time.

• The data entry screen is a crowded mess; we're constantly asking to
  put less stuff on it, and also to put more stuff on it.
• More values lead to users scrolling past things; we're constantly
  asking for fewer things in lists, and also to add more stuff to them. (This
  is sort of a horizontal list...)

I agree with the urgency behind this, but I don't think we can make a
bunch of things significantly less usable in the process either.

Having two functionally simpler data entry screens - one that only enters,
one that only edits - might allow columns that most users don't have to
see. The entry screen could be customized, and I'd not have to worry about
it being used to edit things which aren't visible (which is a fair chunk of
the code behind the current screen) because it would not edit at all. The
(new) edit screen would always show everything, and would only issue UPDATE
commands against existing records. I'm not sure what would happen when a
'save' from the entry screen fails - something more confusing than what
happens now, probably.

I'm not sure that I have any remotely realistic solutions, but I am fairly
confident that we shouldn't just keep adding on until everything collapses
in some spectacular fashion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-701469037,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBD3TVZCCMLISFYSMALSINFZ7ANCNFSM4ES5QNQA
.

[dustymc]

AWG:

• add single "preservation" per part

  • Yay: easy, can do it now, can find a way to make it fit on the existing DE screen
  • Boo: people immediately started asking for more (see "middle" below)
  • Meh: contains all the information that's in the parenthetical "preservation" now (eg, "thing (frozen)"), but it seems that's not enough

• add single full attribute per part (6 columns, 12 parts ==>72 new "fields")

  • Yay: useful for much more than just preservation
  • Boo: can't find screen space (but will look at full redesign options)

• middle ground - add determiner and date (or something) along with the formerly-single column

  • yay: no redeeming qualities
  • boo: DLM hates it - it's most of the work and most of the screen space as the "full" solution, but only a tiny bit of the functionality.

Comment from @Nicole-Ridgwell-NMMNHS: their paleo collection is happy to use the parts bulkloader ("add more" or standalone), lack of non-preservation attributes in core entry/loader is OK

Table for reference;

arctosprod@arctos>> \d specimen_part_attribute
                                         Table "public.specimen_part_attribute"
         Column         |            Type             | Collation | Nullable |                  Default                  
------------------------+-----------------------------+-----------+----------+-------------------------------------------
 part_attribute_id      | bigint                      |           | not null | nextval('sq_part_attribute_id'::regclass)
 collection_object_id   | bigint                      |           | not null | 
 attribute_type         | character varying(30)       |           | not null | 
 attribute_value        | character varying(4000)     |           | not null | 
 attribute_units        | character varying(30)       |           |          | 
 determined_date        | timestamp without time zone |           |          | 
 determined_by_agent_id | bigint                      |           |          | 
 attribute_remark       | character varying(4000)     |           |          | 

[dustymc]

The "middle ground" option seems to come from wanting order, which in some cases could affect suitability for use. Defaulting current_date (rather than NULL) in for determined_date at load would accomplish that, even if it's not strictly accurate - a few-months-off value should still order before some future addition. The parts bulkloader is and will remain available in "add more" if more precision is necessary.

@Jegelewicz @campmlc does that address any concerns?

millions of obscure part + preservation combinations.

We have about 1000 parts, and about 300 without parentheses. A few hundred of them may come from insufficient past procedures - see https://github.com/ArctosDB/arctos/issues/3159. The old model, with preservation and modifier in part, had around 10K unique combinations which distilled down to ~100 parts in the current model. I'm not suggesting that we have an ideal model, but we do have about an order of magnitude less entropy than we've dealt with in the past. I do think that the proposed model is a significant improvement, but I don't see it as carrying enough urgency to force us into a less-usable position either. I do not think we should proceed until we have a full solution that looks better, not just "maybe-tolerable."

[campmlc]

Yes, this solution works for me. It is analogous to specimen event
determined date, which can be " anything on or before the present date" and
still be useful.

On Fri, Oct 9, 2020, 12:43 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

The "middle ground" option seems to come from wanting order, which in some
cases could affect suitability for use. Defaulting current_date (rather
than NULL) in for determined_date at load would accomplish that, even if
it's not strictly accurate - a few-months-off value should still order
before some future addition. The parts bulkloader is and will remain
available in "add more" if more precision is necessary.

@Jegelewicz https://github.com/Jegelewicz @campmlc
https://github.com/campmlc does that address any concerns?

millions of obscure part + preservation combinations.

We have about 1000 parts, and about 300 without parentheses. A few hundred
of them may come from insufficient past procedures - see #3159
https://github.com/ArctosDB/arctos/issues/3159. The old model, with
preservation and modifier in part, had around 10K unique combinations which
distilled down to ~100 parts in the current model. I'm not suggesting that
we have an ideal model, but we do have about an order of magnitude less
entropy than we've dealt with in the past. I do think that the proposed
model is a significant improvement, but I don't see it as carrying enough
urgency to force us into a less-usable position either. I do not think we
should proceed until we have a full solution that looks better, not
just "maybe-tolerable."

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-706343782,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBF3JYSYEL2A5N3ZBZLSJ5KTPANCNFSM4ES5QNQA
.

[Jegelewicz]

It works for me. I can set it up in a collection for demonstration - I will soon be migrating in ALMNH:Inv data and this would be a good test case.

[dustymc]

The core of this is running in test.

Docs are https://handbook.arctosdb.org/documentation/bulkloader.html

https://github.com/ArctosDB/internal/issues/65 means automation isn't automating, so let me know (with enough details to find it) if you need something pushed through.

Please let me know if you find a related form that should have preservation but doesn't, along with any other concerns.

[dustymc]

I think entry/bulkoading is fully implemented, and columns reordered for all bulk-like-tools. I'll keep poking at it and plan on pushing to production by Monday evening if I don't find anything major.

Testing very appreciated.

I believe there are two remaining issues in here, which both need dedicated discussion and agreement so we don't end up in another nature_of_id nightmare.

1. Existing parts. This is a fundamental change in the structure of the data, and we need the whole community on board before we go flipping "thing (pres)" to "thing" with an attribute "pres." I do support this; it's normalization and will make the data more accessible for both entry and discovery.

https://github.com/ArctosDB/arctos/issues/1875 is related - we currently have two potential ways of calculating "tissueness," only pulling from http://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecimen_part_name is currently supported.

1. Display. This is a "minor" UI consideration (it doesn't affect data or structure) and could be delayed or allowed to evolve, but it still needs to be on The Community's radar.

It's been proposed that the SpecimenDetail parts grid get a new "preservation" column, in which attributes
* ethanol (details...)
* ethanol (details...)
* formalin (details...)
* bleach (details...)
* LN2 (details...)
would be displayed as "ethanol;formalin;bleach;LN2", which works for me if and when we all agree to adopt the bare part plus preservation model. This would not make sense if the community rejects migrating to "bare" parts plus attributes.

It's also been proposed that attributes are hidden unless a user request expansion, which seems much less tolerable to me. That would hide 7 of our current 8 (certain to grow) part attributes, which have nothing to do with this Issue.

[campmlc]

Can we implement in one collection at a time, eg MSB?

On Thu, Oct 29, 2020, 12:36 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

I think entry/bulkoading is fully implemented, and columns reordered for
all bulk-like-tools. I'll keep poking at it and plan on pushing to
production by Monday evening if I don't find anything major.

Testing very appreciated.

I believe there are two remaining issues in here, which both need
dedicated discussion and agreement so we don't end up in another
nature_of_id nightmare.

1. Existing parts. This is a fundamental change in the structure of
   the data, and we need the whole community on board before we go flipping
   "thing (pres)" to "thing" with an attribute "pres." I do support this; it's
   normalization and will make the data more accessible for both entry and
   discovery.

1875 https://github.com/ArctosDB/arctos/issues/1875 is related - we

currently have two potential ways of calculating "tissueness," only pulling
from
http://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecimen_part_name
is currently supported.

1. Display. This is a "minor" UI consideration (it doesn't affect data
   or structure) and could be delayed or allowed to evolve, but it still needs
   to be on The Community's radar.

It's been proposed that the SpecimenDetail parts grid get a new
"preservation" column, in which attributes

• ethanol (details...)
• ethanol (details...)
• formalin (details...)
• bleach (details...)
• LN2 (details...)
  would be displayed as "ethanol;formalin;bleach;LN2", which works for me if
  and when we all agree to adopt the bare part plus preservation model.
  This would not make sense if the community rejects migrating to "bare"
  parts plus attributes.

It's also been proposed that attributes are hidden unless a user request
expansion, which seems much less tolerable to me. That would hide 7 of our
current 8 (certain to grow) part attributes, which have nothing to do with
this Issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-718944132,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBHLT3ADLTXOFPVR773SNGY2HANCNFSM4ES5QNQA
.

[Jegelewicz]

I plan to make use of this as I add parts to all of the ALMNH:Inv records that I just uploaded (YAY!). It won't be a test of the bulkloader since the records are already in. If I can find some time this weekend, I'll try to create a bulkload file for test to see how it performs - but I'm not making any promises....

[dustymc]

You could have been using the structure for the last several years.

You (and everyone else) will have the option to enter 'preservation' staring ~Monday, which is more "shortcut" than "new."

"IF collection THEN..." in the UI is impractical, but there's an API.

"IF collection THEN..." in the background (eg, for calculating "tissueness") is slightly less impractical, but would add significant complexity and perhaps significant demands of the CPUs - I think it's impractical enough.

"IF collection THEN..." in things like picking parts is not practical at all and there's no reasonable workaround.

Not sure if that all adds up to yes or no without knowing what precisely you mean by "implement."

[campmlc]

Can you convert legacy for MSB collections in test?

On Thu, Oct 29, 2020, 1:07 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

You could have been using the structure for the last several years.

You (and everyone else) will have the option to enter 'preservation'
staring ~Monday, which is more "shortcut" than "new."

"IF collection THEN..." in the UI is impractical, but there's an API.

"IF collection THEN..." in the background (eg, for calculating
"tissueness") is slightly less impractical, but would add significant
complexity and perhaps significant demands of the CPUs - I think it's
impractical enough.

"IF collection THEN..." in things like picking parts is not practical at
all and there's no reasonable workaround.

Not sure if that all adds up to yes or no without knowing what precisely
you mean by "implement."

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-718961083,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBE5422GC7LWCXHZCETSNG4PHANCNFSM4ES5QNQA
.

[dustymc]

We could, but sounds like a lot of work and I'm not sure what you'd get out of it. Elaborate please.

[campmlc]

So I can show our CMs what it will look and function like and get buy in
before we suddenly make the switch. I can do that for a pretend record, but
that won't catch all potential legacy problems.

On Thu, Oct 29, 2020, 1:48 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

We could, but sounds like a lot of work and I'm not sure what you'd get
out of it. Elaborate please.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-718981951,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBEBHBPNPFP4IZ5RIPLSNHBJTANCNFSM4ES5QNQA
.

[dustymc]

I don't see how going through the migration process would particularly help in understanding the functionality. That'll essentially be tossing the stuff in parentheses from http://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecimen_part_name into http://arctos.database.museum/info/ctDocumentation.cfm?table=ctpart_preservation, but as you say I'd expect lots of time-consuming details.

https://github.com/ArctosDB/internal/issues/65 would change that picture; if test was fresh we could expect to use the migration details in production.

There are ~6K public records in production with a preservation attribute, I think that should be sufficient to demonstrate anything that requires scale.

[campmlc]

I feel like I'm not communicating well. We need to show out CMs how their
existing data will be managed and look once we split part and preservation.
In order to get agreement on #1 above, we also are going to need UI
modifications or no one will support this. If the UI is minor from a
database standpoint, but major from a usability and buy in standpoint, it
would be good to play with a version first in test so everyone agrees
before we mess with their data

On Thu, Oct 29, 2020, 2:30 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

I don't see how going through the migration process would particularly
help in understanding the functionality. That'll essentially be tossing the
stuff in parentheses from
http://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecimen_part_name
into
http://arctos.database.museum/info/ctDocumentation.cfm?table=ctpart_preservation,
but as you say I'd expect lots of time-consuming details.

ArctosDB/internal#65 https://github.com/ArctosDB/internal/issues/65
would change that picture; if test was fresh we could expect to use the
migration details in production.

There are ~6K public records in production with a preservation attribute,
I think that should be sufficient to demonstrate anything that requires
scale.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-719003642,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBAP5TBHHQ2E3FJ5FZLSNHGHHANCNFSM4ES5QNQA
.

[dustymc]

not communicating well.

That's my line! Maybe we should chat - any meeting that we can take over coming up?

how their existing data will be managed

1. Same as part attributes have always worked, but with fewer part names. I cannot see a path to fewer part names that doesn't involve The Community deciding to get on board.
2. Except, new "preservation" in data entry and bulkloader - that's in test now, will be in production ~Monday.

and look

Like all other part attributes, mostly.

UI modifications

Elaborate please - I think I laid out what I think has been requested, and what I think is fatal about it, above.

play with a version first in test

I can do that, but see above - eg hiding attributes would be EVIL for anyone who uses the "location" attribute, which is 527,542 records at the moment.

[campmlc]

Yes, we should have another community discussion, where we can address the
UI. Current situation is not adequate, and in my opinion we need an
alternative. Others on our last call agreed. I think there is middle ground
between hiding all with an expand button and displaying everything. But
perhaps we just st implement this and wait for the response . . .
personally, I'd like to be ready for it just in case and have some
potential solutions on hand in case people are as upset as I think they may
be.

On Thu, Oct 29, 2020, 2:49 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

not communicating well.

That's my line! Maybe we should chat - any meeting that we can take over
coming up?

how their existing data will be managed

1. Same as part attributes have always worked, but with fewer part
   names. I cannot see a path to fewer part names that doesn't involve The
   Community deciding to get on board.
2. Except, new "preservation" in data entry and bulkloader - that's in
   test now, will be in production ~Monday.

and look

Like all other part attributes, mostly.

UI modifications

Elaborate please - I think I laid out what I think has been requested, and
what I think is fatal about it, above.

play with a version first in test

I can do that, but see above - eg hiding attributes would be EVIL for
anyone who uses the "location" attribute, which is 527,542 records at the
moment.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-719013066,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBDZKFE3OSMFCDYKOZTSNHIOFANCNFSM4ES5QNQA
.

[dustymc]

Current situation is not adequate

I still need elaboration. Precisely what is not adequate, and for what purposes?

[campmlc]

Perhaps MSB is unique in this, but our parts tables are very large. Add in
multiple preservation attributes to each, and the entire table is visually
impossible. We need an option to view without attributes.

On Thu, Oct 29, 2020, 3:47 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

Current situation is not adequate

I still need elaboration. Precisely what is not adequate, and for what
purposes?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-719043476,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBGMBH7SKKWYBD6QHYLSNHPITANCNFSM4ES5QNQA
.

[dustymc]

See also https://github.com/ArctosDB/arctos/issues/3159#issuecomment-719048092 - some of those probably really do need normalized (some probably don't), and some of them are probably MSB. I would think that things like 'postcranial' would be important, and by this trend that should be an attribute.

I was thinking perhaps I could have multiple part-views that users could select from, but now I'm thinking that's going to result in you hiding things, and then later when you've completely forgotten about doing that making bad decisions because you can't see the things you hid (eg by choosing a view that doesn't have that information). I don't like writing evil code, and I think that would qualify.

I keep being forced back to the idea that if we're going to deal in more complex data then we're just going to have to deal with more complex views of those data, and that there's no real way around that.

Is there some complex-but-compact view of parts+attributes that might work for everyone? I think that's what you're looking for, right?

[Jegelewicz]

I keep being forced back to the idea that if we're going to deal in more complex data then we're just going to have to deal with more complex views of those data, and that there's no real way around that. Is there some complex-but-compact view of parts+attributes that might work for everyone? I think that's what you're looking for, right?

I think we are going to need to move away from seeing everything at once. A simple list of parts on the record page with a LARGE expand option/message that you can view part detail on a second screen might make this easier.

[dustymc]

I added some complexity to http://test.arctos.database.museum/guid/MSB:Para:27057. The main thing that jumps out at me is that the attributes are a tiny portion of the screen real estate. It's mostly whitespace (which occasionally suggests that a table is the wrong organization), and big vertical stretches from having the full container tree shoved in there.

For that record, a full-width cell across the bottom of the screen might make the most sense, but might also be awkward if there was more not-parts data and less parts. The attributes would fit in a cell with very little stretching, or they could stay under and part location wouldn't need to wrap (as much).

move away from seeing everything at once.

Yea, one way or the other!

expand option

I hate it. "Everything" (ish) on one screen seems critical. I have ways to cheat and probably shouldn't get a say in that, but the rest of The Community should.

Does my suddenly-maybe-not-so-crazy full-width parts table on the bottom do the same thing? It's a scroll rather than a click for most of us, and anyone with a huge screen might not even need to scroll.

[campmlc]

One of our requests was to have Preservation cell next to part name and
disposition, with the conacenated preservation list. Then have an expand
button to show date, determiner etc.

On Thu, Oct 29, 2020, 4:56 PM dustymc [e-mail] wrote:

• [EXTERNAL]*

I added some complexity to
http://test.arctos.database.museum/guid/MSB:Para:27057. The main thing
that jumps out at me is that the attributes are a tiny portion of the
screen real estate. It's mostly whitespace (which occasionally suggests
that a table is the wrong organization), and big vertical stretches from
having the full container tree shoved in there.

[image: Screen Shot 2020-10-29 at 3 40 19 PM]
https://user-images.githubusercontent.com/5720791/97640051-15e9db80-19fd-11eb-8d22-fbbecbba0d12.png

For that record, a full-width cell across the bottom of the screen might
make the most sense, but might also be awkward if there was more not-parts
data and less parts. The attributes would fit in a cell with very little
stretching, or they could stay under and part location wouldn't need to
wrap (as much).

move away from seeing everything at once.

Yea, one way or the other!

expand option

I hate it. "Everything" (ish) on one screen seems critical. I have ways to
cheat and probably shouldn't get a say in that, but the rest of The
Community should.

Does my suddenly-maybe-not-so-crazy full-width parts table on the bottom
do the same thing? It's a scroll rather than a click for most of us, and
anyone with a huge screen might not even need to scroll.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-719069765,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBFHV4ULCTIIYL6ZOOTSNHXIBANCNFSM4ES5QNQA
.

[dustymc]

Preservation cell next to part

The table gets wider, and giving it 2 columns sounds even better.

expand button to show date, determiner etc.

I think that would be more palatable (maybe not for those collections that track storage location), but I'm not sure how it would save any screen real estate either.

[dustymc]

See http://test.arctos.database.museum/guid/MSB:Para:27057

There's still a simple summary in the usual place - it could be rewritten to have parenthetical attributes or something, perhaps.

The top of the screen looks better in my browser - other records are likely to be more lopsided.

and parts is better, I think - it's not displacing anything, no buttons to click, just scroll - which is something a user would expect.

That's just copypasta - probably don't need the exxpand-o-part thing if we keep it there.

Attributes should fit in a cell on the right of the table, which would wrap more (still ~half of the half-width table) but also get rid of the gappiness. There's some complexity in those data - I can't just copy-paste them around, and won't rewrite that if everyone hates this. If there's potential that could probably be in the next release.

/remind me to check back on monday, before releasing.

[dustymc]

data entry and display changes are in production

[ewommack]

The data entry screen is throwing up an error for part preservation as NULL now (issue #3187 ). Is this what we had planned? I thought NULL would be ok for part preservation?

[dustymc]

fixed thx

[dustymc]

AWG will set timeline

need report of things in bulkloader that won't work after the migration

[dustymc]

Migration scheduled for 2021-07-02

[dustymc]

These bulkloader records will need adjusted post-migration. Suggest loading them first, using "bare" parts + preservation immediately.

select guid_prefix,enteredby, count(*) from bulkloader where 
part_name_1 like '%(%' or
part_name_2 like '%(%' or
part_name_3 like '%(%' or
part_name_4 like '%(%' or
part_name_5 like '%(%' or
part_name_6 like '%(%' or
part_name_7 like '%(%' or
part_name_8 like '%(%' or
part_name_9 like '%(%' or
part_name_10 like '%(%' or
part_name_11 like '%(%' or
part_name_12 like '%(%'
  group by guid_prefix, enteredby order by guid_prefix,enteredby
  ;

 guid_prefix |    enteredby    | count 
-------------+-----------------+-------
 APSU:Herp   | dlm             |     4
 APSU:Herp   | gradyjt         |     9
 DMNS:Inv    | fsinger_dmns    |     2
 MLZ:Fish    | zoescholz       |     1
 MSB:Bird    | cwitt           |     1
 MSB:Fish    | esdearmon       |     1
 MSB:Mamm    | AdrienneR       |     4
 MSB:Mamm    | CamerynO        |    98
 MSB:Mamm    | cjburgin        |     2
 MSB:Mamm    | ssbrady         |     1
 MSB:Para    | sara            |    10
 MSB:Para    | sarahmdv        |   636
 MVZ:Fish    | kcrice          |   220
 UAM:Bird    | jjwithrow       |     2
 UAM:Fish    | argastaldi      |     2
 UAM:Herp    | Chileboy02      |     1
 UAM:Inv     | allooman        |    93
 UAM:Inv     | anhoover        |    56
 UAM:Inv     | argastaldi      |    66
 UAM:Inv     | edlindley       |    61
 UAM:Inv     | gmjohnson7      |     2
 UAM:Inv     | jrnapier        |     1
 UAM:Inv     | MWMartins       |     5
 UAM:Inv     | samerriman      |     1
 UAM:Inv     | tmpoirrier      |    57
 UCM:Bird    | ganderegg       |     9
 UCM:Mamm    | ganderegg       |     2
 UMZM:Bird   | MarissaMMI      |     1
 UMZM:Mamm   | AshleySkartved  |    29
 UMZM:Mamm   | MarissaMMI      |     1
 UNM:ES      | julianchavez449 |     1
 UTEP:Ento   | aerochild       |    14
 UTEP:Ento   | Muriel07        |     3
 UTEP:Inv    | aerochild       |     9
 UWBM:Mamm   | danienics1989   |     2
 UWBM:Mamm   | erikajprice     |   109
 UWBM:Mamm   | trsapp          |     4
 UWYMV:Bird  | apoitras        |     1
 UWYMV:Bird  | ewommack        |    37
 UWYMV:Bird  | jmaley1         |     2
 UWYMV:Bird  | nickpineschi    |     1
 UWYMV:Bird  | pauldougherty   |    18
 UWYMV:Bird  | ripesnipe       |     3
 UWYMV:Bird  | sadaniels       |     2
 UWYMV:Fish  | kirk0418        |   120
 UWYMV:Herp  | apoitras        |    66
 UWYMV:Herp  | ewommack        |     2
 UWYMV:Mamm  | ahodge          |     4
 UWYMV:Mamm  | nickpineschi    |     1
 UWYMV:Mamm  | pauldougherty   |     1
 WNMU:Fish   | rj3nn1ngs       |     6

[ewommack]

I wish I could, but I'm not sure I'll have the time. I'm ok with dealing with them post migration.

[dustymc]

Spreadsheet: https://docs.google.com/spreadsheets/d/1AY1EXQzKUSAg9EvZFjayTXcCBciXd_syCxt0GSX2tww/edit#gid=485619714

The intended migration part is in the 'migration' tab. Columns are:

• PART_NAME: current part name, will be deleted once its no longer used
• PART: new part name
• PRES_WHOLE: ignore, what was between the parens
• PRES1: first new preservation method
• PRES2: second new preservation method
• PART_EXISTS: PART exists in https://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecimen_part_name
• PRES1_EXISTS: PRES1 exists in https://arctos.database.museum/info/ctDocumentation.cfm?table=ctpart_preservation
• PRES2_EXISTS: PRES2 exists in https://arctos.database.museum/info/ctDocumentation.cfm?table=ctpart_preservation

Everything with ...exists=no needs mapped to existing values, and/or things need created so there are existing values.

Preservation strangeness that I'd rather not perpetuate:

• 100% ethanol - pretty sure that's not possible.

The format of "complicated" parts isn't consistent - is there some way we can not perpetuate this (and update https://handbook.arctosdb.org/how_to/How-To-Manage-Code-Table-Requests.html#specific-values-considerations)?

We have (at least) two ways of saying "a bunch of junk in a tube"

• blood serum, muscle
• stomach plus crop

And a way of saying something else that looks exactly like "a bunch of junk in a tube"

• carcass, dissected

The 'newpart' tab could be used to create parts (from the "part" column), or they can just be created in the normal way. I will need PART and NEW_PART_DEFINITION to create from the spreadsheet.

I'm very open to better ideas, but I think anything that doesn't have an existing part or preservation will be skipped, and then eventually dealt with in some much more painful way.

Code:

drop table temp_partwpres;

create table temp_partwpres as select part_name from specimen_part where part_name like '%(%' group by part_name;
alter table temp_partwpres add column part varchar;
alter table temp_partwpres add column pres_whole varchar;
alter table temp_partwpres add column pres1 varchar;
alter table temp_partwpres add column pres2 varchar;


update temp_partwpres set part=split_part(part_name,'(',1);


update temp_partwpres set part=trim(part);
update temp_partwpres set part=trim(both ',' from part);

select distinct part from temp_partwpres order by part;


select split_part(part_name,'(',2) from temp_partwpres;

update temp_partwpres set pres_whole=split_part(part_name,'(',2);

update temp_partwpres set pres_whole=trim(both ')' from pres_whole);
update temp_partwpres set pres_whole=trim(both '(' from pres_whole);
update temp_partwpres set pres_whole=trim(both ',' from pres_whole);
update temp_partwpres set pres_whole=trim(both ' ' from pres_whole);
-- run above few times


select distinct pres_whole from temp_partwpres order by pres_whole;


update temp_partwpres set pres1=split_part(pres_whole,',',1);
update temp_partwpres set pres2=split_part(pres_whole,',',2);

update temp_partwpres set pres1=trim(both ' ' from pres1);
update temp_partwpres set pres2=trim(both ' ' from pres2);

select distinct pres1 from temp_partwpres order by pres1;
select distinct pres2 from temp_partwpres order by pres2;





alter table temp_partwpres add column part_exists varchar;
alter table temp_partwpres add column pres1_exists varchar;
alter table temp_partwpres add column pres2_exists varchar;


---------------- fix some easy stuff, found below --------------------



update temp_partwpres set pres1='ethanol, 95%' where pres1='95% ethanol';
update temp_partwpres set pres2='ethanol, 95%' where pres2='95% ethanol';


update temp_partwpres set pres1='ethanol, 75%' where pres1='75% ethanol';
update temp_partwpres set pres2='ethanol, 75%' where pres2='75% ethanol';

update temp_partwpres set pres1='ethanol, 80%' where pres1='80% ethanol';
update temp_partwpres set pres2='ethanol, 80%' where pres2='80% ethanol';

update temp_partwpres set pres1='isopropanol, 40%' where pres1='40% isopropanol';
update temp_partwpres set pres2='isopropanol, 40%' where pres2='40% isopropanol';

update temp_partwpres set pres1='isopropanol, 50%' where pres1='50% isopropanol';
update temp_partwpres set pres2='isopropanol, 50%' where pres2='50% isopropanol';

update temp_partwpres set pres1='isopropanol, 55%' where pres1='55% isopropanol';
update temp_partwpres set pres2='isopropanol, 55%' where pres2='55% isopropanol';

update temp_partwpres set pres1='formalin, 10%' where pres1='10% formalin';
update temp_partwpres set pres2='formalin, 10%' where pres2='10% formalin';



update temp_partwpres set pres1='guanidinium thiocyanate' where pres1='Guanidinium Thiocyanate buffer';
update temp_partwpres set pres2='guanidinium thiocyanate' where pres2='Guanidinium Thiocyanate buffer';


update temp_partwpres set pres1='frozen, flash' where pres1='flash-frozen';
update temp_partwpres set pres2='frozen, flash' where pres2='flash-frozen';


update temp_partwpres set pres1='slide, Cobb metal' where pres1='Cobb metal slide';
update temp_partwpres set pres2='slide, Cobb metal' where pres2='Cobb metal slide';


update temp_partwpres set pres1='DMSO/EDTA' where pres1='DMSO/EDTA buffer';
update temp_partwpres set pres2='DMSO/EDTA' where pres2='DMSO/EDTA buffer';


update temp_partwpres set pres1='potassium dichromate' where pres1='K2Cr2O7';
update temp_partwpres set pres2='potassium dichromate' where pres2='K2Cr2O7';


update temp_partwpres set pres1='injection, latex vascular' where pres1='latex vascular injection';
update temp_partwpres set pres2='injection, latex vascular' where pres2='latex vascular injection';


update temp_partwpres set pres1='Longmire''s lysis buffer' where pres1='Longmire''s buffer';
update temp_partwpres set pres2='Longmire''s lysis buffer' where pres2='Longmire''s buffer';


update temp_partwpres set pres1='ethanol, 70%' where pres1='70% ethanol';
update temp_partwpres set pres2='ethanol, 70%' where pres2='70% ethanol';


update temp_partwpres set pres1='xxxx' where pres1='xxxx';
update temp_partwpres set pres2='xxxx' where pres2='xxxx';


------------ END fix ----------------


update temp_partwpres set part_exists='yes';
update temp_partwpres set part_exists='no' where part not in (select part_name from ctspecimen_part_name);


update temp_partwpres set pres1_exists='yes';
update temp_partwpres set pres1_exists='no' where pres1 not in (select part_preservation from ctpart_preservation);





update temp_partwpres set pres2_exists='yes';
update temp_partwpres set pres2_exists='no' where pres2 not in (select part_preservation from ctpart_preservation);

update temp_partwpres set pres2_exists='yes' where pres2='';
update temp_partwpres set pres1_exists='yes' where pres1='';





select p from (
  select pres1 p from temp_partwpres where pres1_exists='no' union select pres2 p from temp_partwpres  where pres2_exists='no'
) x group by p order by p;

-- see above for fix

select distinct part from temp_partwpres where part_exists='no' order by part;


drop table temp_partwpres_newpart;

create table temp_partwpres_newpart as select distinct part_name, part from temp_partwpres;

alter table temp_partwpres_newpart add existing_part_definition varchar;
alter table temp_partwpres_newpart add new_part_definition varchar;

update temp_partwpres_newpart set existing_part_definition=(select min(description) from ctspecimen_part_name where ctspecimen_part_name.part_name=temp_partwpres_newpart.part_name );


alter table temp_partwpres_newpart add used_buy_cc varchar;


update temp_partwpres_newpart set used_buy_cc=(select string_agg(collection_cde,',') from (select collection.collection_cde from specimen_part inner join cataloged_item on specimen_part.derived_from_cat_item=cataloged_item.collection_object_id inner join collection on cataloged_item.collection_id=collection.collection_id where specimen_part.part_name=temp_partwpres_newpart.part_name group by collection.collection_cde) x );

[ewommack]

stomach plus crop

That is a bunch of stuff in a tube, but it is also pretty specific for bird work. It tells me that they actually dissected the crop and put the contents in the same container as the stomach.

Is this a problem because there are multiple words in the description of the part?

[dustymc]

It's not really a problem at all, we just have two (at least) ways of doing the same thing (listing multiple parts as one) and that always ends up confusing someone (or usually everyone).

Ideally that would just be two parts (that may happen to share a container), but we don't have to go there now. Not creating new "lists" of various formats is easy, and I don't see any reason we should avoid that.

Likewise carcass, dissected should be split into a part and some sort of attribute, but we don't have to go there right now either.

It tells me ...

Then PLEASE update the documentation! https://github.com/ArctosDB/arctos/issues/1450

[ewommack]

I can add the definition for

stomach plus crop

but I'm not sure on where in #1450 to add it...
I'm put the definition as:
stomach plus crop = material found in the parts of the stomach plus material found within the crop of a bird or invertebrate, may include the organs and lining or just the material removed from them during prep

@Jegelewicz does that sound about right?
References from wikipedia: crop

[dustymc]

where in #1450

I edited the original post - there's a spreadsheet now. BUT - that will be a new part, see https://github.com/ArctosDB/arctos/issues/1460#issuecomment-836956821, you can use the newpart tab in https://docs.google.com/spreadsheets/d/1AY1EXQzKUSAg9EvZFjayTXcCBciXd_syCxt0GSX2tww/edit#gid=2049837600

[ewommack]

Should I just update all of the stomach, crop parts in the spread sheet?

[Jegelewicz]

@Jegelewicz does that sound about right?

Definition and Wikipedia link added to the code table.

[Jegelewicz]

Note the changes I proposed in the Google sheet for the part name. I think we should be consistent and lose the "plus". Also, all but one of the part names includes "content" is this needed in the new part name? @ewommack

[ewommack]

I think we should be consistent and lose the "plus"

Yeah that makes sense to me.

Also, all but one of the part names includes "content" is this needed in the new part name? @ewommack

I think some of the confusion is coming from the part - stomach plus stomach contents. I'd say that means you saved both the organ of the stomach, plus everything in it. That's what I normally do, but that is me (keeps all the loose bits more contained, people can go looking for parasites later, etc). Do we know what collections are using the parts:

stomach plus stomach content (70% ethanol)
stomach plus stomach content (95% ethanol)

They should maybe be checked with. If we've got any I'd be willing to change to just stomach.

[campmlc]

MSB:Mamm uses stomach content in general to refer to stomach + stomach
content, but likely not consistently. I'm sure there are cases where just
the contents or a subsample were saved. Stomach is a separate part name,
but likely also a lot of overlap and confusion. We definitely have stomach
tissue that has been saved without the content, but who knows whether a
student writing in a prep book or doing data entry has understood the
difference.

On Tue, May 18, 2021 at 10:09 PM Elizabeth Wommack @.*>
wrote:

• [EXTERNAL]*

I think we should be consistent and lose the "plus"

Yeah that makes sense to me.

Also, all but one of the part names includes "content" is this needed in
the new part name? @ewommack https://github.com/ewommack

I think some of the confusion is coming from the part - stomach plus
stomach contents. I'd say that means you saved both the organ of the
stomach, plus everything in it. That's what I normally do, but that is me
(keeps all the loose bits more contained, people can go looking for
parasites later, etc). Do we know what collections are using the parts:

stomach plus stomach content (70% ethanol)
stomach plus stomach content (95% ethanol)

They should maybe be checked with. If we've got any I'd be willing to
change to just stomach.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-843730595,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBB6MQAQXU3WNBVK3QDTOM2YBANCNFSM4ES5QNQA
.

[Jegelewicz]

@Nicole-Ridgwell-NMMNHS @ebraker and I worked on the migration sheet today for a whole hour! We added ALL parts in a tab called part definitions and started adding definitions for those that need it. We also added a tab for preservations - all preservation attributes needed for the migration are already in the code table with definitions except for the following for which we recommend:

Preservation Type from Old Parts | Current preservation attribute | Tissue influence | Definition | Note
-- | -- | -- | -- | --
100% ethanol |   |   |   | Change to ethanol and put this in the remark
5% buffered formalin |   |   |   | Change to formalin, 5% buffered
70% ethanol/glycerin |   |   |   | Use both preservations "ethanol, 70%" and "glycerin" and put this in both attribute remarks
cleared and stained |   |   |   | Use both preservations "cleared" and "stained"
ethanol/glycerin |   |   |   | Use both preservations "ethanol" and "glycerin"

@dustymc would it help if we add the part names without any associated preservation? For example, there are a lot of "body, ..." but no "body" (we can debate the usefulness of that part name later...). Would adding "body" before the migration help you out?

[dustymc]

add the part names without any associated preservation?

YES!! Everything in the 'newpart' tab is going to cause some problem - I'm torn between ignoring them (and so leaving a legacy of extra part names behind) or creating them without useful definitions (and so digging our 'undefined values lead to more undefined values' hole deeper). I just don't see any great options other than someone creating the things.

The is_tissue flag will not survive this process, no need to worry about it for this.

I'm happy to load parts, update definitions, whatever, from CSV at some point if that makes this easier for you.

debate the usefulness

Yea, that one's been on my list for a while (we have extrabody parts?!?), hopefully having fewer parts will be a good opportunity. Tossing a "consider for deletion" in the definition might be useful.

both attribute remarks

I added a couple column to the migration tab, and hopefully-maybe got the right thing in them.

[ewommack]

debate the usefulness

Yea, that one's been on my list for a while (we have extrabody parts?!?), hopefully having fewer parts will be a good opportunity. Tossing a "consider for deletion" in the definition might be useful.

Oh good, that one confuses me all to pieces. Do I have a body or do I not? How many pieces can be removed before it is not a body?
I'll love to get that one cleared up later.

[Jegelewicz]

@dustymc need you to add definition for reproductive tract:

The biological system made up of all the anatomical organs involved in sexual reproduction. Wikipedia

[dustymc]

done

[Jegelewicz]

Thanks. I am cleaning as much as possible, but there are still going to be a lot of "preservation" that isn't in parens and I really feel we need to do something about all the "swab" and "cestode" and other things that don't seem correct...I guess we can get at those farther down the road.

[dustymc]

"swab" and "cestode"

and age class and ....

Yes I'm optimistic that this will take us from....

arctosprod@arctos>> select count(distinct(part_name)) from ctspecimen_part_name;
 count 
-------
   989
(1 row)

...a thousand parts to...

arctosprod@arctos>> select count(distinct(part_name)) from ctspecimen_part_name where part_name not like '%(%';
 count 
-------
   407
(1 row)

... a slightly more manageable number, that we might have some realistic chance of cleaning.

Not quite as nice as it would be if we'd get rid of the combo-parts though.

arctosprod@arctos>> select count(distinct(part_name)) from ctspecimen_part_name where part_name not like '%(%' and part_name not like '%,%';
 count 
-------
   338
(1 row)

[Jegelewicz]

Not quite as nice as it would be if we'd get rid of the combo-parts though.

We definitely need a better solution for those - but we'll get to that later too!

[Jegelewicz]

with regards to preparation - https://github.com/tdwg/dwc/issues/346

[dustymc]

Another migration issue: from https://github.com/ArctosDB/arctos/issues/3633#issuecomment-853457852, there are parts with embedded preservation having preservation attributes added. These will end up with multiple preservation attributes if they're not fixed before; I can't see any viable alternative to that approach. This isn't "wrong" or any kind of problem, but it may not be what's intended.

create table temp_dpp as select guid,part_name,attribute_type,attribute_value from flat inner join specimen_part on flat.collection_object_id=specimen_part.derived_from_cat_item inner join specimen_part_attribute on specimen_part.collection_object_id=specimen_part_attribute.collection_object_id where part_name like '%(%' and attribute_type='preservation';

temp_dpp.csv.zip

Summary:

arctosprod@arctos>> select guid_prefix,count(*) from flat inner join specimen_part on flat.collection_object_id=specimen_part.derived_from_cat_item inner join specimen_part_attribute on specimen_part.collection_object_id=specimen_part_attribute.collection_object_id where part_name like '%(%' and attribute_type='preservation' group by guid_prefix order by guid_prefix;
 guid_prefix  | count 
--------------+-------
 APSU:Herp    |     4
 CHAS:Inv     |     1
 CHAS:Teach   |     2
 DMNS:Mamm    |     1
 DMNS:Para    |   139
 KNWR:Ento    |     1
 KNWRObs:Fish |     1
 KWP:Ento     |     8
 MSB:Bird     |     5
 MSB:Fish     |     5
 MSB:Herp     |   645
 MSB:Host     |    11
 MSB:Mamm     |   266
 MSB:Para     |    16
 MVZ:Bird     |   346
 MVZ:Mamm     |    36
 NMMNH:Herp   |     1
 NMU:Para     |     1
 UAM:Ento     |  3458
 UAM:Fish     |   107
 UAM:Herp     |     1
 UAM:Mamm     |     9
 UAMObs:Ento  |     1
 UMNH:Mamm    |   217
 UMZM:Bird    |    74
 UMZM:Mamm    |    97
 UNR:Mamm     |     1
 UWBM:Mamm    |     4
 UWYMV:Bird   |     6
 UWYMV:Mamm   |     3
(30 rows)

[Jegelewicz]

I am currently under water with iDigBio presentation next week and TPT taxonomy messes. I plan to return to the part definitions after iDigBio.

[campmlc]

I thought that these embedded terms would be removed and autoconverted to
an attribute?

On Thu, Jun 3, 2021 at 9:31 AM Teresa Mayfield-Meyer <
@.*> wrote:

• [EXTERNAL]*

I am currently under water with iDigBio presentation next week and TPT
taxonomy messes. I plan to return to the part definitions after iDigBio.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-853959806,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBAOVXYL6YVRVDEZO4LTQ6N5RANCNFSM4ES5QNQA
.

[ewommack]

Thanks @dustymc! I'll take a look at the UWYMV specimens. I think some of them may have been me just being confused about how to use preservation with our old parts, but I'll double check.

[ccicero]

@dustymc Please remove the values in 'preservation' for

MVZ:Bird | 346
MVZ:Mamm | 36

So we don't end up with duplicate values when preservation is parsed from part name.

Thanks!

[ewommack]

UWYMV specimens are cleaned up

[dustymc]

@ccicero there are inconsistent data eg

 MVZ:Bird:192496 | whole organism (ethanol)                     | preservation   | ethanol, 70%

not sure which is "real" but maybe we should update the part instead of removing the attribute?

[ccicero]

@dustymc - All of our MVZ:Bird 'whole organism' parts have used 'ethanol' (legacy), even though they're probably mostly formalin-fixed and 70% ethanol. We just didn't have a way to specify that detail in 'the old days.' For these records, let's keep them as 'ethanol' even though the part preservation is different, so that they're consistent with everything up to the change. Then we can change moving forward, but...

How do we handle different preservations in bulkloader/data entry - e.g., formalin-fixed + ethanol, 70% ? Can we do those at the same time, or do we need to load the first one, then batch upload the second one (two-step process)? The latter will be a pain, if that's the route we're going, and I'm guessing people won't bother to do that.

[campmlc]

How do we handle new records going in as "liver (frozen)" with a
preservation attribute of 95% ethanol? Will the parsing just add a new
attribute of frozen, and keep the 95% ethanol? Because we'd need both
attributes retained.

On Mon, Jun 7, 2021 at 2:35 PM Carla Cicero @.*>
wrote:

• [EXTERNAL]*

@dustymc https://github.com/dustymc - All of our MVZ:Bird 'whole
organism' parts have used 'ethanol' (legacy), even though they're probably
mostly formalin-fixed and 70% ethanol. We just didn't have a way to specify
that detail in 'the old days.' For these records, let's keep them as
'ethanol' even though the part preservation is different, so that they're
consistent with everything up to the change. Then we can change moving
forward, but...

How do we handle different preservations in bulkloader/data entry - e.g.,
formalin-fixed + ethanol, 70% ? Can we do those at the same time, or do we
need to load the first one, then batch upload the second one (two-step
process)? The latter will be a pain, if that's the route we're going, and
I'm guessing people won't bother to do that.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-856239382,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBCRPLRK3G547ER25KLTRUUSDANCNFSM4ES5QNQA
.

[ccicero]

Yes, we have the same issue - preservation1 = RNAlater, preservation2 = frozen for the same sample (ie. RNAlater vial went into LN2).

And maybe we also need more refined attributes for frozen, to distinguish -20 vs ultralow vs LN2?

[campmlc]

This would be useful: distinguish -20 vs ultralow vs LN2

On Mon, Jun 7, 2021 at 3:28 PM Carla Cicero @.*>
wrote:

• [EXTERNAL]*

Yes, we have the same issue - preservation1 = RNAlater, preservation2 =
frozen for the same sample (ie. RNAlater vial went into LN2).

And maybe we also need more refined attributes for frozen, to distinguish
-20 vs ultralow vs LN2?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-856275850,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBHXNXBW3AC5ZEBSZR3TRU2XXANCNFSM4ES5QNQA
.

[campmlc]

We record -20 vs ultralow vs LN2 on our datasheets, but this doesn't go
into Arctos.

On Mon, Jun 7, 2021 at 3:42 PM Mariel Campbell @.*>
wrote:

This would be useful: distinguish -20 vs ultralow vs LN2

On Mon, Jun 7, 2021 at 3:28 PM Carla Cicero @.*>
wrote:

• [EXTERNAL]*

Yes, we have the same issue - preservation1 = RNAlater, preservation2 =
frozen for the same sample (ie. RNAlater vial went into LN2).

And maybe we also need more refined attributes for frozen, to distinguish
-20 vs ultralow vs LN2?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-856275850,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBHXNXBW3AC5ZEBSZR3TRU2XXANCNFSM4ES5QNQA
.

[dustymc]

formalin-fixed + ethanol, 70% ?

Extras, or separately. Lots of discussion on that above, the final decision should be in some AWG/Issues notes (that I of course can't find right now...).

new records going in as "liver (frozen)"

That will not be an acceptable part.

distinguish -20 vs ultralow vs LN2

Just data - file a CT request issue.

[campmlc]

But what about records currently as

"liver (frozen)"

that also have a 95% ethanol preservation?

On Mon, Jun 7, 2021 at 4:43 PM dustymc @.*> wrote:

• [EXTERNAL]*

formalin-fixed + ethanol, 70% ?

Extras, or separately. Lots of discussion on that above, the final
decision should be in some AWG/Issues notes (that I of course can't find
right now...).

new records going in as "liver (frozen)"

That will not be an acceptable part.

distinguish -20 vs ultralow vs LN2

Just data - file a CT request issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-856309850,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBCRJYDSVTX3Y2NYFVDTRVDS3ANCNFSM4ES5QNQA
.

[Jegelewicz]

"liver (frozen)"

that also have a 95% ethanol preservation?

They will have two attributes: frozen and ethanol, 95%

[campmlc]

That works. Just making sure I don't have to do something extra with these.

On Mon, Jun 7, 2021 at 5:23 PM Teresa Mayfield-Meyer <
@.*> wrote:

• [EXTERNAL]*

"liver (frozen)"

that also have a 95% ethanol preservation?

They will have two attributes: frozen and ethanol, 95%

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-856324677,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBBST7QX6WC4LDC5O5TTRVIF7ANCNFSM4ES5QNQA
.

[ccicero]

That's fine for now, but once things are separate, it's a pain to have to do two preservation attributes in separate steps...people will complain and not do it. Chris is asking about this now for formalin-fixed, 70% ethanol.

Anyone recall what the final decision was on that? I can't find anything in this year's issue notes or above by searching on either 'preservation' or 'entry'

Code Table issue on 'frozen' added: https://github.com/ArctosDB/arctos/issues/3649

[cjconroy]

In the case of our mammals that are formalin fixed, 70% ethanol, the meaning is that there is a preservation/fixation step followed by storage. Calling both of them "preservation" might not give the full picture. Will you need to have two preservations for all fluid specimens to indicate the fixation followed by the storage? If they are not clearly this, then that, will it be clear what steps were taken? Like, if someone carries 95% ethanol in the field, but later puts specimens in 70%, how will you know that it wasn't the other way around?

[dustymc]

@Jegelewicz @campmlc can you find/remember whatever lead to the one preservation per part approach in data entry? I think it was some issues meeting, but?? Somewhere between https://github.com/ArctosDB/arctos/issues/1460#issuecomment-701469037 and https://github.com/ArctosDB/arctos/issues/1460#issuecomment-706343782, but I think not actually in this issue. (And GitHub's "show a couple more" fold-thing is EVIL.)

One more reason why we should strive to never allow Issues to drag on for nearly this long.

@cjconroy trying to make that distinction was a huge problem in the original model - we ended up with about everything in both "columns" all used inconsistently, which ultimately resulted in around 10,000 ways of saying 100 things if I'm remembering the numbers correctly. I don't think recording WHY you dunked something in formalin can be a usable approach, even if "preservation" isn't quite the right word to describe some values or uses or whatever it is that might cause a distinction. All part attributes have dates which can be used for ordering, or recording (down to the second in the current implementation, but that wouldn't be difficult to change) how long something was in some particular "preservative."

[Jegelewicz]

once things are separate, it's a pain to have to do two preservation attributes in separate steps...people will complain and not do it.

https://github.com/ArctosDB/arctos/issues/1460#issuecomment-517406899
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-668743745
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-701469037
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-694528156
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-705795421
https://github.com/ArctosDB/arctos/issues/1460#issuecomment-718944132

I believe it requires extras in data entry or bulkloading parts separately from the rest of the record (the method I recommend).

how will you know that it wasn't the other way around?

All attributes have a date - we may need to set arbitrary dates for legacy material

[dustymc]

"Code Table Admins today" - https://github.com/ArctosDB/arctos/issues/1460#issuecomment-694528156 is what I've been looking for. We obviously have at least one more committee than my brain can track....

Thanks!

[cjconroy]

I think it is unlikely that people are strictly recording the date they transferred specimens out of formalin and into ethanol. So, that's probably going to also be the preservation date, and thus also made up like the legacy date.

[campmlc]

Really, we should have fixation (formalin, 95% ethanol) and then preservation/storage. We need more Arctos reps for fluid-preserved collections in this discussion. If we want to do this right, we should follow: http://clade.ansp.org/malacology/people/rosenberg/archiving/method/fixation_preservation.html
However, it was such a battle to get to this point where we even have the option of fixation/preservation as something separate from part name, that we are making significant progress.
Having multiple "preservation" attributes with the same legacy date seems an OK price to pay for making the split away from part name. We didn't have that info anyway, so we aren't losing anything. Now we have the option going forward of capturing these dates and determiners and methods(?) when we have the data.
I absolutely agree with @Jegelewicz on all points here https://github.com/ArctosDB/arctos/issues/1460#issuecomment-694528156
Especially if we can make it so that
The "extras" for part preservation should be entered along with the part. This form should just BE how parts are entered.
Just today I was showing a new collection how to enter a part that is both in 95% ethanol and frozen. It is complex enough to enter data - if we can simplify by reducing many options to one, it is easier to learn and remember. Make the attributes field an optional dropdown for each part, if necessary. Put all parts in through the component loader through extras. Why not?

[ccicero]

@cjconroy My understanding is that preservation will be like attributes where you can specify a date and determiner, but I don't see how that would happen in data entry or the bulkloader. That is a question for @dustymc

[dustymc]

Why not?

https://github.com/ArctosDB/arctos/issues/2178

date

https://github.com/ArctosDB/arctos/issues/1460#issuecomment-706343782

order, which in some cases could affect suitability for use. Defaulting current_date (rather than NULL) in for determined_date at load would accomplish that, even if it's not strictly accurate - a few-months-off value should still order before some future addition.

preservation will be like attributes

Preservation is attributes.

[Jegelewicz]

I've been through the part names and added definitions along with A LOT of commentary on preservation/condition that should be removed from the part name as well. @ArctosDB/arctos-code-table-administrators please review stuff and look for things I may have missed or that seem odd.

I'll work on the preservation stuff next.

[Nicole-Ridgwell-NMMNHS]

I've gone through the ES parts in the code table and came up with these edits, but I don't think I actually have permissions to make the changes.

amber:
Change the definition to: Hard translucent fossilized resin produced by coniferous trees, typically yellowish in color.
This removes the term Tertiary which is both incorrect and defunct.

forelimb:
Remove the warning "Consider using limb with anatomical direction reference attribute of anterior."
Limb with part attribute anterior could mean that just the anterior half of the limb is preserved.

hindlimb:
same as above

shell:
Remove "preservation unknown"

tendon, ossified:
Remove warning "Consider using tendon with ossified as preservation"
Ossified isn't a preservation it is a tissue type

[Jegelewicz]

@Nicole-Ridgwell-NMMNHS thanks! changes made and I gave you code table access.

[Nicole-Ridgwell-NMMNHS]

Thanks!

[ebraker]

Looks great Teresa, thanks for doing this! There are a few times where we have a base definition, but where preservation method is included in the definition for one of the variants. Do you want me to point these out, or these will get taken care of (merged with base definition, with preservation defined separately)?
e.g.,

eye: An organ of the visual system. Eyes provide animals with vision, the ability to receive and process visual detail, as well as enabling several photo response functions that are independent of vision. Eyes detect light and convert it into electro-chemical impulses in neurons. Wikipedia
vs.
eye (formalin): eye fixed/preserved in formalin

egg: The organic vessel containing the zygote in which an embryo develops until it can survive on its own, at which point the animal hatches. Wikipedia
vs.
egg (70% ethanol): whole ovum, stored in a solution of 50% isopropanol [[[this one also doesn't make sense]]]

[dustymc]

@ebraker no parentheses-having part should survive this process.

I updated the original issue with the migration path spreadsheet, so I don't have to go dig it out of github's horrid little collapsy-thing.

[dustymc]

migration path is now clean

[Jegelewicz]

@dustymc when I Google "hindquarters" this is what I get...

https://en.wikipedia.org/wiki/Buttocks#Synonyms

Enjoy

[Jegelewicz]

@dustymc these are the compound parts that you can add.
compound_part_adds.zip

[dustymc]

thx/done