Arctos: Can locality and event merging run immediately?

[DerekSikes]

Issue Documentation is http://handbook.arctosdb.org/how_to/How-to-Use-Issues-in-Arctos.html

Describe the bug

I entered data for 25 records and all share identical locality data but when they loaded these ended up with 25 identical data locality records. I no longer see the tool allowing me to merge these locality records (I used the menu - manage / related / locality to find them). What's up?

They should all belong to the same locality and collection event since all their data for both are identical.

Also, when I invoke the 'locality' tool from the manage / related drop down menu it brings me to a page that says 'find collecting events' and should bring me to a page that says 'localities' or something.

Note also, that 25 collecting event records were made and there should be only 1.

Cat records:

ID = Campopleginae
record created date = 2021-05-21

Priority

medium

[Jegelewicz]

@DerekSikes when you bulkload, Arctos can't KNOW they are the same. Give it a bit and if they are indeed identical they will get merged. The way I do this is to create the locality and event ahead of time and name them, then you can use the named locality and event in your bulkload file (makes the file a lot smaller) and you won't have to wait for the merging.

[dustymc]

https://handbook.arctosdb.org/documentation/locality.html#maintenance

That was initially developed to run continuously; they'd probably be merged before you knew what had actually happened. Someone (MVZ??) asked for it to be bumped to a 30 day lag. I'd be VERY happy to reduce the lag back to zero; it would save a lot of confusion, be much easier to manage, save some CPU and maps quota, etc., etc.

[DerekSikes]

I'd also like this lag reduced to zero. Maybe anyone who wants a 30 day lag
can set a flag on the records to tell Arctos to wait on just those?

-Derek

On Mon, May 24, 2021 at 6:38 AM dustymc @.*> wrote:

https://handbook.arctosdb.org/documentation/locality.html#maintenance

That was initially developed to run continuously; they'd probably be
merged before you knew what had actually happened. Someone (MVZ??) asked
for it to be bumped to a 30 day lag. I'd be VERY happy to reduce the lag
back to zero; it would save a lot of confusion, be much easier to manage,
save some CPU and maps quota, etc., etc.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847086761,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACFNUMYMV7XB56KJ7I4MWCLTPJQFLANCNFSM45KBIPMQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

[dustymc]

set a flag o

Named Localities (and Events) are never merged; there's a "flag" that does this (and much more) available for use. (And names can be managed in bulk, if they are meant to be temporary.)

[DerekSikes]

The problem is - after loading I discovered I had been given bad lat/longs
for this site. So now I have the correct lat/longs but to fix this problem
I have to either:

wait 30 days to fix one locality record or

edit 25 different locality records?

Neither is acceptable. Can we get community feedback to eliminate that 30
day lag? Esp since there are ways to prevent merging of identical locality
if one really wants to prevent that!

-Derek

On Mon, May 24, 2021 at 8:46 AM dustymc @.*> wrote:

set a flag o

Named Localities (and Events) are never merged; there's a "flag" that does
this (and much more) available for use. (And names can be managed in bulk,
if they are meant to be temporary.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847186623,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACFNUM7ERJ3PJ3646WU5O73TPJ7FDANCNFSM45KBIPMQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

[Jegelewicz]

@DerekSikes you can manually merge these - if I remember correctly, you should merge the localities first. Just search places for your specific locality (make sure you are searching localities and not events), they should all show up in the results, then hit the Check Dups and from there I think you can figure it out...

Once all the events are using the same locality, you can repeat for the events.

[DerekSikes]

I searched for my cat records - manage - related - localities and got all the localities but there is no Check Dups link. see below

[Jegelewicz]

@dustymc see above - what gives?

[dustymc]

Datatables hates everyone...

Will be fixed in next release.

[DerekSikes]

PS the tool from a catalog search -> manage -> related -> locality

generates a list of collecting events, not localities.

Can we get it to generate localities?

-D

On Mon, May 24, 2021 at 9:20 AM Derek Sikes @.*> wrote:

The problem is - after loading I discovered I had been given bad lat/longs
for this site. So now I have the correct lat/longs but to fix this problem
I have to either:

wait 30 days to fix one locality record or

edit 25 different locality records?

Neither is acceptable. Can we get community feedback to eliminate that 30
day lag? Esp since there are ways to prevent merging of identical locality
if one really wants to prevent that!

-Derek

On Mon, May 24, 2021 at 8:46 AM dustymc @.*> wrote:

set a flag o

Named Localities (and Events) are never merged; there's a "flag" that
does this (and much more) available for use. (And names can be managed in
bulk, if they are meant to be temporary.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847186623,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACFNUM7ERJ3PJ3646WU5O73TPJ7FDANCNFSM45KBIPMQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

[dustymc]

list of collecting events

Same bug as before, the fix got this too.

[DerekSikes]

fyi I did try the normal find localities tool as well and that didn't work:

I entered the lat/longs and chose 'exact' for the search. values = 64.42,
-148.13

This found far more than the 25 records I was interested in (so not an
'exact search' but a 'partial string match' search) but I figured the
'check dupes' tool would isolate them among the hundreds found. Nope. No
duplicates were found.

-D

On Tue, May 25, 2021 at 6:47 AM dustymc @.*> wrote:

list of collecting events

Same bug as before, the fix got this too.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847932957,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACFNUM4GF4NAZGE3EB5LMMLTPO2BNANCNFSM45KBIPMQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

[dustymc]

I think the search works:

BUT only legacy data will carry coordinates like that - anything that's gone through postgres will be padded with zeroes.

I found one of your records, went to https://arctos.database.museum/place.cfm?sch=locality, searched for the locality ID,

clicked dups

check all/click

[campmlc]

weighing in here in support of the original request - I had the same issue
(bad coordinates) with several thousand localities in a bulkload for HWML.
In the process of waiting weeks for the records to automerge we've lost the
time window needed to fix the problems, and now who knows when if ever it
will be done. There were too many to merge manually. I strongly support
reducing the 30 day window or allowing a user to request an automerge as
needed.

On Tue, May 25, 2021 at 9:53 AM dustymc @.*> wrote:

• [EXTERNAL]*

I think the search works:

[image: Screen Shot 2021-05-25 at 8 41 18 AM]
https://user-images.githubusercontent.com/5720791/119527393-2238db80-bd35-11eb-8a13-46db457a1e35.png

[image: Screen Shot 2021-05-25 at 8 42 07 AM]
https://user-images.githubusercontent.com/5720791/119527413-2664f900-bd35-11eb-9006-cd0cfc185db7.png

BUT only legacy data will carry coordinates like that - anything that's
gone through postgres will be padded with zeroes.

I found one of your records, went to
https://arctos.database.museum/place.cfm?sch=locality, searched for the
locality ID,

[image: Screen Shot 2021-05-25 at 8 49 30 AM]
https://user-images.githubusercontent.com/5720791/119528632-35987680-bd36-11eb-8dcb-2f05b3283a29.png

[image: Screen Shot 2021-05-25 at 8 49 42 AM]
https://user-images.githubusercontent.com/5720791/119528688-4517bf80-bd36-11eb-80c3-7517de2f1e03.png

clicked dups

[image: Screen Shot 2021-05-25 at 8 49 42 AM]
https://user-images.githubusercontent.com/5720791/119528737-4f39be00-bd36-11eb-874b-29043438c085.png

check all/click

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847989495,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBFGOIACZEIK6DVF3F3TPPBWNANCNFSM45KBIPMQ
.

[dustymc]

or allowing a user to request an automerge as

See https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847186623, there's already a "no merge" mechanism, adding a bunch more complexity to an already over-complex situation does not make sense to me.

reducing

The only value I think really makes sense is zero; I just can't see any reason to let these pile up for any amount of time (yet there was a request, so I'm obviously missing something).

strongly support

I'm certainly in favor and the tech-bits are a simplification, how do we proceed?

[DerekSikes]

why are extra zeroes added to lat/longs? That artificially inflates
precision. Should they not be exactly as they are entered (if entered as
DD).

-D

On Tue, May 25, 2021 at 8:35 AM dustymc @.*> wrote:

or allowing a user to request an automerge as

See #3617 (comment)
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847186623,
there's already a "no merge" mechanism, adding a bunch more complexity to
an already over-complex situation does not make sense to me.

reducing

The only value I think really makes sense is zero; I just can't see any
reason to let these pile up for any amount of time (yet there was a
request, so I'm obviously missing something).

strongly support

I'm certainly in favor and the tech-bits are a simplification, how do we
proceed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-848034859,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACFNUM6VOJ5MUYJNJDKIAT3TPPGW7ANCNFSM45KBIPMQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

[campmlc]

Absolutely agree on the zeros causing precision problems.
For moving forward with eliminating the 30 day automerge delay, I suggest
leaving the issue up for one week for any comment, and then moving forward.

On Tue, May 25, 2021 at 10:53 AM DerekSikes @.*>
wrote:

• [EXTERNAL]*

why are extra zeroes added to lat/longs? That artificially inflates
precision. Should they not be exactly as they are entered (if entered as
DD).

-D

On Tue, May 25, 2021 at 8:35 AM dustymc @.*> wrote:

or allowing a user to request an automerge as

See #3617 (comment)
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-847186623,
there's already a "no merge" mechanism, adding a bunch more complexity to
an already over-complex situation does not make sense to me.

reducing

The only value I think really makes sense is zero; I just can't see any
reason to let these pile up for any amount of time (yet there was a
request, so I'm obviously missing something).

strongly support

I'm certainly in favor and the tech-bits are a simplification, how do we
proceed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-848034859,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ACFNUM6VOJ5MUYJNJDKIAT3TPPGW7ANCNFSM45KBIPMQ

.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-848046301,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBE74OQXREE27B4IX2LTPPIYHANCNFSM45KBIPMQ
.

[dustymc]

artificially inflates precision

No, there's no implicit anything, precision is explicitly given by max error. 1,1 and 1.0000000000000000000000000000000000000000,1.0000000000000000000000000000000000000000 are precisely the same shape (a point), accuracy (unknown, that's in max error), and precision (exact).

Given those two things and nothing else you (and any georeference calculator) would/should create different error values, but that's procedure.

[campmlc]

So can we change how many zeros are displayed on the record? Because users
are going to assume otherwise.

On Tue, May 25, 2021 at 11:02 AM dustymc @.*> wrote:

• [EXTERNAL]*

artificially inflates precision

No, there's no implicit anything, precision is explicitly given by max
error. 1,1 and
1.0000000000000000000000000000000000000000,1.0000000000000000000000000000000000000000
are precisely the same shape (a point), accuracy (unknown, that's in max
error), and precision (exact).

Given those two things and nothing else you (and any georeference
calculator) would/should create different error values, but that's
procedure.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-848052335,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBF2Z5TCVUI6UIPYC4DTPPJ2TANCNFSM45KBIPMQ
.

[DerekSikes]

hopefully our old 'verbatim' now 'as entered' coordinates should make it
clear how many digits of precision were entered. Right?

Checking this record:

https://arctos.database.museum/guid/UAM:Ento:447170

Coordinates: 64.42 / -148.13 history
https://arctos.database.museum/info/localityArchive.cfm?locality_id=11395131
As Entered: 64.42/-148.13

even the 'transformed' coordinates display to the public as entered.

in the locality record they are 64.4200000000, -148.1300000000

tho.

I'm about to edit that locality tho, now that I see it's merged so those
numbers will change.

Thanks!
Derek

On Tue, May 25, 2021 at 10:47 AM Mariel Campbell @.*>
wrote:

So can we change how many zeros are displayed on the record? Because users
are going to assume otherwise.

On Tue, May 25, 2021 at 11:02 AM dustymc @.*> wrote:

• [EXTERNAL]*

artificially inflates precision

No, there's no implicit anything, precision is explicitly given by max
error. 1,1 and

1.0000000000000000000000000000000000000000,1.0000000000000000000000000000000000000000
are precisely the same shape (a point), accuracy (unknown, that's in max
error), and precision (exact).

Given those two things and nothing else you (and any georeference
calculator) would/should create different error values, but that's
procedure.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-848052335,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ADQ7JBF2Z5TCVUI6UIPYC4DTPPJ2TANCNFSM45KBIPMQ

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-848155519,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACFNUM534KGYOOY562LEB7LTPPWEBANCNFSM45KBIPMQ
.

--

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects, Professor of Entomology
University of Alaska Museum (UAM), University of Alaska Fairbanks
1962 Yukon Drive, Fairbanks, AK 99775-6960
@.* phone: [phone] he/him/his
University of Alaska Museum https://www.uaf.edu/museum/collections/ento/

• search 357,704 digitized arthropod records
  http://arctos.database.museum/uam_ento
  +++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us

[Jegelewicz]

AWG says go ahead an merge immediately!

[dustymc]

Immediate==>stuff disappearing as you create it is annoying, so the scripts are now running with a ~one-hour delay, and that's now pretty easy to adjust if it turns out to be somehow annoying.

[campmlc]

Yay!

On Thu, Jun 10, 2021 at 5:09 PM dustymc @.*> wrote:

• [EXTERNAL]*

Immediate==>stuff disappearing as you create it is annoying, so the
scripts are now running with a ~one-hour delay, and that's now pretty easy
to adjust if it turns out to be somehow annoying.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ArctosDB/arctos/issues/3617#issuecomment-859143833,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADQ7JBEKDD6YRG3N4CVYQSTTSFAZTANCNFSM45KBIPMQ
.