Uwazi: some instances crash when map receives lat,lon as strings

[daneryl]

we should prepare a migration to repair old instances with lat,lon as strings, as with the new validation this can not happen anymore, or we should allow our map component to accept strings also.

[txau]

Related? https://github.com/huridocs/uwazi/issues/2805

[daneryl]

@txau i do not think is related.

[RafaPolit]

There are several aspects to consider here.

• There is always a chance that the map will received corrupted data, we should make it as resilient as possible and render empty or even render a placeholder detailing the error. This requires UI and discussion.
• If we make it resilient, this process should zero out the value of the malformed properties and allow reassigning those values within the UI. That zeroing out means that the 'corrupted' data will be lost forever, no way to fix based on the wrong value.
• Same goes for the migration, if the value is not able to be parsed as number, just 'deleting' that value is throwing away info that could have been useful to the data owner.
• There is another area that would need attention and that is: what happens if the 'big' map on the library finds these malformed values? It's a completely different scenario as the single-entity-view. Here, you probably WANT to be presented will all available dots, and those that are unrenderable removed. Still... should the user be 'aware' somehow that some values were discarded? If so, how?

@simonfossom probably you are in the better position here to guide us as to what is the correct experience for the different type of users when faced with this situations. Especially then when and where to report that the value is wrong and we failed to parse the data. Or simply ignore all errors and just do nothing, but that will be very confusing to admins.

[txau]

@RafaPolit sounds complicated. Can't we just make sure that via validation the values stored in the database are always floats instead of strings? In the case of the existing data, can we assume most of it is going to be properly parsed during the migration?

[RafaPolit]

And how do we migrate those values that can't be parsed? (like the double decimal places reported here: #2779) Do we just discard those and give no option for the user to fix them trying to reverse engineer their data?

It's an option, I'm OK with that.

[txau]

@RafaPolit can you provide an example of double decimal that can't be parsed?

[RafaPolit]

I linked it on the above comment. How do you parse 3.55676.4333 ? Which number is this?

[txau]

@RafaPolit I understand. I guess we have no option but zeroing out the values. Maybe we can create some entries in the activity log for this migration so users are aware of which ones were zeroed out.

[RafaPolit]

Look into reporting the affected documents that couldn't be parsed into the ActivityLog collection.

[RafaPolit]

We have decided not to tackle the 'resilience' of the map at this point, and assume that data is correctly formed.

• [ ] We need to migrate the geolocation fileds to be numbers instead of strings.
• [ ] We probably want to report the 'unparsable' values and log them in the ActivityLog

[fnocetti]

• We probably want to report the 'unparsable' values and log them in the ActivityLog

When an unparsable value is found, should the migration:

• Leave the value unchanged,
• Zero out the coordinates
• Unset the field

[RafaPolit]

We cannot leave the value unchanged, it will break the UI as it is doing now, which is the whole point of this migration.

We should unset the field.