Uwazi: Changing a select property's thesauri target is not sanitizing the data

[RafaPolit]

If you change the thesauri that a select property is pointing to in templates, those entities that already have values are not being sanitized.

[RafaPolit]

Deleting a property is also not deleting the data, either on the database or on the ES index, not sure.

[daneryl]

@RafaPolit data is being sanitised, but it is sanitised taking into account that each property type has a different value type, this is NOT the case anymore since we merged metadataObjects.

all values are arrays, and this was not reflected in the sanitisation, e.g. selects were being reset to blank string, producing a reindex error because the index expects an array.

PR is ready for review.

[RafaPolit]

So we also need a migration to, yet again, fix malformed metadata for anyone that could have deleted a template property?

[fnocetti]

So we also need a migration to, yet again, fix malformed metadata for anyone that could have deleted a template property?

For sure @daneryl can answer this better than me but, as far as I've seen until now the application is tolerant to these malformed empty fields, and it will just step on them when defining a new value. I guess the main problem was the type incoherence on ES that daneryl fixed (This needs review, I'm talking out of the top of my head).
Maybe instead of another migration, it'd make sense to invest some time on that on-demand data consistency check/fix script?

[daneryl]

So we also need a migration to, yet again, fix malformed metadata for anyone that could have deleted a template property?

For sure @daneryl can answer this better than me but, as far as I've seen until now the application is tolerant to these malformed empty fields, and it will just step on them when defining a new value. I guess the main problem was the type incoherence on ES that daneryl fixed (This needs review, I'm talking out of the top of my head).
Maybe instead of another migration, it'd make sense to invest some time on that on-demand data consistency check/fix script?

yes this needs a migration, good catch @RafaPolit, if anyone changed a select property thesauri then the empied entity values on the db are wrong and reindex will always fail. im going to prepare the migraton on this same PR.

[daneryl]

@RafaPolit @fnocetti i added a migration on the PR that transforms every metadata value that is blank string to empty array, i think this should be enough ?

[fnocetti]

I think it is. I'll get back to the QA!