Uwazi: EntitiesIndex now does not continue upon error, making reindex impossible

[RafaPolit]

https://github.com/huridocs/uwazi/blob/026038e9577732868d3a0d4d5362652823b91c1f/app/api/search/entitiesIndex.js#L68-L77

With the introduction of this new code in #2931 we solved a problem that failing to reindex a single entity will not report back to client.

The side effect is that now, a full collection reindex will trigger the error and stop the process. This causes library to show only X amount of documents until the reindex process stopped and nothing else.

This is also preventing our deploy procedure to actually deploy even past the first instance.

@daneryl @konzz @fnocetti @mfacar Let's discuss what is the expected behavior for all scenarios and implement that. If we need to fix our deployment process, that is OK, but let's make sure these are good enough reasons.

[daneryl]

@RafaPolit maybe the current approach will force us to not have the mess of reindex errors we had in the future, obviously we need to do something soon to be able to deploy now, but this can be the approach when we fix all the reindex errors we have now ?

[RafaPolit]

Can we really assume a zero-error tolerance in something like this at this point? Also, can we afford to, potentially, block all other deployments because one instance failed? Maybe in this case we can pass a flag like 'stop on error' with false for the global reindex and provide better feedback of what went wrong?

[fnocetti]

IMHO bulk processes should be fault-tolerant and provide decent feedback to the user on how to fix the part of the batch that failed, specially if the non-indexed element wouldn't stop the application working in ideal conditions. So I agree with you @RafaPolit in that we should implement a "do not stop" mechanism and provide better feedback.