Sormas-project: Fix error when trying to pull all persons as a user with access to all data

[lebiku]

Bug Description

Seit der Umstellung auf v52.1 erhalten wir vermehrt Timeouts / Server Exceptions beim Abfragen der geänderten Datensätze (cases und contacts). In einigen Fällen funktioniert der Zugriff über die /contacts/all/{since} Schnittstelle. /cases/all/{since} hat seit dem Server Update immer Probleme gemacht.

Als Workaround laden wir temporär alle Cases / Contacts und filtern dann nach dem Datum. Aus Laufzeitgründen benötigen wir aber dringend einen Fix der API. In 1.51.1 funktionieren dieselben Requests noch.

Steps to Reproduce

1. GET Request mit Postman (Username / Passwort)
2. https://be.sormas.ch/sormas-rest/cases/all/1608194578 (zum Zeitpunkt die letzten paar Minuten)
3. Timeout / Server Exception

Expected Behavior

Die API Schnittstellen sollten die geänderten Datensätze zurückgeben. Timeout wie bisher falls zu grosser Payload (Zeitsteempel zu lange in Vergagenheit) ist akzeptabel (Verhalten der Version 1.51.1)

Screenshots

System Details

• Device: Windows 10
• SORMAS version: 1.52.1
• Android version/Browser: Chrome 87.0.4280.88
• Server URL: be.sormas.ch
• User Role: Sormas rest-user

Additional Information

[MateStrysewske]

@lebiku Sind die User in der Lage, die Anwendung abgesehen von der REST-Schnittstelle problemlos zu benutzen, d.h. die Fallliste, Kontaktliste etc. ohne unerwartete Performance-Probleme zu laden?

Wenn es Server Exceptions gibt, können Sie uns Stack Traces zur Verfügung stellen?

[lebiku]

@MateStrysewske Abgesehen von der Rest Schnittstelle sind mir keine Probleme (Performace, oder anderes) bekannt. Es ist nur über die Schnittstelle in ein externes Programm wo wir Probleme haben.

Momentan können wir einmal spät abends die Alle Fälle / Kontakte holen und unser externes Programm mit den Sormas Daten aktualisieren. Der Kunde (Contact Tracing des Kantons Bern) möchte dies aber periodisch auch während der Arbeitszeiten durchführen können. Alle Fälle / Kontakte zu laden und danach zu filtern dauert aber lange und kann daher nicht tagsüber gemacht werden.

Server Logs habe ich leider keinen Zugriff. Der Server ist bei einem Hoster in der Schweiz. Ich habe nur Zugriff auf die Schnittstelle.

[MateStrysewske]

Können Sie mir einen bestimmten Zeitpunkt nennen, zu dem Sie versucht haben, die Daten abzurufen und einen Fehler erhalten haben?

[lebiku]

Ja, heute morgen um 10. Ich wollte zum Testen die Änderungen ca. der letzten halben Stunde. Mit 1.51.1 konnte ich immer bis ca. 2 Tage zurück die Änderungen holen. Das waren dann jeweils so 1000 cases und 2000 contacts. Bei längeren Zeiträumen kriegte ich Timeout Fehler.

[ChristopherRiedel]

@lebiku Hello, thank you very much for your contribution. Could you please write Issues in English next time? Many developers and contributors of SORMAS don't speak German and then can't help/participate. Thank you in advance.

[lebiku]

Sure. Thank you! Von meinem/meiner Galaxy gesendet
-------- Ursprüngliche Nachricht --------Von: Christopher Riedel [e-mail] Datum: 17.12.20 14:34 (GMT+01:00) An: hzi-braunschweig/SORMAS-Project [e-mail] Cc: Markus Wälchli [e-mail], Mention [e-mail] Betreff: Re: [hzi-braunschweig/SORMAS-Project] sormas-rest API Timeouts / Server Exceptions bei Abfrage der geänderten Datensätze (#3821)
@lebiku Hello, thank you very much for your contribution. Could you please write Issues in English next time? Many developers and contributors of SORMAS don't speak German and then can't help/participate. Thank you in advance.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[MateStrysewske]

@lebiku There's two problems that I've identified by analysing the logs that I've received. One is that requests to retrieve all changed cases leads to an OutOfMemoryError because the Java heap space is exceeded. There's nothing we can quickly do about this because it requires further analysis that we won't be able to do in time for the upcoming release today. You might however want to contact your server host and ask them to increase the heap space that is available to SORMAS, or to increase the amount of RAM that the server is allowed to use.

The second problem is associated with trying to retrieve all persons since a specific date via https://be.sormas.ch/sormas-rest/persons/all/{since}. Have you also tried that and run into errors?

[lebiku]

@MateStrysewske Many thanks for quick investigation. I'll contact our server host asap. We identified the isssue with the persons endpoint on Version 1.51.1. Since we don't need to poll the persons we didn't investigate this further. But, yes the issue with persons/all/{since} was there in the previous version too.

[MateStrysewske]

@lebiku Okay, good to hear. If you need it for anything, you can probably access it without errors by using a user that has the "Überwachungsleitung" user role in addition to the "ReST Benutzer".

[MateStrysewske]

For the developer that fixes this issue: The problem is that contact user filter that is built in ContactService and called in PersonService:195 might return null for a user that has no viewing limitations (e.g. national users). The fix is probably to not use cb.and in line 200, but AbstractAdoService.and instead because that method can handle null parameters.

[lebiku]

@MateStrysewske The hoster doubled the RAM. But we still get no response from server with v1.52.1, even with timestamps just going few seconds back in history. Hence, there seems to be a major issue with the change from v 1.51.1 to 1.52.1.

Can you tell us when the bug is going to be addressed? We have already issues with the workaround, which incresase the more cases and contacts we have and grows daily.

[bernardsilenou]

@lebiku This bug would be addressed in the release that is scheduled on January 20, 2020. Hope you can manage till then.

[MateStrysewske]

@bernardsilenou Have you received reports about this from other health authorities as well or is it, for some reason, limited to this specific server?

[bernardsilenou]

@MateStrysewske I got nothing in this line. This may be because many servers have not upgraded to 1.52.1

[MateStrysewske]

@lebiku Judging from your latest comment, this means that you get a timeout even when there have been absolutely no cases added since the timestamp you use, right?

[lebiku]

@MateStrysewske Yes in accordance to my observations, the api timed out even when no cases had been altered. Concerning the contacts I sometimes got responses with data. However I think to remember that I got back too many contacts in that case. I didn't analyse that further though. I can redo the requests and report if it helps.

[MateStrysewske]

@lebiku That would be good, any additional information helps. However I'm a bit confused that you're getting a timeout even when the list that's returned should just be empty. That could hint at a general problem with the server itself. Can you say how many cases you have in the system right now?

[lebiku]

@MateStrysewske It's a bit embarassing but the tests I did with the doubled RAM before christmas was based on TimeStamps in seconds. I now tested with miliseconds (as we do in our client app) and the requests to cases and contacts work. I still need to investigate longer time spans as we have in the client app. Up to few days.

Is there a filter option coming to the api endpoints? This would help to decrease the response size. In 24 hours we have more than a thousand records that change which quickly leads to responses in the size of 2+ MB.

Sorry, for the inconveniences coming from wrong testing.

[lebiku]

@MateStrysewske FYI. I tested the productive system with a request of all changed cases since 1.1.2021. I got a response, taking 2m 19s, having size of 14.7 MB and including 3778 changed cases. Hence, with the increased memory we can work. Thanks again for your time.

[MateStrysewske]

@lebiku That's great to hear, thank you! I don't think there are plans for filtered API endpoints right now, but if that's something you could really use, it might be worth contacting the HZI and discuss it with them.

[bernardsilenou]

@lebiku Should we close this issue or you think further investigation is needed?
If you wish that we investigate further, then we may need to carry it over to our performance team and we would need your contribution and presence in the meeting.

[lebiku]

@bernardsilenou I think you can close the issue. The main problem I see in the compuation time and response size if longer time periods are queried, but that is probably another topic requiring filters in future. Our approach currently works with the implemented solution.

[MateStrysewske]

@bernardsilenou @lebiku There's still a bugfix associated to this issue that I will merge today. I will close the issue afterwards.