Packit-service: 'GET /api/copr-builds' yields nothing in prod

[TomasTomecek]

tt at hops ~ curl -s 'https://prod.packit.dev/api/copr-builds?page=1&per_page=50'
tt at hops ~ curl -s 'https://prod.packit.dev/api/copr-builds'
tt at hops ~   

is there a bug in the API code?

[csomh]

Looking at the logs, it seems that the process (or thread) crashed and then is restarted whenever the I call this endpoint.

The OpenShift monitoring graphs show a peak in CPU whenever this happens.

Are resources enough?

[lachmanfrantisek]

The call is more complicated since it tries to merge all chroots:

https://github.com/packit-service/packit-service/blob/dcc01a0ab55a1a4caa63be9522bc9257ddcbe0bb/packit_service/service/api/copr_builds.py#L56-L89

[csomh]

Maybe we shouldn't load all 22000+ copr builds into memory at once and have something like CoprBuildModel.get(first, last) instead?

[IceWreck]

So instead of doing pagination provided by Flask RestX, which loads everything into memory and then serves info about 10 or 20 builds at a time, we should fetch those 10 or 20 from the DB ?

That seems way more efficient but one (minor) issue with doing that is that we don't know how many builds to fetch from the DB to get a fixed number (say 10 or 20) of builds in the API after merging chroots.

[csomh]

So instead of doing pagination provided by Flask RestX, which loads everything into memory and then serves info about 10 or 20 builds at a time, we should fetch those 10 or 20 from the DB ?

Yes. Afaik, the data loaded is not cached between API calls, so what is happening now, is that the whole data is loaded in memory, but only a very small part of it is used.

That seems way more efficient but one (minor) issue with doing that is that we don't know how many builds to fetch from the DB to get a fixed number (say 10 or 20) of builds in the API after merging chroots.

Could this be achieved with a 'grouped by' in CoprBuildModel? Such a thing seems to me to have a better place in the database layer than in the API.

[IceWreck]

Could this be achieved with a 'grouped by' in CoprBuildModel? Such a thing seems to me to have a better place in the database layer than in the API.

👍 I'll work on this while modifying the API.

[csomh]

@IceWreck finished this and it's merged, but let's wait until this reaches prod and close after that.

[csomh]

This works now. :tada:

[TomasTomecek]

nice! big ♥ to @IceWreck, and @csomh, obviously :D