Shields: F-Droid badge displays new version too early

[kitsunyan]

Are you experiencing an issue with...

• [x] [shields.io](https://shields.io/#/)
• [ ] My own instance
• [ ] [badge-maker NPM package](https://www.npmjs.com/package/badge-maker)

:beetle: Description

Basically, there are the following steps in F-Droid build process:

1. checkupdates — checks for updates, alters metadata, and commits it to the fdroiddata repo, ~6h delay after release.
2. build — builds packages from metadata, ~24h delay.
3. deploy — publishes new packages when build is complete, ~12h delay.

F-Droid version is fetched from metadata in repository, which means it will display a new version about a day and half earlier than the package is actually published.

I'm using this badge to show the difference between GitHub release and F-Droid release. But it doesn't make any sense because F-Droid release gets "updated" in 6 hours after GitHub release, but the package wasn't actually updated in F-Droid yet and won't be updated near ~24 hours or even more.

:link: Link to the badge

F-Droid release:

:bulb: Possible Solution

I'm not quite sure how to do this best. It's possible to parse the response from the web page, but that's not a good solution. Another solution is parsing the old XML repo index, but it's a really huge XML file to parse.

[PyvesB]

Hello @kitsunyan, thanks for flagging this up.

I'm not quite sure how to do this best. It's possible to parse the response from the web page, but that's not a good solution. Another solution is parsing the old XML repo index, but it's a really huge XML file to parse.

Yes I think you're right by saying that these would not be good solutions as they would significantly slow down badge rendering and add complexity.

@niccokunzmann do you know if any progress was made on the F-Droid API that we requested a couple of years ago? Do you have any other ideas to solve this in a simple and efficient way?

[niccokunzmann]

I could add something: The web page may also not be what the Android apps use, I think I read something about this.
The release process is in my view:
(1) metadata repository
(2) build
(3.1) website build in parallel to (3.2) index creation for apps (3.3) wiki page generation, ...

So, there are several stages.

I think about these steps and will not take them:

• Ask in the F-droid community forum about the build process, the badge and this issue
• look up the documentation for how f-droid builds apps
• add a badge which uses the website (is not the same as is on the phone)
• add a badge which uses the app index

@niccokunzmann do you know if any progress was made on the F-Droid API that we requested a couple of years ago?

I did not track it or nobody implemented it. I do not know now where to look it up.

[kitsunyan]

I asked in F-Droid matrix chat, it seems right now there is no other ways other than website/index parsing. Something should be implemented on their side (either static JSON generator or dynamic request handler).

[PyvesB]

I asked in F-Droid matrix chat, it seems right now there is no other ways other than website/index parsing. Something should be implemented on their side (either static JSON generator or dynamic request handler).

Roger that, thanks for asking on their chat. Could you please open an issue on the F-Droid repository requesting a proper API to be used by third-parties? All Shields.io would require is an endpoint that accepts an app ID and returns the currently _published_ version of the app. 😉

[eighthave]

You could also implement a process that polls for updates to https://f-droid.org/repo/index-v1.jar, then generates the API however you want, and deploys the JSON files to a static GitHub/GitLab Pages site. Downloading and verifying the index is trivial with fdroidserver as a Python lib:

url = 'https://f-droid.org/repo/index-v1.jar?fingerprint=gettherealone'
indexdata, etag = index.download_repo_index(url, etag=etag)

[PyvesB]

Back on GitLab, it was indicated that the index file was only "updated a few times a week". Users here are complaining about delays of a few hours, so would using that file even help us in this case?

Regardless, I really don't think that it should fall on Shields.io's shoulders to build its own API for a service that does not provide one.

Building a bespoke workflow to download, generate and publish JSON files adds significant complexity, maintenance overhead and would probably require us to rewrite the Python parser if we wanted to stick to our core technologies. Where would we host it? Who would look after it? Who would pay for it?

Doing the download and parsing within the main Shields.io servers themselves would be more manageable, but still comes with some code complexity and maintenance overhead. There are performance considerations as well: the JSON index is big, it's over 16MB and 400k lines of JSON and will likely keep on growing as F-Droid gets more apps published. How big will it get in a year's time for example? Even if we did things asynchronously with a polling mechanism to refresh our cached results, I wouldn't be keen on bringing that kind of processing into the main servers.

[calebcartwright]

Regardless, I really don't think that it should fall on Shields.io's shoulders to build its own API for a service that does not provide one.

I completely agree. Our core mission is providing badges, and in the case of dynamic data (vast majority of badges) we're entirely reliant on being able to retrieve the requisite data from the upstream service/tool/platform. If F-Droid can't/won't provide an API, then perhaps folks that work in that space could create their own community project which Shields (and others) could consume.

[kitsunyan]

I'm working on API implementation which will work on GitLab CI, so it won't burden anyone with hosting. Basically, it checks for changes in F-Droid repo every hour and updates JSON files accordingly.

It's a POC, and I'm waiting for next deploy to test it works. I will be nice to see something like that as F-Droid GitLab project. Ideally — with configured webserver, so all requests to (for instance) https://f-droid.org/api/v1/applications would be redirected to applications subdirectory of the repo. It even might be possible to implement 404 page as JSON response.

[PyvesB]

Thanks for picking this up @kitsunyan! That will hopefully provide a good alternative in the absence of any official F-Droid API. 👍🏻

[kitsunyan]

It seems like it works.

It would be nice to have such API officially, but it need to be implemented using their Jekyll plugin. I tried to understand how it works, and it seems, that it will take significantly much more time to do this. I also don't know Ruby (which is probably not the main problem to me).

I think for now it will be good to move to self-hosted CI-driven API implementation. It also would be better if someone else took maintainership of this implementation, since I'm extremely lazy maintainer :>

[eighthave]

if you get into the official f-droid.org website, it will be maintained.

[PyvesB]

Thanks for championing this piece of work @kitsunyan!

[niccokunzmann]

For completeness: Here is the announcement of the API in the Forum: https://forum.f-droid.org/t/f-droid-now-provides-json-api/11016