Powerlevel10k: Distributing gitstatus binaries

[vimpostor]

I was wondering If you have considered another way to distribute the compiled gitstatusd binaries. Right now they are pushed as is as a binary into the repo, which makes the repo bigger every time.
If this continues, powerlevel10k cloners might have to download a very large repo in the future.

There are many alternatives that are just as comfortable as just pushing to the repo. If you want to keep the git workflow you could even create a separate git submodule that holds the binaries, and only update the submodule HEAD in this repo. This way, once the submodule grows too large, you could just create another submodule and update the linked submodule here and thus avoid making this repo larger every time.

Other alternatives are not using native git features, but these might be more cumbersome.

[romkatv]

If this continues, powerlevel10k cloners might have to download a very large repo in the future.

11 out of 12 documented installation methods create a shallow clone of powerlevel10k repository (the only exception is installation through Zim), so while it's true that the size of the full repository grows every time I rebuild gitstatusd binaries, the amount of data powerlevel10k users have to pull and store doesn't grow.

Having said that, the size of the shallow repository is still quite large as it contains gitstatusd binaries for many platforms, even the ones that the user isn't going to need. If I were starting from scratch I would do it differently. Right now I don't know of a way to reduce the size of powerlevel10k installation without breaking existing users. If you do, please share. I can offer an extra installation and update method that would pull and store less data. I'll likely do that some time in the future (not soon though).

[z0rc]

https://rtyley.github.io/bfg-repo-cleaner/ should be able to help with cleaning up old gitstatus blobs from repo. Though I'm not entirely sure how various plugin managers would handle forced pushes to repo after bfg done with cleaning.

[romkatv]

https://rtyley.github.io/bfg-repo-cleaner/ should be able to help with cleaning up old gitstatus blobs from repo.

Why? Whom would it help?

Though I'm not entirely sure how various plugin managers would handle forced pushes to repo after bfg done with cleaning.

Force push would break virtually everyone.

[z0rc]

Why? Whom would it help?

This was a suggestion if you're planning on removing old gitstatus blobs from git history.

[romkatv]

Oh, got it. No, I'm not planning on rewriting Git history. It'll break existing users.

[vimpostor]

Yeah, I agree force pushing is a no-go.
But what about my initial proposal - hiding the binaries in a git submodule that could be swapped out once it is too large. My installation method (zgen) uses a git submodule update --init when you zgen update, so it is backwards compatible. Are there any installation methods, where this would cause a problem?

Also while it is true that you intially begin with a shallow clone, your local repo copy still grows larger every time you update powerlevel10k.

[romkatv]

But what about my initial proposal - hiding the binaries in a git submodule that could be swapped out once it is too large.

I don't understand what you are suggesting.

My installation method (zgen) uses a git submodule update --init when you zgen update, so it is backwards compatible.

Great!

Are there any installation methods, where this would cause a problem?

Yes. For example, this: https://github.com/romkatv/powerlevel10k#manual.

Also while it is true that you intially begin with a shallow clone, your local repo copy still grows larger every time you update powerlevel10k.

You can shrink it. It's up to you to decide how much history to store.

[romkatv]

You can shrink it. It's up to you to decide how much history to store.

Experimentally, running the following commands seems to do the trick. The repository gets shrunk to the same size as if you deleted it and then ran git clone --depth=1 again.

git fetch --depth=1
git reflog expire --expire=now --all
git gc --prune=all

[vimpostor]

But what about my initial proposal - hiding the binaries in a git submodule that could be swapped out once it is too large.

I don't understand what you are suggesting.

I am suggesting you create a separate repo with the binaries inside it and then add it to this repo via git submodule add. Then every time you update the binary repo you just update the HEAD commit hash in this repo and users have to run git submodule update --init every time they update to get the actual new binary within the submodule (this way you can drop the submodule "junk" once it grows too large and just swap out with a new submodule).

I see however how this is still just as much misusing git features as it is with uploading binaries in the real repo.

Perhaps it might really be better to think of a better solution where only your architecture's binary is downloaded.
This issue was just a suggestion btw. Since this issue isn't that much of a problem right now, feel free to close it.

[romkatv]

I am suggesting you create a separate repo with the binaries inside it and then add it to this repo via git submodule add. Then every time you update the binary repo you just update the HEAD commit hash in this repo and users have to run git submodule update --init every time they update to get the actual new binary within the submodule (this way you can drop the submodule "junk" once it grows too large and just swap out with a new submodule).

Do you believe this would reduce the size of git clones and/or bandwidth used when updating powerlevel10k? If so, why? If not, what's the advantage?

I see however how this is still just as much misusing git features as it is with uploading binaries in the real repo.

Removing the "misuse" of anything is not very motivating for me. If you can describe the practical problem you are having, this would be helpful.

For example, perhaps you cannot afford disk space used by the powerlevel10k? If so, please explain how you are installing and updating powerlevel10k, how much space it uses and how much lower it would need to be to become affordable.

Or perhaps you cannot afford the bandwidth for updates? If so, please explain how you are updating powerlevel10k, how much bandwidth it uses and how much lower it would need to be to become affordable.

Knowing what problem you are trying to solve will allow me and you to evaluate different solutions. Once the problem is understood, suggestions would be valuable. Right now they are premature.

As I mentioned earlier, I'll probably introduce an alternative installation method that is more bandwidth and space efficient. This is very low on my priority list though because no one has asked for it. My time is limited, so I work on features I know will be useful to someone. Right now I cannot be sure whether anyone would care if the size of powerlevel10k installation dropped from 30MB to 3MB.

[vimpostor]

Do you believe this would reduce the size of git clones and/or bandwidth used when updating powerlevel10k? If so, why? If not, what's the advantage?

It would not reduce the size if you just have a shallow clone. But it would reduce the size over time once you have gained enough history by pulling again and once the submodule is swapped out.

However, as you mentioned, one can also achieve this by manually removing the git history (although this has the disadvantage that every user has to do it instead of the repo owner).

Removing the "misuse" of anything is not very motivating for me. If you can describe the practical problem you are having, this would be helpful.

You are right, there is no practical problem that I am having, neither storage nor bandwidth. I was just opening a discussion on this topic, as I was worried by always seeing large binary diffs when updating powerlevel10k in the past.

I am closing this issue, as it has come clear to me that there really is no practical problem here and as it has become apparent that this "problem" is borderline bikeshedding.

Thanks for considering anyqay and for developing powerlevel10k!

[romkatv]

Do you believe this would reduce the size of git clones and/or bandwidth used when updating powerlevel10k? If so, why? If not, what's the advantage?

It would not reduce the size if you just have a shallow clone. But it would reduce the size over time once you have gained enough history by pulling again and once the submodule is swapped out.

Hm... I don't think I understand how this could work. Could you post example commands that I (as the owner of powerlevel10k) would run to "swap out" a module, and the commands users would run?

It's fine with me if you say no. I realize it can be quite a bit of work to explain what you mean and given that no one is holding their breath for solution it may not be worth spending your time on it.

[vimpostor]

Right, I don't think it is worth the time following this topic any further as it is solving a non-problem with too much hassle, but I will shortly explain what I meant with swapping out the module:

1. Create a new repo containing binary files: github.com/romkatv/powerlevel10k-data
2. Add it to the romkatv/powerlevel10k repo: git submodule add github.com/romkatv/powerlevel10k-data
3. Fix a bug in gitstatusd and upload new gitstatusd binaries to github.com/romkatv/powerlevel10k-data

4. Pull upstream changes in the powerlevel10k repo: cd /path/to/submodule && git pull && cd /root/of/powerlevel10k && git add -A && git commit -m "Update gitstatusd binaries" && git push

5. Repeat step 3 and 4 maybe a hundred times until the powerlevel10k-data repo is very large (maybe 100MB) and so is the submodule

6. Create a new binary file repo: Create repo github.com/romkatv/powerlevel10k-data-2
7. In powerlevel10k run git submodule deinit -f path/to/submodule and commit it
8. In powerlevel10k run git submodule add github.com/romkatv/powerlevel10k-data-2
9. You are done, you now have removed the large 100MB git submodule and can now push new gitstatus binaries in powerlevel10k-data-2 submodule

As for the users: Nothing would change at least for zgen users. When zgen users update (or initially clone for that matter), submodules are always included in the update (zgen calls git submodule update --init).

But as I said, this was just an initial proposal, it's a bit cumbersome and another solution that would only download the binary for your architecture, would definitely be better.

I think it's fine to just stick with a shallow clone instead of over-engineering this.

[romkatv]

In the instructions you've described, what takes care of deleting powerlevel10k-data-1 in the clone? Does zgen call git clear -df after update or something?

[vimpostor]

In the instructions you've described, what takes care of deleting powerlevel10k-data-1 in the clone? Does zgen call git clear -df after update or something?

You are right, I didn't think of that. The user would still have to clear it manually. However, at least fresh cloners can get the repo (even when they don't shallow clone) without getting the whole backlog of gitstatus binaries.

[romkatv]

However, at least fresh cloners can get the repo (even when they don't shallow clone) without getting the whole backlog of gitstatus binaries.

Right now users have a choice to fetch the full history, no history or something in between. If powerlevel10k switches to rotating submodule, the option to fetch full history will disappear. No new options will be available to users.

[romkatv]

@z0rc @magnus-gross FYI: Powerlevel10k Git repository no longer contains gitstatusd binaries. The size of powerlevel10k directory after git clone --depth=1 is 2MB.

[vimpostor]

Nice, I didn't even know git subtree. It seems to do a better job than git submodules.

[romkatv]

Nice, I didn't even know git subtree. It seems to do a better job than git submodules.

It's a different tool for a different job. FWIW, git subtree wasn't a part of the solution for removing binaries from powerlevel10k repo. The solution, as always, was to write a shit-ton of code.