Yay: Add -git package Github support

[Jguer]

When you yay -Syu --devel yay should check if the latest commit has a different SHA than the installed version and update accordingly.

Steps:

• Get source URL
  RPC does not expose the source URL. Alternatively we could parse .SRCINFO, it makes me sad but it works.
• if source does not include github.com and .git ignore
• Compare saved SHA in ~/.config/yay/vcs.json with SHA from api.github.com/repos/user/repo/branches (only master branch for now)
• If SHA differs add to update list

[maximbaz]

Just wondering, does it really have to be working only with github and have a .git suffix? Both sound like unnecessary restrictions to me.

If the package name ends with -git, just take the source and run a git pull on it. If that fails - then skip. But it will almost never fail, because most -git packages do have a correct source that git can clone.

---

Instead of parsing .SRCINFO, maybe it's better to parse PKGBUILD file? I know you don't want to handle branches in the beginning, but generally it's a desirable thing. It would be cool if during the first installation I could edit PKGBUILD and append #branch=feature-x to the source and yay uses that branch and saves that branch name in ~/.config/yay/vcs.json, and when upgrading with yay -Syu --devel I no longer need to edit PKGBUILD to define the branch again, yay would just git pull and use that same branch.

I use this workflow (of manually appending #branch=feature-x) quite often for a couple of projects which I follow on github, in order to test some pull request or a feature branch.

Ah, this is the main feature that I'm waiting before finally deleting yaourt 😊

[Jguer]

• yay/aur/vcs will be a generic package where providers can be added.

  • One of those providers can be a vanilla git puller.

• "Both sound like unnecessary restrictions to me." :

  • The github provider doesn't need to pull the package (which I like) using github's api to get the last commit.
  • Doesn't need a .git suffix (necessarily)

if !(strings.Contains(source, "git://") ||
        strings.Contains(source, ".git") ||
        strings.Contains(source, "git+https://")) {
        return
} // probably means there's some sweet sweet git cloning going on. 

• I haven't written the branch parser, that's why it's still only using master, it's laziness + time constraints and PKGBuild auto edit is something that needs thought to ensure it doesn't break perfectly good PKGBuilds. This could be it's own issue as it is an enhancement over this issue.

EDIT & PS: I wrote yay to remove other AUR helpers from my system. Now I have them all installed because I need to test against them 😆

[maximbaz]

The github provider doesn't need to pull the package (which I like) using github's api to get the last commit.

Do you mean to get the latest commit SHA, to determine if there's something new? Based on my experiments, I honestly think you shouldn't spend time on this, the time it takes to do a git pull on the already latest repo is equal to time it takes to use Github API:

❯  time curl -s https://api.github.com/repos/Jguer/yay/git/refs/heads/master
{
  "ref": "refs/heads/master",
  "url": "https://api.github.com/repos/Jguer/yay/git/refs/heads/master",
  "object": {
    "sha": "b72ee99a6172fb1142014375be5d8d972f61e92e",
    "type": "commit",
    "url": "https://api.github.com/repos/Jguer/yay/git/commits/b72ee99a6172fb1142014375be5d8d972f61e92e"
  }
}
curl -s https://api.github.com/repos/Jguer/yay/git/refs/heads/master  0.02s user 0.01s system 5% cpu **0.587 total**

❯  time git pull
Already up-to-date.
git pull  0.02s user 0.01s system 5% cpu **0.598 total**

Tried that several times now, a generic vanilla git pull works just as fast as using Github API.

Or you have something else in mind, something that actually replaces git pull to get the contents of the repo, the snapshot without the history? That also might now be ideal, suppose I ran yay -Syu --devel 10 minutes ago and after that there was a new commit that changed one line. With the git pull, the one-line change would be fetched immediately, while with the snapshot download you'd download everything from scratch again.

I wrote yay to remove other AUR helpers from my system. Now I have them all installed because I need to test against them :laughing:

😂

ADDITION: as for the branch parsing and staying on that branch during upgrades, yes, definitely should be handled separately. Just keep that in mind, that it will come. With parsing .SRCINFO that might be tricky to add branches later, that's why I'm thinking perhaps it is worth starting with parsing PKGBUILD even now, not to rewrite the whole thing when it comes to add branches support.

[rmarquis]

@Jguer If I may, I'd suggest to have a look at the implementation as done in pacaur. It relies on makepkg directly and is thus more generic: it works with all supported VCS packages (gi, svn, hg, ..).

[Jguer]

Do you mean to get the latest commit SHA, to determine if there's something new? Based on my experiments, I honestly think you shouldn't spend time on this, the time it takes to do a git pull on the already latest repo is equal to time it takes to use Github API:

So you would store the aur source on site? And compared to git clone ?

Just went ahead and read it.
https://github.com/rmarquis/pacaur/blob/master/pacaur#L1255

Isn't it a bit wasteful (and slow) to do a full clone on, for example, an icon pack each time to check if it's updated?

Please persuade me, I'm still looking for the best way

[rmarquis]

If you're are reffering to the behavior of makepkg, you'll find everything you need on the pacman-dev ML. This has been debated numerous times over the years.

[Jguer]

I think I will implement it pacaur style for now as it really is the most generic approach (and fastest to implement for now), and doing it only for packages with correct sufixes and I'll keep checking for options. Still need to make a lot of changes to perfect package building itself before worrying about this in greater detail.

[maximbaz]

So you would store the aur source on site? And compared to git clone ?

Sure, the repo will be cloned to /tmp/yaytmp and be stored there for some time. Next time user wants to upgrade -git package, yay will see that it is already cloned and just do a git pull.

• If there were no new changes, this will be very fast operation.
• If there are a couple of new commits, this will also be very fast, git pull is faster than query Github API and then downloading zip archive of the current master.

This way you are not going to do git clone every single time, you do git clone if and only if the repo was not already cloned earlier.

---

Maybe consider using a different folder to store cloned repos. /tmp is (often/always) cleared after reboot, so there is a risk of doing full git clone more often than needed. Maybe cloned repos of -git packages could be stored in $XDG_CACHE_HOME (defaults to $HOME/.cache) - this folder is not cleared after reboot.

[AndydeCleyre]

Please keep in mind some repositories are hefty. A cloned Firefox repo alone can be ~5.4GiB, and the sum of the installed sizes of all my -git packages is currently over 9 GiB.