Homebrew-cask: Discussion - Cask Updates

[colindunn]

So I have been promising this for ages but I wasn't contributing for a while due to work and home commitments. Now that I'm back, it's time to write all this up. Here goes...

What I'm doing works fine for one person, but it would be better to try to roll this stuff into a community driven workflow maybe? @vitorgalvao tried the outdated cask issue system in github a while ago which was a great concept so I know a combined effort is hard to achieve.

All of this stuff is done using ruby scripts plugging into brew. Can't say the scripts are amazing but they work. I re-use as much of the Hbc framework as possible so that the scripts survive core updates and don't need updating very often.

Cask Scanning

brew cask doscan
In random cask order, this script checks all of the URLs associated with a cask for 200 response and then uses one of the following methods to see if there has been an update:

• check the appcast for an update
• if it's a github appcast, check if the version in cask is the same as the github latest link
• use a regex check on a specified URL
• check a redirect location on a specified URL
• OR ignore - generally due to language, needs login to find an update or similar
• OR skip - due to :latest

I'm using a local sqlite DB for storing the relevant data, so anything that doesn't pass the checks gets marked for needing a check. I can override a check or use something else if the current one doesn't suit. For example some of the appcasts, which are valid, are painful to use because of frequent updates. I generally start using a regex check instead. The process stops at 50 casks to check which gives me plenty to look at.

Cask Checking

brew cask needscheck
Once I have a list of casks to check, I then run this command. What it does is runs the scan check again (because someone might have already updated the cask, or there was an URL timeout on the previous attempt) and then if it still fails it uses pry to display the cask check data to me, which includes the errors or updates it found as well as relevant cask data and links. A set of available commands is also displayed.

Now this part is still pretty manual. I still need to go off and check the links for issues or check the download links etc, but I then add data about the cask so that it doesn't come up again. I record things like homepage changes, proposed appcasts, regex or link checks if required, whatever I find. The idea is for the cask not to come up again if at all possible unless there is a change. If the appcast is valid but changed and the download hasn't, then I record the new appcast checkpoint.

Most appcasts are pretty good and the URLs don't generally change so I can get the new version easily just by quickly looking. Github latest is even better because it follows the redirect and gives me the new version number (and ignores alphas etc). Regex is also very good at displaying the latest version number. So then I just type fix <version> and it creates a local git repo branch and starts the next phase. Or I use edit <version> to edit the cask before submission in case changes are required. Before the local branch is created though, it checks the download link for 200 to catch obvious issues like wrong version entered or a download link change.

The point is, some of the checks allow me to just type in the new version number without even checking anything on the pages and if something goes wrong it just doesn't submit to the pipeline and then I check more thoroughly. With storing this updated data, I don't get casks back again for check unless something really has updated, but sometimes of course there is a web page update on the site and a check needs updating but not the cask itself.

Cask Downloading

brew cask testcask
I use a CI/CD pipeline locally to download and install casks and the first step is downloading. I download the cask and record the failed sha checksum, updating the cask.rb file with this new checksum and also the appcast checksum if applicable. If everything is good I cache the download and push the changes back into the local repo branch. I do it this way to catch download issues early and also run multiple downloads at the same time if I have things queued.

Cask Testing

brew cask testcask
Same script but it should now pass the download and an audit and style check.
Then this happens:

• run all of the scripts in developer/bin recording the output and also ls the /Applications folder
• install the cask
• uninstall the cask
• run all of the scripts in developer/bin recording the output and also ls the /Applications folder again
• diff those outputs and fail if there is anything different, displaying the diff output

If everything passes here, then blind submit the cask PR using hub. If it fails I need to take a look and update my local repo copy with anything needing changes and it goes through the whole process again.

This one is single threaded so that if anything does go wrong, I destroy the OSX VM and start it again fresh so that the next check is valid. I don't have the resources locally to run more.

---

So now you have the rundown. I'm using gogs and drone.io for my local CI/CD along with vagrant running OSX 10.13.1 inside virtualbox inside `docker. It's all a cludge specific to my system and the resources I have available and my ability to script it to my needs.

More than happy to share my scripts along with the discussion bearing in mind that they are specific to my setup and environment and would need a lot of tweaking to be multi-user. Anyway, I've finally put this stuff down.

[vitorgalvao]

Coincidentally, I have been working on the auto-update-via-appcast script these past few days.

The basic idea is taking shape, and my recent updates to both shotcut and bartender were already automated by it.

It is a system that works with every appcast in HBC, because it doesn’t try to be smart about it. A few methods are in place for common actions, and then each cask require a single line of custom code to extract the exact version as the cask expects it.

Then it uses cask-repair to trigger the update, but I’m thinking of changing that to first get all needed updates and only run them after that.

Sharing your scripts in a repo would be the best way for us to be able to take a look.

[colindunn]

https://github.com/colindunn/cask-updates

then each cask require a single line of custom code to extract the exact version as the cask expects it

Out of interest, what are you doing there? How are you storing it?

[vitorgalvao]

Out of interest, what are you doing there?

Running a retrieve_* method that gets the appcast (from the cask itself) that reads it a certain way depending on the specified format and then parses it for the exact info.

retrieve_appcast('xml').css('item')[-1].at('enclosure').attr('sparkle:shortVersionString') # Bartender
retrieve_appcast('text').split("\n")[1].split(',')[2] # Google Chrome
retrieve_github_release['tag_name'].split('-')[-1] # Processing
retrieve_github_release['assets'][0]['name'].split('-')[-1].split('.')[0].gsub(/(\d{2})(\d{2})(\d{2})/, '\1.\2.\3') # Shotcut

How are you storing it?

At present I’m not. I go straight from getting the appcast version to comparing with the cask version and running cask-repair if they differ.

But I’m thinking about a simple cask_name,cask_version pair, one per line in a text file. Then making a shell script that parses that and runs cask-repair repeatedly is piece of cake.

[colindunn]

So it sounds like your thinking is a background process that is single-threaded and then figuring out each appcast type to add, which you have done for the common ones.

My efforts are less concerned about the known appcast types and trying to cover the more unusual or non-existent appcasts

At some point once you have that one happy, maybe we could incorporate regex or redirect link checking in your tool? Not sure if that is of interest

[vitorgalvao]

I think you’re misunderstanding. As I said before, the method works for all casks, not common ones. I don’t treat each as a type of appcast, because they are all different. But every appcast is of a set of file types: json (generic or github release), xml, html, or text. I load each into the type of parser that’ll understand it, and then drill down to the specifics. That is why two similar appcasts in the example (processing and shotcut are both github releases) have wildly different lines.

maybe we could incorporate regex or redirect link checking in your tool? Not sure if that is of interest

There’s no need. The tool itself is only for getting the latest version. cask-repair has all those checks already, and that is always the end step for updating.

[colindunn]

Not sure I understand.

For example scenebuilder doesn't have an appcast and I've just updated it by polling https://gluonhq.com/download/scene-builder-mac/ and the redirect location there gives me the current download and from there the version to add to the cask.

Are you saying you have that kind of cask covered as well?

[commitay]

@colindunn Thanks for the writeup and posting the scripts, very interesting.

run all of the scripts in developer/bin recording the output and also ls the /Applications folder ...

Your comment in https://github.com/caskroom/homebrew-cask/pull/41162#discussion_r152472082 prompted me to try to add something similar to our CI.

Currently a very rough draft https://github.com/caskroom/homebrew-cask/pull/41111#issuecomment-346278461

[colindunn]

Thanks @commitay will keep an eye on that one

[vitorgalvao]

Are you saying you have that kind of cask covered as well?

One thing at a time. First it’s the casks with appcasts that matter. But as to your question, yes, I also have that covered in a single line.

Net::HTTP.get_response(URI('https://gluonhq.com/download/scene-builder-9-mac/'))['location'].sub(%r{.*-([\d\.]+)\.dmg}, '\1')

That’s a simple example for scenebuilder. And at some point the biggest part of that line (everything before .sub) will be replaced by a call to a method, which is what the aforementioned retrieve_* are.

So what about when one of these makes a change to how they publish that fails the script? Then cask-repair will notice, because it’s equipped to do so (try to submit without looking but fail on errors) and we evaluate. We’ll then either improve the script or decide if it was a one-off change.

The nature of parsing these upstream sources is that they’re not doing it for us, so there may be breaks. Just like HBC itself, it needs to be tailored for each individual cask, but with a common set of tools.

[colindunn]

Thanks @vitorgalvao now this makes a lot more sense to me. I'm going to start moving my own stuff over to a functional system like this and start pulling the version numbers automatically in a similar manner, then I'll have a whole bunch of data that could be massaged to fit.

Would the overall intention be to use something like this to replace the appcast entirely?

I would normally go and check the homepage when updating a cask to check on things like OSX version requirements and update if I see anything. Some casks are obvious because the links change so tests would fail but some wouldn't be picked up unless you checked.

If these are all blind-submit (based on testing that they are valid and install etc) would that kind of update just rely on end-user reporting? Any thoughts in that area?

[vitorgalvao]

Would the overall intention be to use something like this to replace the appcast entirely?

That is a strong possibility, yes.

If these are all blind-submit (based on testing that they are valid and install etc) would that kind of update just rely on end-user reporting? Any thoughts in that area?

Nothing would be merged automatically. The auto-update system would always call cask-repair anyway, i.e. submit a PR to be reviewed.

[core-code]

For example scenebuilder doesn't have an appcast and I've just updated it by polling https://gluonhq.com/download/scene-builder-mac/ and the redirect location there gives me the current download and from there the version to add to the cask.

i've stumbled across this as well. for some apps the only automate-able way to check for the version number seems to be to look at their redirect. sadly this information can't be used in an appcast entry.
however, i've had an idea there. it would be trivial to write a CGI that takes a URL as a parameter, and then outputs the redirect target

so, if you have
www.vendor.com/app.dmg
which redirects e.g. to
www.vendor.com/app_v1.2.3.dmg

you could call my CGI like this
myserver.com/redirectCGI?url=www.vendor.com/app.dmg
and it would give you this string back
www.vendor.com/app_v1.2.3.dmg

so you could now add
appcast 'myserver.com/redirectCGI?url=www.vendor.com/app.dmg'
to the given cask for the app.

i think this would be useful for apps that have no other version info online, and its certainly quite noise free

[core-code]

i've now implemented this, e.g. have a look here:
http://www.corecode.io/cgi-bin/check_urls/check_url_redirect.cgi?url=https://gluonhq.com/download/scene-builder-mac/

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.