Virtual-environments: macOS environment versioning

[Malinskiy]

Describe the bug
I've got a concern with the latest update to macos-latest environment:
How do I version which environment I'm using? Effectively I don't see any docs on the new macos-latest. I assume since it's catalina then it would be macos-10.15, but it seems it can only be referenced as macos-latest. I'd like to have an option to use the same environment in the future to have reproducible builds instead of chasing bugs that are related to GitHub updates of the environment.

For example a gentleman reached out to me via action's issues.
The issue is that he's using the macos-latest environment and he mentions that tr command is not available in his version of the macos environment. When I try to do the same I don't see any problem. I suspect that latest is now pointing to the new version. This is prone to random build fails because obviously we can't test all of the env snapshots against all the possible actions on github. So having a way to specify an exact snapshot (git sha of the source version of the snapshot since it's probably automated?) would be awesome.

Virtual environments affected

• [x] macOS 10.15
• [ ] Ubuntu 16.04 LTS
• [ ] Ubuntu 18.04 LTS
• [x] Windows ?

Expected behavior
I expect to have a predictable environment version that gives reproducible builds

Actual behavior
Uncontrollable changes of environment lead to confusion of action users and end-users of actions, inability to debug issues and bad experience overall.

1. Use macos-latest
2. Wait until there is invisible change to the environment
3. Find something broken
4. Another change
5. Try to reproduce. Find another issue
6. Repeat

This does feel like an already solved problem. Docker for example allows latest tag, but it's highly discouraged from actual usage. The only 2 concerns that I see here are:

1. Increased storage usage for all the different environments
2. Harder to have a pre-warmed instances for execution for all the different versions of environments there might be

Thanks,
Anton

[ikesyo]

Huge +1, see also #78 which is bit related to this.

[hugovk]

+1

See also https://github.com/actions/virtual-environments/issues/79.

[traversaro]

There is a similar issue for Windows, windows-2016 was deprecated in https://github.com/actions/virtual-environments/issues/68, but for some reason also windows-2019 disappeared from the docs, and now only windows-latest is available.

[yueno12]

+1

[mhils]

Adding another use case for this: There are quite a lot of tools (take e.g. PyInstaller) that require you to compile with the oldest OS version that needs to be supported.

If GitHub Actions only supports -latest, we can't serve users on older yet still supported macOS versions. We have users complaining about this before we even shipped a release coming from our new CI system (https://github.com/mitmproxy/mitmproxy/issues/3728).

[hugovk]

Hmm, the docs say macos-latest is macOS Catalina 10.15 but it looks like Python 3.5-3.8 are still on macOS 10.13 High Sierra and only PyPy3 is on macOS 10.15 Catalina.

From checking the logs of a recent run:

$ grep -ril "macOS.*10.13" .
./10_macOS-latest Python 3.5.txt
./macOS-latest Python 3.5/9_Test.txt
./macOS-latest Python 3.5/8_Build.txt
./macOS-latest Python 3.5/12_After success.txt
./7_macOS-latest Python 3.8.txt
./8_macOS-latest Python 3.7.txt
./macOS-latest Python 3.7/9_Test.txt
./macOS-latest Python 3.7/8_Build.txt
./macOS-latest Python 3.7/7_Install macOS dependencies.txt
./macOS-latest Python 3.7/12_After success.txt
./macOS-latest Python 3.8/9_Test.txt
./macOS-latest Python 3.8/11_Docs.txt
./macOS-latest Python 3.8/8_Build.txt
./macOS-latest Python 3.8/7_Install macOS dependencies.txt
./macOS-latest Python 3.8/12_After success.txt
./macOS-latest Python 3.6/9_Test.txt
./macOS-latest Python 3.6/8_Build.txt
./macOS-latest Python 3.6/7_Install macOS dependencies.txt
./macOS-latest Python 3.6/12_After success.txt
./9_macOS-latest Python 3.6.txt

$ grep -ril "macOS.*10.14" .

$ grep -ril "macOS.*10.15" .
./6_macOS-latest Python pypy3.txt
./macOS-latest Python pypy3/9_Test.txt
./macOS-latest Python pypy3/8_Build.txt
./macOS-latest Python pypy3/12_After success.txt

Please could someone confirm what is what and make it clearer in the docs and logs?

[thejoebourneidentity]

Hey everyone, thanks for the feedback! I'm one of the Product Managers on the Actions team. I agree this is inconsistent. Part of the inconsistency is in the product, and part of it is in the docs.

In the product, we do not support using macos-10.15 as a label to reference the macOS virtual environment. We're going to get that fixed. We do, however, support the use of the label windows-2019 to reference the windows virtual environment.

In the docs, the only version label for a latest environment we document is ubuntu-18.04. We'll update the docs to make sure it's clear you can use windows-latest or windows-2019, and that you can use macos-latest or macos-10.15.

[github-actions[bot]]

This issue has not had any activity for 45 days and will be closed in 45 days if there continues to be no activity.

[hugovk]

@github-actions Please keep open

[kceb]

Related: for some reason recent macOS build started failing whenever I try to use jq. Created issue here: https://github.com/actions/virtual-environments/issues/526

Would be great to have versioning to rule out github updates from being a source of build failures.

[miketimofeev]

@kceb we've already implemented the versioning! You can find an image version in the $ImageVersion environmental variable.

[Malinskiy]

@miketimofeev that's awesome to hear! now that we have visibility, how do we request a specific environment version for our builds?

[miketimofeev]

@Malinskiy unfortunately, only one version is available at the moment. This variable helps us identifying issues

[kceb]

@miketimofeev I noticed there are multiple versions used by my runner, is it possible I can lock to a version? ex: lock to runner version: 2.165.2?

I believe it is currently 2.168.x, but I noticed this recent version for some reason slows down certain steps (like the setup step) in my build 5x. When macOS build time is so expensive, it's worth optimizing wherever possible.

[alepauly]

@miketimofeev I noticed there are multiple versions used by my runner, is it possible I can lock to a version? ex: lock to runner version: 2.165.2?

I believe it is currently 2.168.x, but I noticed this recent version for some reason slows down certain steps (like the setup step) in my build 5x. When macOS build time is so expensive, it's worth optimizing wherever possible.

Hey @kceb, to be clear the runner version actually refers to the small client that runs in the VM and manages the workflow. The code for the runner is at https://github.com/actions/runner. The virtual-environment version is actually in the ImageVersion variable. That's the one that indicates what version was build from this repo. It's a bit confusing since the hosted runners run the runner but hope this helps 😄. Thanks!

[StephanBijzitter]

I came to this issue because I found that the only two values accepted are "macos-10.15" and "macos-latest", which are both are same thing.

I run tests with Selenium 3 on Safari 11, which in turn requires "macos-10.13" to exist. Running on "macos-10.15" means I'd have to use Selenium 4 with Safari 13, but all my other tests would still have to use Selenium 3.

That's inconvenient, but next to that it also means I'm not able to test any older browsers that I still need to support.

I guess it's a logistics issue, but for me this is extremely important. The only option I currently have is to contact Browserstack and ask them for an open source license, so that I can run tests on Safari 11 with their servers as host and github actions as client. And while Browserstack is awesome, it does introduce a performance penalty by having to send literally everything over the internet.

[maxim-lobanov]

Hello @StephanBijzitter ,
Currently,

• GitHub Actions supports only macos-10.15
• AzDO supports macos-10.14 and macos-10.15

Unfortunately, we don't have plans to add macos-10.13 because it was removed (deprecated) from Azure DevOps recently (when 10.15 was added)

[maxim-lobanov]

Hello everyone, I am going to close this issue.
Currently, the following macOS images are available:

• GitHub Actions supports only macos-10.15 and macos-11.0
• AzDO supports macos-10.14 and macos-10.15

We update every image on weekly basis and for now, there is no way to freeze specific image version and avoid such updates.
Please let us know if you have any additional questions