Virtual-environments: OpenSSL 1.0.2t installation location conflicts with Homebrew on MacOS

[simons-public]

Describe the bug
OpenSSL 1.0.2t has been installed at /usr/local/opt/openssl on the MacOS image. Homebrew is also installed and used for installing software. Hombrew normally creates a symlink at /usr/local/opt/openssl that points to the latest installed version. OpenSSL 1.0.2 is not supported by Homebrew anymore (Homebrew/homebrew-core#46876).

This can cause issues when the compiler or linker are looking in /usr/local/opt/openssl but the user and/or build scripts expect OpenSSL 1.1 headers and libraries to be at that location.

Area for Triage:
Apple
Packages

Question, Bug, or Feature?:
Bug

Virtual environments affected

• [x] macOS 10.15
• [ ] Ubuntu 16.04 LTS
• [ ] Ubuntu 18.04 LTS
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior
Software/build scripts expect to find the version of OpenSSL installed by Homebrew (1.1.x) symlinked at /usr/local/opt/openssl

Actual behavior
Software/build scripts find OpenSSL 1.0.2t installed at /usr/local/opt/openssl

1. Step 1

ls -l /usr/local/opt | grep openssl
lrwxr-xr-x  1 runner  admin   24 Apr 25 21:13 openssl -> ../Cellar/openssl/1.0.2t
lrwxr-xr-x  1 runner  admin   28 Apr 25 21:07 [email protected] -> ../Cellar/[email protected]/1.1.1g

[al-cheb]

Hello, @simons-public

Hombrew does not really create a symlink by default at /usr/local/opt/openssl that points to the latest installed version.

# Install the latest openssl
$ /usr/local/bin/brew install openssl
$ /usr/local/bin/brew link openssl --force
Warning: Refusing to link macOS provided/shadowed software: [email protected]
If you need to have [email protected] first in your PATH run:
  echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile

For compilers to find [email protected] you may need to set:
  export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
  export CPPFLAGS="-I/usr/local/opt/[email protected]/include"

$ ls -la /usr/local/bin/openssl
ls: /usr/local/bin/openssl: No such file or directory

$ ls -la /usr/local/opt/openssl*
lrwxr-xr-x  1 runner  admin  28 May  7 03:14 /usr/local/opt/[email protected] -> ../Cellar/[email protected]/1.1.1g

As we can see above there are no any symlinks at /usr/local/bin/ folder for openssl.

# Install OpenSSL 1.0.2t
$ brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/8b9d6d688f483a0f33fcfc93d433de501b9c3513/Formula/openssl.rb

$ ls -la /usr/local/opt/openssl*
lrwxr-xr-x  1 runner  admin  24 May  7 03:20 /usr/local/opt/openssl -> ../Cellar/openssl/1.0.2t
lrwxr-xr-x  1 runner  admin  28 May  7 03:14 /usr/local/opt/[email protected] -> ../Cellar/[email protected]/1.1.1

# Set OpenSSL 1.0.2t as default
$ ln -sf /usr/local/Cellar/openssl/1.*/bin/openssl /usr/local/bin/openssl

Software/build scripts expect to find the version of OpenSSL installed by Homebrew (1.1.x) symlinked at /usr/local/opt/openssl

To my mind ,a best practice is to use Homebrew to find where Homebrew packages are installed on mac.

$ brew --prefix openssl
/usr/local/opt/[email protected]

Why can't we deprecate OpenSSL 1.0.2t and use it as the default version on macOS image? - OpenSSL 1.1+ breaks some pipeline tasks - https://github.com/microsoft/azure-pipelines-tasks/issues/11852 . And for that reason we manually create a symlink for openssl 1.0.2t binary to the /usr/local/bin/openssl to preserve backward compatibility.

[simons-public]

@al-cheb Homebrew _does_ actually by default create a symlink by default at /usr/local/opt/openssl that points to the latest installed version. The warning there is about what Homebrew calls linking, which is putting links in /usr/local like the /usr/local/bin/openssl you mentioned.

You can test this by running brew install openssl on a fresh install with no other packages and looking at the symlinks with ls - l /usr/local/opt. I did that this morning myself just to double check.

The command you ran above was ls -la /usr/local/bin/openssl but that is looking for the openssl _command_ in /usr/local/bin which is a different directory than /usr/local/opt.

While I'm aware of the brew --prefix command, this symlink to version 1.0.2 at /usr/local/opt/openssl does break expectations from anyone that has experience with the software looking for the headers and libraries. I had issues where the compiler was using openssl 1.1 headers, but the linker was finding openssl 1.0.2 in the search path first so there were undefined references preventing linking.

It took me a long time to figure out the reason for the linking errors on Azure because these errors weren't happening on any other Macs I tested on with Homebrew and openssl installed.

This is a deviation from normal Homebrew installations. It should be corrected or at the least mentioned on the image readmes that list installed software for 10.14 & 10.15. Something like adding /usr/local/opt/openssl/ to that line of the installed software list so people know that isn't pointing to 1.1.

If there is an issue with pipeline tasks being broken by newer versions I would suggest that it would be better to change the pipeline task to call the old version directly, instead of symlinking it to the default /usr/local/opt/openssl.

[al-cheb]

@simons-public,
Yep, you are right. My fault, I have double checked and confirmed about symlinks. I will update documentation about OpenSSL 1.0.2t point to the /usr/local/opt/openssl/ until the task will not be fixed - https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/Common/ios-signing-common/ios-signing-common.ts

[simons-public]

@al-cheb Thanks!

[al-cheb]

@simons-public, macos-10.15-Readme.md has been updated and included point to OpenSSL 1.0.2t 10 Sep 2019 (/usr/local/opt/openssl -> /usr/local/Cellar/openssl/1.0.2t)

Feel free to open the thread if you have any concerns.