Brew: Brewed curl cannot find the CA certificates and all bottle downloads fail

[ccwang002]

• [x] ran a brew command and reproduced the problem with multiple formulae? If it's a problem with a single, official formula (not cask) please file this issue at Linuxbrew/homebrew-core: https://github.com/Linuxbrew/homebrew-core/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• [x] ran brew update and can still reproduce the problem?
• [x] ran brew doctor, fixed all issues and can still reproduce the problem?
• [x] ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

Installing any package from bottle such as parallel gave the following curl certificate error. Curl complained that the certs are under /home/linuxbrew/.linuxbrew instead of the actual Homebrew folder (HOMEBREW_PREFIX=/diskmnt/Projects/Users/lwang/.linuxbrew). However, the brewed version of curl was installed successfully with the certificates available at $HOMEBREW_PREFIX/etc/openssl/cert.pem.

curl: (77) error setting certificate verify locations:
  CAfile: /home/linuxbrew/.linuxbrew/etc/openssl/cert.pem                                                 
  CApath: /home/linuxbrew/.linuxbrew/etc/openssl/certs

If I set the CA path to the correct location by ~/.curlrc with HOMEBREW_CURLRC=1, the bottles can be successfully downloaded again.

What happened (include command output)

For example, upgrading parallel bybrew upgrade parallel gave:

Command output

==> Upgrading 1 outdated package:

parallel 20181122 -> 20190122

==> Upgrading parallel

==> Downloading https://linuxbrew.bintray.com/bottles/parallel-20190122.x86_64_linux.bottle.tar.gz


curl: (77) error setting certificate verify locations:

  CAfile: /home/linuxbrew/.linuxbrew/etc/openssl/cert.pem

  CApath: /home/linuxbrew/.linuxbrew/etc/openssl/certs

Error: Failed to download resource "parallel"

Download failed: https://linuxbrew.bintray.com/bottles/parallel-20190122.x86_64_linux.bottle.tar.gz

Warning: Bottle installation failed: building from source.

==> Downloading https://ftp.gnu.org/gnu/parallel/parallel-20190122.tar.bz2

curl: (77) error setting certificate verify locations:

  CAfile: /home/linuxbrew/.linuxbrew/etc/openssl/cert.pem

  CApath: /home/linuxbrew/.linuxbrew/etc/openssl/certs

Trying a mirror...

==> Downloading https://ftpmirror.gnu.org/parallel/parallel-20190122.tar.bz2

==> Downloading from https://mirror.us-midwest-1.nexcess.net/gnu/parallel/parallel-20190122.tar.bz2

curl: (77) error setting certificate verify locations:

  CAfile: /home/linuxbrew/.linuxbrew/etc/openssl/cert.pem

  CApath: /home/linuxbrew/.linuxbrew/etc/openssl/certs

Error: An exception occurred within a child process:

  DownloadError: Failed to download resource "parallel"

Download failed: https://ftpmirror.gnu.org/parallel/parallel-20190122.tar.bz2

  

The brewed curl was installed correctly with the certificates.

$ ls $HOMEBREW_PREFIX/etc/openssl/cert.pem
/diskmnt/Projects/Users/lwang/.linuxbrew/etc/openssl/cert.pem

$ head -n 4 $HOMEBREW_PREFIX/etc/openssl/cert.pem
##
## Bundle of CA Root Certificates
##
## Certificate data from Mozilla as of: Wed Jan 23 04:12:09 2019 GMT

The bottles can be downloaded again by setting the correct CA path at ~/.curlrc:

cacert=/diskmnt/Projects/Users/lwang/.linuxbrew/etc/openssl/cert.pem
capath=/diskmnt/Projects/Users/lwang/.linuxbrew/etc/openssl/certs

And run the brew command again with HOMEBREW_CURLRC=1 brew upgrade parallel, the bottled package can be successfully installed.

What you expected to happen

I except that the brewed curl should access the brewed CA certificates even at the non-standard homebrew location and the bottled packages can be installed.

Step-by-step reproduction instructions (by running brew commands)

1. Set up the Homebrew (linux version) on a non-standard location
2. Install any package, say parallel by brew install parallel. The installation will fail
3. Set the ~/.curlrc and install it again by HOMEBREW_CURLRC=1. The installation will succeed

[sjackman]

@ccwang002 Does brew reinstall curl help? Does it pour the curl bottle or build it from source?

[ccwang002]

The curl was built from source. brew reinstall curl does not help either.

[sjackman]

It's really quite weird that curl is looking for /home/linuxbrew/.linuxbrew/etc/openssl/ when your prefix is /diskmnt/Projects/Users/lwang/.linuxbrew. Could you please report…

grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/.linuxbrew/opt/openssl/
grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/.linuxbrew/opt/curl/

[jonchang]

I've actually hit this issue as well on my non-default prefix Linuxbrew install. Running brew reinstall -s curl fixed the issue.

[jonchang]

I suspect the issue is here:

      args << "--with-ca-bundle=#{etc}/openssl/cert.pem"
      args << "--with-ca-path=#{etc}/openssl/certs"

which I suspect hardcodes the path to the certificates in the bottle.

[ccwang002]

@sjackman here is the output of the grep search:

$ grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/.linuxbrew/opt/openssl/
(nothing)
$ grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/.linuxbrew/opt/curl/
/diskmnt/Projects/Users/lwang/.linuxbrew/opt/curl/lib/libcurl.so.4.5.0
/diskmnt/Projects/Users/lwang/.linuxbrew/opt/curl/lib/libcurl.a

[sjackman]

Please report the output of…

brew config
brew --prefix
brew remove curl
brew install -s curl
grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/.linuxbrew/opt/curl/

[iMichka]

Maybe have a look at: https://github.com/Linuxbrew/brew/wiki/FAQ#why-does-curl-fail ?

[JamezQ]

I can confirm that I had the same issue and brew reinstall -s curl fixes it.

[ccwang002]

@sjackman here is the output (I reinstalled the homebrew at /diskmnt/Projects/Users/lwang/linuxbrew):

Command output

$ brew config
HOMEBREW_VERSION: 2.0.1-29-g9084a62
ORIGIN: https://github.com/Homebrew/brew
HEAD: 9084a629a534ca5e291fbc985116f3ba0f5b7d11
Last commit: 20 hours ago
Core tap ORIGIN: https://github.com/Linuxbrew/homebrew-core
Core tap HEAD: 123c41474b80a1ed7073e5d75215c38959da7142
Core tap last commit: 16 hours ago
HOMEBREW_PREFIX: /diskmnt/Projects/Users/lwang/linuxbrew
HOMEBREW_REPOSITORY: /diskmnt/Projects/Users/lwang/linuxbrew/Homebrew
HOMEBREW_CELLAR: /diskmnt/Projects/Users/lwang/linuxbrew/Cellar
HOMEBREW_CACHE: /diskmnt/Projects/Users/lwang/.cache/Homebrew
HOMEBREW_TEMP: /run/user/1020
HOMEBREW_DEV_CMD_RUN: 1
HOMEBREW_LOGS: /diskmnt/Projects/Users/lwang/.cache/Homebrew/Logs
HOMEBREW_MAKE_JOBS: 6
HOMEBREW_SYSTEM_CURL_TOO_OLD: 1
CPU: 32-core 64-bit skylake
Homebrew Ruby: 2.3.7 => /diskmnt/Projects/Users/lwang/linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/bin/ruby
Clang: N/A
Git: 2.20.1 => /diskmnt/Projects/Users/lwang/linuxbrew/bin/git
Curl: 7.64.0 => /diskmnt/Projects/Users/lwang/linuxbrew/opt/curl/bin/curl
Kernel: Linux 3.10.0-862.14.4.el7.x86_64 x86_64 GNU/Linux
OS: CentOS Linux release 7.5.1804 (Core) 
Host glibc: 2.17
/usr/bin/gcc: 4.8.5
glibc: 2.23
gcc: 5.5.0_4
xorg: N/A

$ brew --prefix
/diskmnt/Projects/Users/lwang/linuxbrew

$ brew uninstall --ignore-dependencies curl

$ brew install -s curl
Updating Homebrew...
==> Downloading https://linuxbrew.bintray.com/bottles/curl-7.64.0.x86_64_linux.bottle.tar.gz
Already downloaded: /diskmnt/Projects/Users/lwang/.cache/Homebrew/downloads/3f7451ddc3cb6e438c5123d967d5a32db4c425e0ce33e00afa132bbfeb085d10--curl-7.64.0.x86_64_linux.bottle.tar.gz
==> Pouring curl-7.64.0.x86_64_linux.bottle.tar.gz
🍺  /diskmnt/Projects/Users/lwang/linuxbrew/Cellar/curl/7.64.0: 450 files, 3.7MB
Warning: curl 7.64.0 is already installed and up-to-date
To reinstall 7.64.0, run `brew reinstall curl`
$ grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/linuxbrew/opt/curl/
/diskmnt/Projects/Users/lwang/linuxbrew/opt/curl/lib/libcurl.a
/diskmnt/Projects/Users/lwang/linuxbrew/opt/curl/lib/libcurl.so.4.5.0

It seems that the bottled curl has harcoded the CA path. I followed @jonchang's suggestion to force installing curl form the source by brew reinstall -s curl, and now everything works.

$ grep -rl /home/linuxbrew /diskmnt/Projects/Users/lwang/linuxbrew/opt/curl/
(nothing)

Maybe the wiki should add a note to force installing curl from the source. Thanks for your help!

[sjackman]

Ah, there was a regression four days ago in commit https://github.com/Linuxbrew/homebrew-core/commit/fc4a4f0c3fdf676ce086ad93066e2ee8a223ad4b#diff-a7a59ba41131628f779debf43858c613.
This necessary code was removed.

  pour_bottle? do
    reason "The bottle needs to be installed into #{Homebrew::DEFAULT_PREFIX} when built with OpenSSL."
    satisfy { OS.mac? || HOMEBREW_PREFIX.to_s == Homebrew::DEFAULT_PREFIX }
  end

@iMichka Are you able to fix this up? I noticed that the macOS bottles are also missing.
https://github.com/Linuxbrew/homebrew-core/blob/3c44801c2676fc6f4cf0f6d4a36056c98af46557/Formula/curl.rb#L8-L12

[iMichka]

Woups. My bad. I'll fix this ASAP.

[iMichka]

See https://github.com/Linuxbrew/homebrew-core/pull/11659

[sjackman]

Thanks, Michka!