Linuxbrew-core: Curl with brew fails to download git tar.xz

[jwhite007]

If Homebrew was updated on Aug 10-11th 2016 and brew update always says Already up-to-date. you need to run: cd "$(brew --repo)" && git fetch && git reset --hard origin/master && brew update.

Please follow the general troubleshooting steps first:

• [x] Ran brew update and retried your prior step?
• [x] Ran brew doctor, fixed as many issues as possible and retried your prior step?
• [ ] If you're seeing permission errors tried running sudo chown -R $(whoami) $(brew --prefix)?
  

  Bug reports:

Curl with brew fails to download git tar.xz; however, if I issue the call to curl myself, it works. I put the tar.xz for git, git's html docs, and git's man pages into .cache/Homebrew/. Upon brew upgrading git again, brew proceeds past the download of the git tar.xz, because it's found in the cache, but then it fails attempting to download the manpages which brew should find in the cache as well. It may be due to brew not using the curl that I've installed with brew.

Please see all relevant output:
https://gist.github.com/jwhite007/c8cdaab5b7d9c5c46b7b28827af655ba

[sjackman]

Are you using a HTTP proxy?

[jwhite007]

@sjackman
Shaun,

Thanks for the reply. I'm using both HTTP and HTTPS proxies.

I'm not sure if you remember. I told you that my company had moved to a new IT infrastructure with a very strict firewall. Well... IT finally trusted me enough to give me access to the proxy. I am now free to use Linuxbrew again. Yipee!!!

Thanks for your hard work.

Cheers,
James

[sjackman]

Oh, yes, I do remember that story. Whew. I'm glad to hear that your situation has improved.

How have you configured the proxy? Try

export http_proxy=http://<proxyhost>:<proxyport>
brew fetch hello

See https://github.com/Homebrew/brew/blob/master/docs/Tips-N'-Tricks.md#using-homebrew-behind-a-proxy

[jwhite007]

That is how I have my proxy configured. brew fetch hello works just fine as do downloads for all other linuxbrew packages. The only problems I've had so far are for the git associated xzs. I have no problem with curl -O https://www.kernel.org/pub/software/scm/git/git-2.10.1.tar.xz The curl that I am calling there is the linuxbrew curl.

[sjackman]

Wow, that's very strange. I don't really have any suggestions how to troubleshoot that. If this issue only affects brew fetch git, and you have git working now by downloading the files manually into the cache, then I suggest we just leave this issue open and see if it affects anyone else.

[jwhite007]

Well git doesn't install, because even though I have the xzs to the htmldocs and manpages in the cache, brew does not find them. As soon as it gets to the point where it's downloading the htmldocs, the install chokes.

$ ls .cache/Homebrew/git-*2.10.1*
.cache/Homebrew/git-2.10.1.tar.xz  .cache/Homebrew/git-htmldocs-2.10.1.tar.xz  .cache/Homebrew/git-manpages-2.10.1.tar.xz
$ brew fetch git
==> Downloading https://www.kernel.org/pub/software/scm/git/git-2.10.1.tar.xz
Already downloaded: /home/sbcuser/.cache/Homebrew/git-2.10.1.tar.xz
SHA256: a9c32df9061cfc2c194ecaa0357ba273c1f24a0dbf0d856b03dcedafcd0a4dc6
Resource: html
==> Downloading https://www.kernel.org/pub/software/scm/git/git-htmldocs-2.10.1.tar.xz

curl: (35) SSL connect error
Error: Failed to download resource "git--html"
Download failed: https://www.kernel.org/pub/software/scm/git/git-htmldocs-2.10.1.tar.xz

[jwhite007]

The problem is that brew is using /usr/bin/curl, an older version that doesn't work properly...

$ /usr/bin/curl -O https://www.kernel.org/pub/software/scm/git/git-htmldocs-2.10.1.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) SSL connect error

$ /usr/bin/curl --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

$ which curl
/toolbox/NGS/.linuxbrew/bin/curl

$ curl --version
curl 7.50.3 (x86_64-pc-linux-gnu) libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

$ curl -O https://www.kernel.org/pub/software/scm/git/git-htmldocs-2.10.1.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1042k  100 1042k    0     0   674k      0  0:00:01  0:00:01 --:--:--  710k

$ echo $PATH
/toolbox/NGS/.linuxbrew/lib/perl5/perl_local/bin:/toolbox/NGS/.linuxbrew/bin:/toolbox/NGS/.linuxbrew/sbin:/home/sbcuser/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin

[jwhite007]

Another problem is that brew does not look for cached downloads of resource xzs.

[sjackman]

The filenames of the cached resources are different than what you expect:

~/.cache/Homebrew/git--html-2.10.1.tar.xz
~/.cache/Homebrew/git--man-2.10.1.tar.xz

[jwhite007]

From git.rb...

  resource "html" do
    url "https://www.kernel.org/pub/software/scm/git/git-htmldocs-2.10.1.tar.xz"
    sha256 "40f9ce980f91c0391438e5d04a1bb11b128c6b6bf272eac44d6fca80f2dacfcb"
  end

  resource "man" do
    url "https://www.kernel.org/pub/software/scm/git/git-manpages-2.10.1.tar.xz"
    sha256 "89ff7aae275252d6b01cac6479e10e1be2e0208181ff14decf200606b5223611"
  end

[sjackman]

Those files are named as above when stored in the .cache directory. Try it.

[jwhite007]

Hmm. Interesting. I changed the names, and now the upgrade works. Thanks for your help. I see that brew on mac uses brewed curl. Why not the same on linux?

[sjackman]

On Mac Brewed curl is used if you're using MacOS < 10.9, otherwise the system curl is used.

[sjackman]

I've changed the default curl to the brewed curl in PR Linuxbrew/brew#143. I'll merge it once it passes CI. Thanks for the bug report, James.

[jwhite007]

I guess that I don't understand so well what's written in git.rb.

Thanks for the help and the fix, Shaun!

[sjackman]

I've merged the fix. No worries, James. Thanks again for the bug report.

[jwhite007]

I'm always happy to help, even if it's just reporting bugs.

[ijkim88]

I'm running into an issue similar to this, but I'm having trouble downloading git with the brewed curl:

$ brew install https://www.kernel.org/pub/software/scm/git/git-2.13.1.tar.xz
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Error: Failure while executing: /home/ikim/.linuxbrew/opt/curl/bin/curl --remote-time --location --user-
agent Linuxbrew/1.X.Y\ (Linux;\ x86_64\ 3.10.0-514.21.2.el7.x86_64)\ curl/7.54.1 --progress-bar --fail 
https://www.kernel.org/pub/software/scm/git/git-2.13.1.tar.xz -o 
/home/ikim/.cache/Homebrew/Formula/git-2.13.1.tar.xz

$ brew install http://www.kernel.org/pub/software/scm/git/git-2.13.1.tar.xz
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

I've tried fetch as well and curl -O. The system /usr/bin/curl -O seems to work fine.

I edited the openssl formula to pull https://curl.haxx.se/ca/cacert-2017-06-07.pem instead of https://curl.haxx.se/ca/cacert-2017-01-18.pem since it was outdated. Not sure if this has any effect.

I'm within a company network, but I set the http_proxy and https_proxy env variables correctly.

$ /usr/bin/curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

$ ~/.linuxbrew/bin/curl --version
curl 7.54.1 (x86_64-pc-linux-gnu) libcurl/7.54.1 OpenSSL/1.0.2l zlib/1.2.11
Release-Date: 2017-06-14
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

[sjackman]

I'm not able to reproduce this error on my local development box, so it may be tricky to troubleshoot.
Try…

brew update
brew upgrade curl
brew fetch git

[sjackman]

As a workaround, you can try

curl -o ~/.cache/Homebrew/git-2.13.1.tar.xz https://www.kernel.org/pub/software/scm/git/git-2.13.1.tar.xz
brew fetch git

[ijkim88]

Here's my output:

$ brew update
Already up-to-date.
$ brew upgrade curl
Error: curl 7.54.1 already installed
$ brew fetch git
==> Downloading https://www.kernel.org/pub/software/scm/git/git-2.13.1.tar.xz

curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Error: Failed to download resource "git"
Download failed: https://www.kernel.org/pub/software/scm/git/git-2.13.1.tar.xz

Can I override some setting to use the system curl instead of the brewed curl?

This affects other https downloads as well after brew installs curl.

[ijkim88]

If I use the system curl and download to ~/.cache/Homebrew then do a brew install git, it seems to work. But I will have to do this manually for any other install that downloads over https.

Would it make sense to add a HOMEBREW_SYSTEM_CURL variable or something similar?

[sjackman]

It'd be preferable to figure out why the brewed curl isn't working for you. As a workaround you can edit the file ~/.linuxbrew/Library/Homebrew/brew.sh and make the following change

-  HOMEBREW_CURL="curl"
+  HOMEBREW_CURL="/usr/bin/curl"

[sjackman]

Another workaround option is to delete both of the symlinks:

~/.linuxbrew/bin/curl
~/.linuxbrew/opt/curl/bin/curl

or alternatively

ln -s /usr/bin/curl ~/.linuxbrew/opt/curl/bin/curl

[ijkim88]

Thanks! I'll use the workaround for now. I'll look into why the brewed curl isn't working and see what I can find.

[ijkim88]

It looks like it's not related to the brewed curl. It may have something to with the proxy server. But the workaround seems to be working.