Julia: Julia's packaged libcurl is not built with proxy support

[tedsteiner]

On a fresh download of Julia there are issues using a proxy when downloading packages. As a simple example, the following happens when you try to add a package:

(v1.1) pkg> add DataFrames
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
ERROR: failed to clone from https://github.com/JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:Net, curl error: Unsupported proxy 'https://XXX.XXX.XX.XX:XXXX/', libcurl is built without the HTTPS-proxy support.
)

(I blanked out my proxy information, but the server information was correct.)

The issue is that the official build of Julia packages a build of libcurl that does not support connections through a proxy. I'm guessing this is just a compile flag that someone needs to fix.

There is a simple workaround on Ubuntu 16.04 -- just delete Julia's packaged version of libcurl.so. So if you go into `julia-1.X.X/lib/julia/ and delete libcurl.so (and symlinked friends), Julia seems to kick over to the system libcurl and works without issue.

Here is my version info, for reference:

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

I got the same behavior on the long term stable version 1.0.3.

[ViralBShah]

I certainly have used the bundled libcurl in the past with proxy servers. Not sure if something changes recently. Is there a particular flag we need to pass to the build?

[ViralBShah]

Maybe this is an issue with https rather than proxy?

[tkluck]

Unsupported proxy 'https://XXX.XXX.XX.XX:XXXX/'

@tedsteiner, I've seen many cases where one needs to set http:// for the proxy, even for https_proxy. This does not come at the expense of lost security since the client will use the CONNECT method and do a secure TLS handshake with the upstream server. It's likely that this will solve the issue you are seeing.

(That doesn't mean Julia shouldn't consider just adding the right flags to its libcurl build, or just use system build whenever possible.)

[joshjob42]

I have a similar, though subtly different problem using v1.1.0; with an error message

dev Distributions
Cloning git-repo `http://github.com/JuliaStats/Distributions.jl.git`
ERROR: failed to clone from http://github.com/JuliaStats/Distributions.jl.git, error: GitError(Code:ECERTIFICATE, Class:SSL, the SSL certificate is invalid: 0x10000 - The certificate is signed with an unacceptable key (eg bad curve, RSA too short).)

I've set http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY in ENV, and the relevant variables in ~/.gitconfig to my corporate proxy, and toggled the https proxies between truly being https and instead being the http (as suggested by @tkluck) , and have also tried changing Pkg.setprotocol! to git (which freezes/doesn't do anything/times out), https, and http (both of which give this error message).

Pretty much at the end of my rope as I've run through every potential solution I've found in the issues here. Not sure if I should drop this in this issue or move to a different one, but since this one is about proxy problems and is very new, I thought I'd try here first.

[ViralBShah]

I think this should be a separate issue.

[tedsteiner]

@ViralBShah Changing my https_proxy to use http did fix the issue. I have seen a similar issue before but didn't think of that in this case, thank you.

However, it would be nice to have this fixed because on Ubuntu the system-wide proxy settings automatically add http:// and https:// for the two proxy environment variables. So using Julia's libcurl requires me to reset the system proxy locally before starting Julia.

Edit: Sorry, thanks to @tkluck for the proxy tip

[StefanKarpinski]

This isn't a Julia issue per se, it's a libcurl issue as far as I can tell. There's only so much it makes sense to hack into external libraries to make things like this work. We already maintain forks of a lot of libraries and not only is it a resource drain to do that, but people actively get upset at us for not using the vanilla versions of libraries like this. So I'm disinclined from doing anything given that there is a way to make this work. Better documentation is always welcomed though.

[tedsteiner]

I agree that hacking into or changing external libraries is undesirable. However, the default library on Ubuntu works as desired while Julia's packaged version does not. This made me suspect that there may just be a build option for libcurl that is configured by default on Ubuntu but not by Julia's packaging system, and, if so, enabling that option would make proxies work correctly out of the box. If not, hopefully this issue helps people who search for help on this error message in the future.