Carthage: HTTP/HTTPS Proxy Regression in Carthage 0.29.0

[jimstump]

• carthage install method: [ ] .pkg, [X] homebrew, [ ] source
• which carthage: /usr/local/bin/carthage
• carthage version: 0.29.0
• xcodebuild -version: Xcode 9.3 Build version 9E145
• Are you using --no-build? No
• Are you using --no-use-binaries? No
• Are you using --use-submodules? No
• Are you using --cache-builds? No
• Are you using --new-resolver? No

Cartfile

github "datatheorem/TrustKit" == 1.5.2

• Carthage command used: carthage update --platform iOS

Environment Considerations
I am running Charles Proxy on my mac (Port 8888 as indicated below), with the following environment variables set from my ~/.profile. Charles Proxy is configured with the correct credentials for my company's corporate HTTP proxy (other external HTTP requests work). My computer has no external Internet access without going through the corporate HTTP proxy.

• echo $http_proxy: http://127.0.0.1:8888
• echo $https_proxy: http://127.0.0.1:8888
• echo $HTTP_PROXY: http://127.0.0.1:8888
• echo $HTTPS_PROXY: http://127.0.0.1:8888
• curl --head https://api.github.com/repos/datatheorem/TrustKit/releases/tags/1.5.2:

HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
Server: GitHub.com
Date: Sat, 14 Apr 2018 21:41:47 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1947
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 56
X-RateLimit-Reset: 1523744619
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: "bb9389054bfa79414e134673f834a533"
Last-Modified: Tue, 28 Nov 2017 01:57:11 GMT
X-GitHub-Media-Type: github.v3; format=json
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.041056
X-GitHub-Request-Id: 8C01:2462:DE88E4:2235503:5AD2759B

Carthage Output

*** Fetching TrustKit
*** Checking out TrustKit at "1.5.2"
*** xcodebuild output can be found in /var/folders/10/8l_ybyzx2g7g2mjmlg1mr8j4412dbs/T/carthage-xcodebuild.9cZd4r.log
*** Skipped installing TrustKit.framework binary due to the error:
    "GitHub API request failed: networkError(Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x7facbdf085a0 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://api.github.com/repos/datatheorem/TrustKit/releases/tags/1.5.2?, NSErrorFailingURLKey=https://api.github.com/repos/datatheorem/TrustKit/releases/tags/1.5.2?, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.})"
*** Building scheme "TrustKit" in TrustKit.xcodeproj

Actual outcome
Carthage did not appear to download the external dependency due to a network error. Based on my environment, my assumption is that the HTTP(S) proxy was not used. This seems to be a regression in 0.29.0 as everything works correctly using 0.28.0 (everything else stayed the same - I used the brew switch command to swap versions).

Attempting to figure out what might have changed between those versions relating to proxies led me to pull request #2371 which was merged right before the release of 0.28.0. I have not, however, attempted to build carthage before and after that pull request merged to test the behavior.

Expected outcome
Carthage should be able to fetch external dependencies using the configured HTTP/HTTPS proxy.

[tmspzz]

@jimstump Thanks for opening an issue.

I am a little perplexed as before 0.29.0 there was no proxy support.

Could it be that this in an issue with your git configuration?

What is the value if http.proxy in your ~/.gitconfig (global) and `git/config (local) to your repo

[tmspzz]

~... and I think I found the problem: https://github.com/Carthage/Carthage/blob/088ab9982cc483f87e0ae20532dc437285c1dc5c/Source/CarthageKit/Project.swift#L600~

Edit: nope, this should be using the proxy settings too.

[tmspzz]

@jimstump did you export your https_proxy variables?

I can see all traffic going via Charles to the final host. Proof is that if I turn off Charles and keep my proxy variables everything fails when running carthage

[jimstump]

@blender thanks for looking into this. The directory that I made for testing before opening this issue was not a git repo, so there is no local git config that would be interfering. However, I do have a global git config to point to Charles as well:

git config --global http.proxy: http://localhost:8888
git config --global https.proxy: http://localhost:8888

Also, I have exported my proxy variables from my ~/.profile file, which is sourced from my ~/.bash_profile (see below):

~/.bash_profile

# source the .profile, if it exists
if [ -f ~/.profile ]; then . ~/.profile; fi

~/.profile

export http_proxy=http://127.0.0.1:8888
export https_proxy=http://127.0.0.1:8888
export HTTP_PROXY=$http_proxy
export HTTPS_PROXY=$https_proxy

I am not at work this week, so I'm not connected to my normal "proxy-only" network. However, your comment about you seeing all traffic via Charles made me compare the requests that I see in Charles between Carthage 0.28.0 and Carthage 0.29.0.

When using Carthage 0.28.0 in a directory that only has the Cartfile from the original post, I see the following Github-related requests in Charles:

• https://api.github.com/repos/Carthage/Carthage/releases?page=1&per_page=2
• https://github.com/datatheorem/TrustKit.git/info/refs?service=git-upload-pack
• https://api.github.com/repos/datatheorem/TrustKit/releases/tags/1.5.2?

However, when using Carthage 0.29.0 in a clean directory with just that Cartfile, I only see the following Github-related requests in Charles:

• https://github.com/datatheorem/TrustKit.git/info/refs?service=git-upload-pack

I disconnected from the network between runs in order to guarantee that any keep-alive connections were dropped.

This makes me think that in Carthage 0.29.0, while the request for the TrustKit refs is using the configured proxy, the API request for the given tag is not (that API request is also the request that was throwing the error when connected to my corporate network).

[tmspzz]

@jimstump I'm experiencing the same.

I think that:

• This https://github.com/datatheorem/TrustKit.git/info/refs?service=git-upload-pack is the clone/fetch of the repository going via the proxy

The other 2 requests don't appear in Charles when setting https_proxy or any other variation.

But, nevertheless with https_proxy and http_proxy set I have debugged and saw that:

1. The requests do happen and succeed
2. The proxy is configured in the URLSession they use. I have even forced ignoring local and remote caches and nilled the urlCache because I though maybe some caching is going on.

[jimstump]

@blender, So I did some searching today and came across a StackOverflow Answer related to manually adding proxy information to a URLSession that mentioned that the value for kCFNetworkProxiesHTTPProxy and kCFNetworkProxiesHTTPSProxy "should be a hostname and not contain any URL scheme". This appears to be validated by documentation for kCFNetworkProxiesHTTPProxy and kCFNetworkProxiesHTTPSProxy.

Looking at Pull Request 2371, it _appears to me_ that the value of these keys in the dictionary are full URLs (like I have set in my environment variables).

I would be interested in @jakeadams thoughts, since he worked on the original pull request. A quick and dirty test of replacing the values of said proxy keys with proxyURL.host in Proxy.swift appeared to work for me in my environment (I saw all three above URLs in Charles), but I'm concerned that I might be missing something.

[jtambolic1]

From the development work I've done, kCFNetworkProxiesHTTPProxy should be the hostname, and kCFNetworkProxiesHTTPPort should be the port number. Unfortunately those only cover HTTP connections. For HTTPS connections, you'll need to define the keys "HTTPSProxy" and "HTTPSPort", and set "HTTPSEnable" to true. Last time I checked, the constants for those were defined on macOS but not iOS or tvOS.

[tmspzz]

@jimstump I can confirm that by omitting the scheme and using the host I can see the traffic in Charles.

Can you please try this: https://github.com/Carthage/Carthage/pull/2428

[jimstump]

@blender I can confirm that #2428 works for me. Thanks for creating the PR, I was going to work on that tomorrow.

[tmspzz]

@jimstump nice job, well done. 🤝

[jimstump]

Since #2428 was merged, and I was able to successfully run the latest code in master in my proxy environment, I'm closing this issue.

Thanks everyone!