Elixir: cannot install archive directly from github: (Mix) httpc request failed with: {:bad_status_code, 403}

[jlgeering]

Environment

$ mix --version
Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Mix 1.4.4

running on OSX, installed with brew

Current behavior

trying to install

$ mix archive.install https://github.com/edib-tool/mix-edib/releases/download/v0.10.0/edib-0.10.0.ez
Are you sure you want to install archive "https://github.com/edib-tool/mix-edib/releases/download/v0.10.0/edib-0.10.0.ez"? [Yn]
** (Mix) httpc request failed with: {:bad_status_code, 403}

Could not fetch archive at:

    https://github.com/edib-tool/mix-edib/releases/download/v0.10.0/edib-0.10.0.ez

Please download the archive above manually to your current directory and run:

    mix archive.install ./edib-0.10.0.ez

downloading archive using https://httpie.org/

$ http --download https://github.com/edib-tool/mix-edib/releases/download/v0.10.0/edib-0.10.0.ez
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment; filename=edib-0.10.0.ez
Content-Length: 74756
Content-Type: application/octet-stream
Date: Wed, 31 May 2017 06:13:41 GMT
ETag: "038febcecb8a374cd164efb10f500e50"
Last-Modified: Thu, 25 May 2017 04:24:22 GMT
Server: AmazonS3
x-amz-id-2: 10OpSNiYZhNmi9mjGo4q1eND7o5RGlgzw88VJWmIPcIAHSZZwaivwoJQXHpmeOw9jq/C0Oo2UeU=
x-amz-request-id: 2D77A751553F94A3

Downloading 73.00 kB to "edib-0.10.0.ez"
Done. 73.00 kB in 0.30451s (239.74 kB/s)

installing downloaded archive

$ mix archive.install ./edib-0.10.0.ez
Are you sure you want to install archive "./edib-0.10.0.ez"? [Yn]
* creating .mix/archives/edib-0.10.0

Expected behavior

archive can be installed without having to download it first

[josevalim]

@jlgeering I cannot reproduce it here. Can you still consistently reproduce it?

[josevalim]

Closing this for now. We will gladly reopen it if we have more info.

[ishitatsuyuki]

I'm on Arch Linux and also suffering from this:

mix --version
Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Mix 1.6.5 (compiled with OTP 20)

Do you have any tips on getting traces from the underlying httpc library?

[josevalim]

@ishitatsuyuki are you behind a proxy? Can you download the file outside of Mix? 403 is related to authorization, so I wonder why GitHub is not allowing it. Maybe a combination of IP and user agent is causing it to refuse it?

[ishitatsuyuki]

I'm not behind a proxy, and I can download with either curl or Firefox.

[josevalim]

Hrm, I can think of two possible root causes then:

• You can try to setting the user agent in curl to the same that Mix sets. In this case: 'user-agent: Mix/1.6.6'
• You can force curl to use --ipv4 (although this is unlikely to trigger a 403)

Other than that, I don't understand why it would work on curl but not on httpc. :(

[ishitatsuyuki]

No, it doesn't reproduce. My guess is that Mix/httpc is parsing the redirect wrongly or stripping information from it. If the path is wrong or the signed query headers are stripped, it will result in the S3 service returning 403.

[josevalim]

@ishitatsuyuki oh, good call. Do you have the address of the archive you are trying to install? I will take a look.

[ishitatsuyuki]

I was doing this:

mix archive.install https://github.com/wende/elchemy/releases/download/0.7.4/elchemy-0.7.4.ez

Though, the issue reproduces with OP's case of edib as well.

[josevalim]

@ishitatsuyuki unfortunately it works locally. :(

Can you please compile elixir from master and see if you can find more information? It should be doable with:

git clone https://github.com/elixir-lang/elixir.git
cd elixir
make clean compile

This is the code that fails:

https://github.com/elixir-lang/elixir/blob/4f25ccd2d04adce3c93b2e30988cd793ba6c86ec/lib/mix/lib/mix/utils.ex#L585-L586

Maybe you can rewrite this clause to do something like this:

      {:ok, {{_, status, _}, _, _}} = response ->
        IO.inspect response
        {:remote, "httpc request failed with: {:bad_status_code, #{status}}"}

Run make mix, then try to install the archive and see if it prints more information.

You can also call:

:httpc.set_option(:verbose, :verbose, :mix)
# :httpc.set_option(:verbose, :debug, :mix)
# :httpc.set_option(:verbose, :trace, :mix)

Right after this line and run make mix and then try again and see if you can find more info.

[ishitatsuyuki]

I did that and what I found is that it was indeed S3 returning 403. Though, I couldn't judge what's acting wrong even with the trace logs.

The trace logs gave me internal dump of the http related data structures, but the links contained inside that was valid, which means there's probably something wrong at a different layer. I'll try MITM-ing some time later.

[ishitatsuyuki]

What I found through mitmproxy was that the request made was missing ? before the query strings. Any idea how I can locate the root cause?

[josevalim]

The query string is missing on the initial request or after the redirect?

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[ishitatsuyuki]

No, just the ? character (after redirect). A sample:

https://github-production-release-asset-2e65be.s3.amazonaws.com/81605260/ae34400c-5f4b-11e8-8a9e-08305ce0e20bX-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180630%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180630T090917Z&X-Amz-Expires=300&X-Amz-Signature=686c7570244df7d7ef73573faefc6641a29361762b6796633addc9dbbc98d34a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Delchemy-0.7.4.ez&response-content-type=application%2Foctet-stream

[ishitatsuyuki]

I think things are consistently broken with Mix 1.6.6 + OTP21.

Here are a list of environments that reproduces the issue:

• Mix 1.6.6, OTP 19 or 21
• Mix 1.6.5, OTP 20 (OTP 19 from Docker Hub works)

I've confirmed this behavior from both US and JP, which means this is location independent.

[josevalim]

@ishitatsuyuki so I couldn't reproduce it 3 days ago but now I just tried again with the same Elixir and OTP versions and it failed. So my suspicion is that it is probably broken for a while but the error is showing up depending on external factors. I am investigating.

[josevalim]

Got it: https://github.com/erlang/otp/pull/1859 :D

Thanks a lot for your inputs and feedback!