Packer: Packer download always fails with 502 Bad Gateway while wget works

[Wenzel]

• Packer version v1.2.5
• Host platform: Ubuntu 18.04 LTS
• Debug log: Gist

As you can see from the following screenshot, i cannot download Windows images from MSDN anymore, it always fail with a 502 Bad Gateway error:

At the beginning i thought this was a network issue with Microsoft, until i found out that wget was working fine.

There is no reason wget would work better than Packer for a simple GET request.
So, is Packer setting some strange HTTP headers in the GET request ?

How can i investigate ?

Thanks !

[rickard-von-essen]

I can repro this with that exact URL.

[Wenzel]

I have a wireshark capture of wget

vs Packer

[Wenzel]

Holy shit, if you have Packer's User-Agent, the download fails !

WTF.

[rickard-von-essen]

Since this is a http only call, It's probably easy to look at tcpdump and see what happens. (as you just did)

[rickard-von-essen]

Awesome! 👊

[mwhooker]

yikes. This might be their subtle way of telling users to use a local cache. We'll see if we can escalate with them

[SwampDragons]

hmm. @boumenot is there a whitelist that our useragent isn't on, perhaps? Are you the right person at Microsoft to talk to about this?

[boumenot]

No. I can ask around, see if I can find someone. Let me go shine the bat signal.

[SwampDragons]

Thanks!! <3

[Wenzel]

What about adding a command line flag to set the User-Agent in the next release ?
or setting a wget like User-Agent ?

Nothing prevents Microsoft from blocking us again (if they unblock us in a near future...)

[rickard-von-essen]

@Wenzel this is on Microsoft to fix they are clearly violating the HTTP standards.

This is for statistical purposes,
the tracing of protocol violations, and automated recognition of user
agents for the sake of tailoring responses to avoid particular user
agent limitations.

RFC 2616: 14.43 - User-Agent

[SwampDragons]

@Wenzel Microsoft is a pretty close partner of HashiCorp and it's unlikely that this is intentional. Once we get this fixed, I think we can trust that it won't happen again.

[Wenzel]

Do we have some news about this issue ?
@boumenot did you managed to shine the bat signal ? :bulb: :bat:
Tried again today, and still bad gateway :(

[SwampDragons]

@boumenot if you're not getting traction, I can escalate through HashiCorp's partners team instead.

[arizvisa]

Just some thoughts on the consideration of exposing the http user agent to the user...

TLDR: specify the http user agent via an environment variable; use packer/helper/useragent to assign a default one, and then locate all the places that net/http.client is used and fetch the variable from there.

If for some reason that it's decided that the ability to specify the http user agent to packer should be exposed, I think it should either be an environment variable such as "HTTP_USER_AGENT" (like in the cgi protocol, bsd's fetch(1), or w3m), or stashed as a template variable (and then assigned into an environment variable for child processes to inherit). This way it's exposed to everything via os.Getenv without having to manually expose it via some api that one needs to ensure is used.

The reason being is that there's a couple of places where http.client.Do is used but the UserAgent is not assigned (post-processor/vagrant-cloud/client.go comes to mind, and also probably plugins that have to implement it themselves). So other than just the DownloadClient, the User-Agent should be used for all the net/http.client "PUT" requests. Unfortunately, it's only being used for just a couple of builders.

This is likely due to the user-agent being expected to be imported from github.com/hashicorp/packer/helper/useragent and then manually specified in DownloadConfig or as a property for net/http.client. This methodology is prone to being overlooked unless packer exposes some standard api that is used for doing transfers or something to plugins and stuff (which would be a lot of work for only the benefit of making it easier to write robust code).

If it's in an environment variable, however, then it's easy to default the User-Agent at the various times of processing that net/http.client is used to upload/download without needing to import the useragent helper. (Such as at startup, at the parsing of the template, then at the iso or guest-additions download, at time-of-use in the builder/post-processor, etc).

Anyways...

[boumenot]

@SwampDragons, please do. Another voice would be appreciated. I was on vacation, so I have had to restart this. I made a dozen new friends, but I have not gotten a definitive person to address the issue.

[boumenot]

Made some more friends. The are going to see if it is DDoS protections kicking in, and what can be done to address it. In the interim, he would like to know if you replace care.dleservice.microsoft.com with download.microsoft.com is it still blocked.

[SwampDragons]

@Wenzel ^^

[Wenzel]

@boumenot not sure http://download.microsoft.com/... exists ?

Can you provide me the exact URL ?

[boumenot]

Sorry, I don't know either. I am just relaying information from the other side. They are working to see if they can fix the issue, so I wouldn't worry about the download vs dl part.

[boumenot]

Try http://download.microsoft.com/download/2/5/4/254230E8-AEA5-43C5-94F6-88CE222A5846/14393.0.160715-1616.RS1_RELEASE_CLIENTENTERPRISEEVAL_OEMRET_X64FRE_EN-US.ISO

Please note that they are still trying to fix the underlying issue, so this shouldn't be necessary when they do.

[Wenzel]

It's a win :tada:

[mwhooker]

thanks for digging into this @boumenot

[claytonjgilmer]

I am having a similar issue with the link http://care.dlservice.microsoft.com/dl/download/evalx/win7/x64/EN/7600.16385.090713-1255_x64fre_enterprise_en-us_EVAL_Eval_Enterprise-GRMCENXEVAL_EN_DVD.iso. curl and packer fail to pull down the ISO, but wget works fine.

[Wenzel]

Hello,
do we have some news about this issue ?
are we waiting feedback from an insider at Microsoft ?

[SwampDragons]

yeah, I'm still waiting to hear from them. :-/

[SwampDragons]

Is this still an issue?

[Wenzel]

Hi @SwampDragons ,

I don't have this kind of issues anymore, so lets close it.

Thanks.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.