Packer: file provisioner with winrm communicator is unusably slow..

[trodemaster]

I have been building a couple different winrm based windows build with VMware Fusion/Workstation. Some of the files I need to copy are near 100MB and transfer is taking a very long time. Currently I'm seeing around 40KB/s. Workarounds have been discussed but wanted to get an issue open for tracking.

For reference this template will demonstrate how slow the transfer goes.
https://github.com/trodemaster/packer-WinSrv2012r2/blob/winrm/

Logs are not having any useful information. Let me know if any other details are needed.

[cbednarski]

Thanks for the report! Your link is a 404 for me.

As an aside I think it might be useful to add timestamps to the logs so I can see delays: #2649

[cbednarski]

I believe the related discussion is here: Slow upload of files using file provisioner?.

My initial speculation is that this is related to winrm using HTTP/SOAP as the transport layer, as opposed to SSH. As a workaround it might be faster to use SSH transport for large files.

[pecigonzalo]

As far i know this is because we use WinRM as the transport, and its not incredibly fast the way we are doing it.
Another workaround if you are building locally would be to use the HTTP server that Packer provides.
Virtualbox EG: https://www.packer.io/docs/builders/virtualbox-iso.html#http_directory

Ill give a look to this code on the WinRM project and see if i can find a way of making it faster.

[trodemaster]

Here is the correct link for the repo.
https://github.com/trodemaster/packer-WinSrv2012r2/tree/winrm

Using the http server is a usable workaround for now. Having winrm move files at a reasonable speed should be a goal for the packer communicator however. I'll collect a log so we have a detailed reference of the current state. Thanks!

[pecigonzalo]

Im working on paralellising the WinRM transfer, that should greatly improve the speed, but its not easy :D

[pecigonzalo]

Im working on this under https://github.com/pecigonzalo/winrmcp/tree/f-Parallel
Still a WIP.
Any input on this will help, ill submit a PR to winrmcp once its on a working condition.

[pecigonzalo]

https://github.com/packer-community/winrmcp/pull/6

[DenverOps]

Anyone have any thoughts on fixing this? I switched my build files completely over to winrm. I will need to move this back to SSH since I can't copy needed ISO for install.

[pecigonzalo]

@pknath19 the PR is under the link i posted before, ill check if i can get any traction on that, otherwise ill fork and ask packer to use my fork.

[DenverOps]

Thanks. I just seen that after I posted. Looks Good!!! @pecigonzalo

[StefanScherer]

@pecigonzalo Worth reading http://www.hurryupandwait.io/blog/whats-new-in-the-ruby-winrm-gem-15
Although it's Ruby (useful for Vagrant) there are some interesting improvements going on like implementing PSRP.

[pecigonzalo]

@StefanScherer ill give it a look now

[christrotter]

@pecigonzalo Thanks for the pointer to the Packer HTTP server! Also ran into the file provisioner over WinRM issue. I ended up setting up IIS on my laptop to host files, ha.

Alternatively, there's also the guest_additions_url (although I haven't been able to get that working with a 'file' url).

[StefanScherer]

Unfortunately the built-in HTTP server is not available in provisioner scripts, at least the variable to fetch the random port. But you can limit the port range to one single port, that might work.

[cbednarski]

Unfortunately the built-in HTTP server is not available in provisioner scripts, at least the variable to fetch the random port. But you can limit the port range to one single port, that might work.

This is probably the way forward. From the research I have done on WinRM, it uses an XML over HTTP protocol. This means any binary is chunked, converted into base64, and sent over an XML payload. This is never going to be fast. For small files this is OK but for large files you'll need to use something else.

[AndrewFarley]

Sorry to bring this thread back to life, but I'm curious if there is any possibility of embedding into the windows provisioner for packer a simple standalone server (rsync/http/netcat/ssh/etc) which you can install onto the server and then accept large file transfers. Similar in a sense to to Github's LFS, it would be a special option for the provisioner that when enabled would install and run a one-time server to facilitate pushing large files (assumably do this in userdata before the user's userdata). Otherwise, external engineering must be done in a build process (such as pushing to S3 then pulling from S3 in the instance, or rolling our own file-push mechanisms).

[trodemaster]

Hey Andrew see this issue I filed for a feature addition that allows http access to files https://github.com/mitchellh/packer/issues/2869

Currently I just wrote a simple http file server in go. I set the address as an environment variable and use powershell to download large files over http. For our release automation it does pull from an external s3 bucket as you suggest.

[AndrewFarley]

@trodemaster Thanks, well, it looks like this won't be moving any time soon but I'll follow that bug and see if it goes somewhere and use it in the future, but build a custom/manual implementation of this via S3 for now. Cheers

[sandersaares]

Using Copy-Item -ToSession in PowerShell it takes 6 seconds to copy packer.exe to a local VM via WinRM.
Using the "file" provisioner it takes minutes.

I therefore reject the claim that this is slow by design. There is simply a deficiency of some sort in Packer that causes WinRM copies to be extremely slow. Whether it is a missing feature (could be this fast copy mechanism is part of some new API) or simply a slow implementation is unclear to me but this issue should be reopened, as there is clearly room for improvement.

[rickard-von-essen]

Using Copy-Item -ToSession in PowerShell it takes 6 seconds to copy packer.exe to a local VM via WinRM.
Using the "file" provisioner it takes minutes.

@sandersaares Could you create a new issue and provide as much details as possible. This is not rely a Packer issue but more of a upstream bug with the winrm
or winrmcp libraries.

[sandersaares]

I rather lack the motivation to file an issue about a problem unfixed for 2 years with a wontfix label, sorry. It just seems like wasted bureaucracy.

[rickard-von-essen]

@sandersaares I think it was closed because of lack of understanding of the problem. You seems to be able to provide enough details to make this progress. If we have a benchmark number of how fast it should be and can reproduce this we can probaly nail down why it's slow and provide the upstream library with enough facts for them to solve it.

[pecigonzalo]

Lack of understanding? It was discussed extensively under https://github.com/packer-community/winrmcp/pull/6 and here...

[rickard-von-essen]

That seems inconclusive. If that is the solution or a big step forward _please try to drive that change_ and then open a PR or a issue to update winrmcp.

[pecigonzalo]

The solution maybe, but the problem is not inconclusive. mwrock even wrote a blog post explaining it.

[rickard-von-essen]

That is great, hope someone can translate it into executing code.

[sandersaares]

I hope so too! Here is a log and image with some additional perspective on the issue.

2017-08-09T07:29:08.4172438Z ==> Windows2016-HyperV: Uploading Content => C:/Setup
2017-08-09T07:38:56.0005396Z ==> Windows2016-HyperV: Provisioning with Powershell...

[lmayorga1980]

Tried to upload a simple packages.config file

180B Oct 5 11:54 files/chocolatey/packages.config

and literally is taking more than 5 minutes to upload the file

  {
      "type": "file",
      "source": "files/chocolatey/packages.config",
      "destination": "C:\\ProgramData"
    },

Packer Version: 1.1.0

[sandersaares]

I just gave up and am now uploading everything to an HTTP server on the host and then downloading it on the guest...

[mwhooker]

I just spend a fair amount of time rewriting winrmcp to compress the the file and upload in parallel (up to 250 workers) and didn't notice a significant speed increase. I tested with a 100MB file sourced from urandom, so the compression benefits won't be noticeable with the test data. A local transfer only went from 35 minutes to 27m24s.

You can check out the branch here if interested https://github.com/packer-community/winrmcp/compare/master...mwhooker:high_parallelism

I'm not sure I will pursue it, because the gains don't seem to be spectacular, and I want to refocus on making sure we can upload files over better protocols.

It's worth noting that with a local VM, using the code in master, I can usually transfer up to 10mb in under 3-4 minutes. If it's taking longer than that, I would inspect your winrm config. Would also be happy to get people testing my branch to get a better sense of any gains

[pecigonzalo]

Be careful with the amount of workers, as far as I recall, that many are going to slow things down. Something got worse tho, because I dont remember the speed being this bad.

Again, check the PR to accomplish this:
https://github.com/packer-community/winrmcp/pull/6

[sandersaares]

I would suggest exploring deeper what the PowerShell command to copy files over WinRM does (Copy-Item -ToSession). It seems to copy at "full speed" for the given conditions (disk/network bound).

From what I can tell, the relevant code is CopyFileStreamToRemoteSession at https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L4493 which in turn executes the helper PSCopyFileToRemoteSession from https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L8992 on the remote host.

I tested with PowerShell desktop before, not PowerShell Core, so it is possible that the behavior is different in this case.

[pecigonzalo]

Yeah that uses the new winrm upload format, as mwrock explained on his blog
post, but the code seems fairly complex.

What do you mean with the split being dubious tho?

On Sat, Oct 7, 2017, 7:47 PM Sander Saares [e-mail] wrote:

I would suggest exploring deeper what the PowerShell command to copy files
over WinRM does. It seems to copy at "full speed" for the given conditions
(disk/network bound). The entire "split file into segments to be uploaded"
approach used here is extremely dubious to start with.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/packer/issues/2648#issuecomment-334953627,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJcvia4WrVVus9W29xEE_dmuqAjRqAhJks5sp7k2gaJpZM4FvhOK
.

[sandersaares]

What do you mean with the split being dubious tho?

I just was thinking that transferring a few kilobytes per operation could not possibly be fast no matter how you parallelize or compress it. I would expect such logic to work on megabytes, not kilobytes. But in the end, it is the benchmarks that have authority to speak on the matter, so my speculation might be irrelevant here.

[pixelicous]

Just want to give my two cents.. i saw so many threads, this is obviously not a WinRM issue, i've seen files sent much quicker..

The most quickest solution for me was to use powershell provisioner inline, download azcopy using web request, install the msi silently and use the exe file to download the files to the vm from an azure storage

I tried using the ssh communicator with OpenSSH, no problem bringing up a server and connecting to it with my private key, for some reason couldn't get it to work with packer.. it started provision but for some reason started getting empty certificates from the keyvault, i checked the secret, it was indeed empty.. i will open an issue on it..

[cepefernando]

Another solution is to upload the files to s3 on the host running packer and then download them from the instance, you can attach an IAM Role to the Packer instance as well. It worked for me as I needed to copy around 80mb of artifacts to the AMI.

[pixelicous]

@cepefernando basically described what i was describing for aws instead of azure

[arizvisa]

If anyone wants to fix this performance issue, there's a post I made at the bottom of PR packer-community/winrmcp#6 that explains what the likely performance problem is. Unfortunately this isn't an issue that packer can solve because it depends on the packer-community/winrmcp library. So, the fix will have to be implemented and merged there before it can get sync'd into packer.

The issue with winrmcp is that for transferring chunks, they're doing an echo "content" >> $file for each chunk. That results in an fd being opened/closed per xfer of each chunk as well as using a command that is intended for writing output (which possibly parses the arguments for formatting and such) to the console instead of writing unprocessed content to a file. A possibly faster method would be to use Out-File -Append which likely doesn't process anything, or an even better method would be to keep the fd open while transferring chunks and using a lower-level .net class, System.Io.StreamWriter, to append to the target file.

I suggest creating an issue over on packer-community/winrmcp and then linking the reference here so that packer doesn't suffer from being stupidly slow when xferring data via winrmcp.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.