Packer: WinRM uploaded PowerShell fails on Win 2016 Base in AWS

[sc250024]

Greetings,

Packer builds are failing when I attempt to use the scripts block in a powershell provisioner with Windows Server 2016 base images in AWS with the following error:

2018/04/22 14:31:23 ui error: Build 'amazon-ebs' errored: Error processing command: Error uploading ps script containing env vars: Couldn't determine whether destination was a folder or file: unknown error Post https://<temp-aws-eip>:5986/wsman: read tcp 100.70.14.130:52062-><temp-aws-eip>:5986: read: connection reset by peer

As you can see in the Gist link below, I'm executing some basic inline scripts first (EC2Launch configs), and then some simple Write-Host commands. I then attempt to run a Write-Host command from inside a test PowerShell script called test.ps1, which immediately fails.

    amazon-ebs:
2018/04/22 14:28:37 ui:     amazon-ebs:
2018/04/22 14:28:37 ui:     amazon-ebs: TaskPath                                       TaskName                          State
    amazon-ebs: TaskPath                                       TaskName                          State
2018/04/22 14:28:37 ui:     amazon-ebs: --------                                       --------                          -----
    amazon-ebs: --------                                       --------                          -----
2018/04/22 14:28:37 ui:     amazon-ebs: \                                              Amazon Ec2 Launch - Instance I... Ready
    amazon-ebs: \                                              Amazon Ec2 Launch - Instance I... Ready
    amazon-ebs: Some test output #1...
2018/04/22 14:30:52 ui:     amazon-ebs: Some test output #1...
2018/04/22 14:30:52 ui:     amazon-ebs: Some test output #2...
    amazon-ebs: Some test output #2...
    amazon-ebs: Some test output #3...
2018/04/22 14:30:52 ui:     amazon-ebs: Some test output #3...
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 [INFO] command 'powershell -executionpolicy bypass -file "%TEMP%/packer-elevated-shell-5adc7fe5-52d1-8dbd-cba2-4ae73f08077d.ps1"' exited with code: 0
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 [INFO] RPC endpoint: Communicator ended with: 0
2018/04/22 14:30:52 [INFO] 353 bytes written for 'stdout'
2018/04/22 14:30:52 [INFO] 0 bytes written for 'stderr'
2018/04/22 14:30:52 [INFO] RPC client: Communicator ended with: 0
2018/04/22 14:30:52 [INFO] RPC endpoint: Communicator ended with: 0
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 [INFO] 353 bytes written for 'stdout'
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 [INFO] 0 bytes written for 'stderr'
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 [INFO] RPC client: Communicator ended with: 0
2018/04/22 14:30:52 [INFO] (telemetry) ending powershell
2018/04/22 14:30:52 [INFO] (telemetry) Starting provisioner powershell
2018/04/22 14:30:52 ui: ==> amazon-ebs: Provisioning with Powershell...
==> amazon-ebs: Provisioning with Powershell...
2018/04/22 14:30:52 ui: ==> amazon-ebs: Provisioning with powershell script: /Users/myuser/Documents/git-projects/mycompany/mycompany-packer/provisioners/powershell/test.ps1
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 Opening /Users/myuser/Documents/git-projects/mycompany/mycompany-packer/provisioners/powershell/test.ps1 for reading
==> amazon-ebs: Provisioning with powershell script: /Users/myuser/Documents/git-projects/mycompany/mycompany-packer/provisioners/powershell/test.ps1
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 Uploading env vars to ${env:SYSTEMROOT}/Temp/packer-env-vars-5adc807c-b3f9-4afe-d77d-5c55eb5c066c.ps1
2018/04/22 14:30:52 packer: 2018/04/22 14:30:52 [INFO] 76 bytes written for 'uploadData'
2018/04/22 14:30:52 [INFO] 76 bytes written for 'uploadData'
2018/04/22 14:30:53 [INFO] (telemetry) ending powershell
2018/04/22 14:30:53 ui: ==> amazon-ebs: Cancelling the spot request...
==> amazon-ebs: Cancelling the spot request...
2018/04/22 14:30:57 packer: 2018/04/22 14:30:57 Waiting for state to become: cancelled
2018/04/22 14:30:57 packer: 2018/04/22 14:30:57 Using 2s as polling delay (change with AWS_POLL_DELAY_SECONDS)
2018/04/22 14:30:57 packer: 2018/04/22 14:30:57 Allowing 300s to complete (change with AWS_TIMEOUT_SECONDS)
2018/04/22 14:30:57 ui: ==> amazon-ebs: Terminating the source AWS instance...

Packer version: 1.2.2
Host platform: AWS
Github gist: https://gist.github.com/sc250024/15eb5bbb7419ccb31f5bb694f26398b8

[DanHam]

@sc250024 Hi Scott. I've taken a quick look at this. As a first guess , are you sure that sysprep isn't tearing down the WinRM connection and causing the read: connection reset by peer error?

I noticed the unknown error Post https://<temp-aws-eip>:5986/wsman: read tcp 100.70.14.130:52062-><temp-aws-eip>:5986 is seen when Packer first establishes the WinRM connection.

2018/04/22 14:27:31 packer: 2018/04/22 14:27:31 [INFO] Attempting WinRM connection...
2018/04/22 14:27:31 packer: 2018/04/22 14:27:31 [DEBUG] connecting to remote shell using WinRM
2018/04/22 14:28:01 packer: 2018/04/22 14:28:01 [ERROR] connection error: unknown error Post https://<temp-aws-external-ip>:5986/wsman: dial tcp <temp-aws-external-ip>:5986: i/o timeout
2018/04/22 14:28:01 packer: 2018/04/22 14:28:01 [ERROR] WinRM connection err: unknown error Post https://<temp-aws-external-ip>:5986/wsman: dial tcp <temp-aws-external-ip>:5986: i/o timeout
2018/04/22 14:28:06 packer: 2018/04/22 14:28:06 [INFO] Attempting WinRM connection...
2018/04/22 14:28:06 packer: 2018/04/22 14:28:06 [DEBUG] connecting to remote shell using WinRM
2018/04/22 14:28:12 packer: 2018/04/22 14:28:12 Checking that WinRM is connected with: 'powershell.exe -EncodedCommand aQBmACAAKABUAGUAcwB0AC0AUABhAHQAaAAgAHYAYQByAGkAYQBiAGwAZQA6AGcAbABvAGIAYQBsADoAUAByAG8AZwByAGUAcwBzAFAAcgBlAGYAZQByAGUAbgBjAGUAKQB7ACQAUAByAG8AZwByAGUAcwBzAFAAcgBlAGYAZQByAGUAbgBjAGUAPQAnAFMAaQBsAGUAbgB0AGwAeQBDAG8AbgB0AGkAbgB1AGUAJwB9ADsAIABlAGMAaABvACAAIgBXAGkAbgBSAE0AIABjAG8AbgBuAGUAYwB0AGUAZAAuACIA'
2018/04/22 14:28:13 packer: 2018/04/22 14:28:13 [INFO] starting remote command: powershell.exe -EncodedCommand aQBmACAAKABUAGUAcwB0AC0AUABhAHQAaAAgAHYAYQByAGkAYQBiAGwAZQA6AGcAbABvAGIAYQBsADoAUAByAG8AZwByAGUAcwBzAFAAcgBlAGYAZQByAGUAbgBjAGUAKQB7ACQAUAByAG8AZwByAGUAcwBzAFAAcgBlAGYAZQByAGUAbgBjAGUAPQAnAFMAaQBsAGUAbgB0AGwAeQBDAG8AbgB0AGkAbgB1AGUAJwB9ADsAIABlAGMAaABvACAAIgBXAGkAbgBSAE0AIABjAG8AbgBuAGUAYwB0AGUAZAAuACIA
2018/04/22 14:28:14 packer: 2018/04/22 14:28:14 [INFO] command 'powershell.exe -EncodedCommand aQBmACAAKABUAGUAcwB0AC0AUABhAHQAaAAgAHYAYQByAGkAYQBiAGwAZQA6AGcAbABvAGIAYQBsADoAUAByAG8AZwByAGUAcwBzAFAAcgBlAGYAZQByAGUAbgBjAGUAKQB7ACQAUAByAG8AZwByAGUAcwBzAFAAcgBlAGYAZQByAGUAbgBjAGUAPQAnAFMAaQBsAGUAbgB0AGwAeQBDAG8AbgB0AGkAbgB1AGUAJwB9ADsAIABlAGMAaABvACAAIgBXAGkAbgBSAE0AIABjAG8AbgBuAGUAYwB0AGUAZAAuACIA' exited with code: 0
2018/04/22 14:28:14 ui: amazon-ebs: WinRM connected.

Not sure what's going on there (maybe you do?) but, for the time being at least, I'm calling that out as a bit of a red herring in the fatal error you are getting, as all of the prior commands seemingly works OK despite of it...

Can you try and run the template again but omit the sysprep command to see if it runs OK? e.g. take out the following command from the inline provisioner in your template:

"C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown",

Also noticed you were running winrm quickconfig -q to set up your WinRM connection. I'm certain this is unrelated to the issues you are seeing, but setting WinRM up this way can cause you issues - see the A quick aside/warning section of the docs HERE for the full explanation.

[christrotter]

I'm running into the same issue, and using the same sysprep scripts.

 "provisioners": [
  {
    "type": "powershell",
    "script": "./scripts/run-buildscripts.ps1"
  },
  {
    "type": "windows-restart"
  },
  {
    "type": "powershell",
    "inline": [
      "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule",
      "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown"
    ]
  }
]

Everything runs fine up until this point, then I get:

==> amazon-ebs: Provisioning with powershell script: /var/folders/0t/n0rb00jj2gxgvdbhr2znx5ph0000gp/T/packer-powershell-provisioner696022292
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
Build 'amazon-ebs' errored: Error processing command: Error uploading ps script containing env vars: Couldn't determine whether destination was a folder or file: unknown error Post http://EC2.IP.AD.DR:5985/wsman: read tcp MY.IP.AD.DR:63667->EC2.IP.AD.DR:5985: read: connection reset by peer

I used the WinRM configuration as outlined in that doc you linked above, but with this addition (since it would never get past '_waiting for winrm_'.

set-item WSMan:\localhost\Client\AllowUnencrypted -Value True -Force
set-item WSMan:\localhost\Client\Auth\Basic -Value True -Force
set-item WSMan:\localhost\Client\TrustedHosts -Value * -Force
Enable-PSRemoting -force

FWIW, this used to run fine until I separated out my build scripts from inside user-data (our old process) to a separate powershell script provisioner.

Finally, we had the 'run-buildscripts' script crash out silently (_msi failed to download or something_), and magically the process finished up correctly, AMI exported successfully. So something in that script is blowing things up - _not_ the EC2 Launch sysprep scripts.

So for others with this issue, it's probably something in your own scripts causing this. I still haven't found explicitly what is the source, but feel pretty certain it's our script, not AWS's, nor a Packer bug. (_although the output is pretty vague_ :) )

[christrotter]

As a follow-up, I made the following changes, and everything is working again:

• add logging output
• tried again

lol. So definitely something in our script being silly. No idea why, as of yet. I will update if I find out.

[SwampDragons]

@sc250024 have you had a chance to try Dan's suggestions?

[sc250024]

@SwampDragons I will take a look tomorrow, and post the result. I think the suggestion is correct, but I will for sure verify it.

[DanHam]

Error processing command: Error uploading ps script containing env vars: Couldn't determine whether destination was a folder or file: unknown error

@christrotter @sc250024 I just noticed - that's the same error message users have reported over in #6240. I wonder if both of you are running into the same issue? Like you they are only seeing this issue some of the time...

This issue is caused by a bug (race condition) in an upstream package. #6261 is an attempt to 'fix' this. Unfortunately, there hasn't been anyone willing to test this yet. Would either of you be willing to give it a try and report back?

[DanHam]

@SwampDragons I think this may be a duplicate of #6240 which might be fixed by #6261...

Looks like @sc250024 is on Windows so I wondered if you could do the honours and provide a binary with the possible fix?

[sc250024]

@DanHam Per your original reply, removing the line...

"C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown"

...did in fact cause the build to succeed afterwards. I have created a gist of that SysprepInstance.ps1 script here: https://gist.github.com/sc250024/5d4539e31b982787bab919e1ac7d6162. I don't see anywhere where it might kill a WinRM connection, but I _did_ try running that script directly from an RDP session, and running it in fact does kill the connection.

In general, it seems that one needs to run that script at the end of building a Windows 2016 image using Packer on AWS since it seems to kill the connection regardless of how it's run.

[sc250024]

@DanHam For what it's worth, I am encountering the same issue as https://github.com/hashicorp/packer/issues/6240 now. See the new gist of the debug output below:

https://gist.github.com/sc250024/e6047427724eabf00c960445f415fcca

And it seems to be related now to the file provisioner:
Packer 1.2.3 on macOS 10.13.4

[DanHam]

I don't see anywhere where it might kill a WinRM connection, but I did try running that script directly from an RDP session, and running it in fact does kill the connection.

That script looks to be a wrapper (with some Amazon specific bits thrown in) around the sysprep utility itself. I agree that there's nothing in the script itself that's obviously causing the disconnect; this tends to suggest sysprep itself is responsible. At a guess, it could be that sysprep is tearing down the WinRM connection and settings directly, or, given that it tore down your RDP connection as well, it could be that sysprep is resetting the Windows Firewall and causing the disconnect that way.

The option documented HERE may help out with running that sysprep script in a way that works well with Packer.

I recently saw that jen20 used that script in preparing his Windows instances. If you want to check it out see HERE and HERE for full details - it's really rather neat! That build uses SSH rather than WinRM and configures a custom firewall rule to allow the connection - so I wonder if a custom firewall rule doesn't get wiped out by the sysprep step in the same way as perhaps the inbuilt WinRM, Windows Remote, and RDP ones do.

I am encountering the same issue as #6240 now.

Dammit! One step forward... I do sympathise! However, I've personally been unable to reproduce the error reported in #6240 using my (admittedly very simple) packer-testing templates HERE.

Have you been able to test the possible fix for #6240 by building yourself a binary with the changes from #6261 applied?

[DanHam]

Looks like @sc250024 is on Windows so I wondered if you could do the honours and provide a binary with the possible fix?

@SwampDragons Sorry, that was wrong... got confused. @sc250024 is using OS X... Could you provide a binary for Mac over in #6240 to see if that helps out?

[sc250024]

@DanHam I will try your suggestions, and get back on this post.

[DanHam]

@sc250024 Great! So #6261 has now been merged so you should be good just building off master.

[Justin-DynamicD]

Would just like to voice that I ran into this same issue as described above, and the winrm race condition fix in 1.2.4 has solved this scenario for me. Specifically came off build 1.2.1 onto 1.2.4.

[hc0105]

I had same problem too using packer version 1.3.3
I tried like below in provisioner..

...

1. sysprep-ec2config.ps1
2. windows-restart
3. another powershell (script that needs restart..)

I solved problem with above help of above comment. like this

...

1. windows-restart
2. another powershell (script that needs restart..)
3. sysprep-ec2config.ps1

[bumblebee939]

@DanHam @sc250024
So how do one get past the sysprep step without killing the winrm connection.
I am running into that issue as of today ?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.