Packer: Using windows-restart with session-manager-plugin leads to timeout and 100% cpu load

[pascal-hofmann]

Overview of the Issue

Using windows-restart with session-manager-plugin does not work.

Restarting an EC2 instance will always terminate the session manager session.

session-manager-plugin: Exiting session with sessionId: …

packer does not handle this gracefully. It should try to recreate the SSM session.

Instead it consumes 100% cpu because it keeps looping in: https://github.com/hashicorp/packer/blob/aa50b2185f338e685dbca141752071454f5d5918/builder/amazon/common/ssm_driver.go#L109-L122

This code should test whether stderrCh/stdoutCh have been closed by assigning a second parameter to the receive expression and terminate/and try to restart the ssm session in this case.

Reproduction Steps

• Configure the builder to use Session Manager Connections ("ssh_interface": "session_manager")
• Use the windows-restart provisioner:
  ``` "provisioners": [
  {
  "type": "windows-restart",
  "restart_command": "shutdown /r /t 60 /f /d p:4:1 /c \"packer restart\""
  }]

### Packer version

1.6.0

### Simplified Packer Buildfile

https://gist.github.com/pascal-hofmann/53e9f417e843384563f428dadcb3fd27

### Operating system and Environment details

macOS 10.15.5, packer installed via homebrew.

### Log Fragments and crash.log files

…
2020/06/17 09:16:27 packer-provisioner-windows-restart plugin: Reboot already in progress, waiting...
2020/06/17 09:16:37 packer-provisioner-windows-restart plugin: Check if machine is rebooting...
2020/06/17 09:16:37 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [DEBUG] session-manager-plugin:
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [DEBUG] session-manager-plugin:
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [DEBUG] session-manager-plugin: Exiting session with sessionId: ….
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [DEBUG] session-manager-plugin:
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: 'ssh: unexpected packet in response to channel open: ', attempting reconnect
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2020/06/17 09:17:13 packer-builder-amazon-ebs plugin: [ERROR] reconnection error: dial tcp [::1]:8095: connect: connection refused
…
2020/06/17 09:20:33 packer-provisioner-windows-restart plugin: [INFO] 0 bytes written for 'stdout'
2020/06/17 09:20:33 packer-provisioner-windows-restart plugin: [INFO] 0 bytes written for 'stderr'
2020/06/17 09:20:33 packer-provisioner-windows-restart plugin: echo didn't succeed; retrying...
2020/06/17 09:20:37 ui error: ==> amazon-ebs: Timeout waiting for machine to restart.
2020/06/17 09:20:37 [INFO] (telemetry) ending windows-restart
2020/06/17 09:20:37 ui: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2020/06/17 09:20:37 ui: ==> amazon-ebs: Terminating the source AWS instance...
…
```

[nywilken]

Hi @pascal-hofmann thanks for reaching out. I will take a look at getting this fixed for the next release.

[nywilken]

Hi there :wave: we've opened PR #10003 with a fix that will retry to establish a new session after an instance restart or session termination. Would you mind testing with the latest binaries https://app.circleci.com/pipelines/github/hashicorp/packer/7748/workflows/be705362-505a-4834-9988-fcb1b07f724c/jobs/89316/artifacts and letting us know if the patched binaries resolve your issue?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.