Packer: Script disconnected unexpectedly while long curl

[geostant]

Seems like long curl (about 2 minutes)
disconnect my session in the middle.

That isn't consist as sometime it works and sometimes not...

Packer version : 1.3.3
Platform : AWS EBS (Ubuntu)

Debug log:
https://gist.github.com/geostant/16f8ade96ecbdecc0a30e99df6ad7135

[geostant]

I tried today to use wget, got the same issue

[umerebryx]

Any update on this? I am also facing the same issue if a command runs for longer time packer build process is killed.

[geostant]

@umerebryx
I gave up on this a long time ago...

[azr]

Hello there; the original gist posted is a 404. But I think this issue could be due to the fact that the communicator will hang up after a while if nothing happens.

[geostant]

@azr In my case the curl was against docker domain (getting the installation file)
I think it's responsive enough

[azr]

Do you have the configuration to reproduce it ?

[umerebryx]

In my case it abruptly fails while the script is running and there is continuous output. Here is where my build stops:

amazon-ebs: zlib 1.0.0
amazon-ebs: installing bundled gems: /usr/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)
amazon-ebs: test-unit 3.2.7
amazon-ebs: power_assert 1.1.1
amazon-ebs: minitest 5.10.3
amazon-ebs: xmlrpc 0.3.0
amazon-ebs: rake 12.3.0
amazon-ebs: net-telnet 0.1.1
amazon-ebs: did_you_mean 1.2.0
amazon-ebs: installing rdoc: /usr/share/ri/2.5.0/system
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [ERROR] Remote command exited without exit status or exit signal.
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] RPC endpoint: Communicator ended with: 2300218
2019/09/11 12:13:15 [INFO] 100078 bytes written for 'stdout'
2019/09/11 12:13:15 [INFO] 15982 bytes written for 'stderr'
2019/09/11 12:13:15 [INFO] RPC client: Communicator ended with: 2300218
2019/09/11 12:13:15 [INFO] RPC endpoint: Communicator ended with: 2300218
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] 15982 bytes written for 'stderr'
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] 100078 bytes written for 'stdout'
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] RPC client: Communicator ended with: 2300218
2019/09/11 12:13:15 [INFO] (telemetry) ending shell
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
2019/09/11 12:13:57 [INFO] (telemetry) ending amazon-ebs
2019/09/11 12:13:57 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2019/09/11 12:13:57 machine readable: amazon-ebs,error []string{"Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding \"expect_disconnect\": true or \"valid_exit_codes\": [0, 2300218] to the shell provisioner parameters."}
==> Builds finished but no artifacts were created.
2019/09/11 12:13:57 [INFO] (telemetry) Finalizing.

Another thing I observed is the script always fails at 6 mins and ~47 sec which make me think if there is a global timeout set somewhere that is causing this behaviour?

My packer version is 1.4.3 and running it on Mac.

[azr]

Do you have easy to reproduce steps ?

It looks like to me that the command crashed:

amazon-ebs: installing rdoc: /usr/share/ri/2.5.0/system
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [ERROR] Remote command exited without exit status or exit signal.

OOM ?

[umerebryx]

I will setup a temp repo for you to reproduce this. I have done another test by running same build script from an ec2 instance c4.2xlarge and this time everything completed without any issues. I will double check my local setup to see this is not a memory issue as you mentioned (but shouldn't OOM through a segmentation fault locally?).

[anthonymag]

This is happening for me every other build or so, it's pretty inconsistent.
Always seems to happen while installing software-properties-common
Last three failures:

    amazon-ebs: + apt-get install software-properties-common -y
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
    amazon-ebs: The following additional packages will be installed:
    amazon-ebs:   python3-software-properties
    amazon-ebs: The following packages will be upgraded:
    amazon-ebs:   python3-software-properties software-properties-common
    amazon-ebs: 2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    amazon-ebs: Need to get 29.6 kB of archives.
    amazon-ebs: After this operation, 0 B of additional disk space will be used.
    amazon-ebs: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.9 [9,452 B]
    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.9 [20.1 kB]
    amazon-ebs: Fetched 29.6 kB in 0s (2,446 kB/s)
    amazon-ebs: (Reading database ... 87051 files and directories currently installed.)
    amazon-ebs: Preparing to unpack .../software-properties-common_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking software-properties-common (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Preparing to unpack .../python3-software-properties_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking python3-software-properties (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Processing triggers for man-db (2.7.5-1) ...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
Build 'amazon-ebs' errored: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.

==> Builds finished but no artifacts were created.
Build step 'Execute shell' marked build as failure
Finished: FAILURE

    amazon-ebs: + DEBIAN_FRONTEND=noninteractive
    amazon-ebs: + apt-get install software-properties-common -y
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
    amazon-ebs: The following additional packages will be installed:
    amazon-ebs:   python3-software-properties
    amazon-ebs: The following packages will be upgraded:
    amazon-ebs:   python3-software-properties software-properties-common
    amazon-ebs: 2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    amazon-ebs: Need to get 29.6 kB of archives.
    amazon-ebs: After this operation, 0 B of additional disk space will be used.
    amazon-ebs: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.9 [9,452 B]
    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.9 [20.1 kB]
    amazon-ebs: Fetched 29.6 kB in 0s (1,911 kB/s)
    amazon-ebs: (Reading database ... 87051 files and directories currently installed.)
    amazon-ebs: Preparing to unpack .../software-properties-common_0.96.20.9_all.deb ...
==> amazon-ebs: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.
==> amazon-ebs: Step "StepProvision" failed, aborting...
Build 'amazon-ebs' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: unexpected EOF

==> Builds finished but no artifacts were created.
Build step 'Execute shell' marked build as failure
Finished: FAILURE

    amazon-ebs: + export DEBIAN_FRONTEND=noninteractive
    amazon-ebs: + DEBIAN_FRONTEND=noninteractive
    amazon-ebs: + apt-get install software-properties-common -y
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
    amazon-ebs: The following additional packages will be installed:
    amazon-ebs:   python3-software-properties
    amazon-ebs: The following packages will be upgraded:
    amazon-ebs:   python3-software-properties software-properties-common
    amazon-ebs: 2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    amazon-ebs: Need to get 29.6 kB of archives.
    amazon-ebs: After this operation, 0 B of additional disk space will be used.
    amazon-ebs: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.9 [9,452 B]
    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.9 [20.1 kB]
    amazon-ebs: Fetched 29.6 kB in 0s (2,694 kB/s)
    amazon-ebs: (Reading database ... 87051 files and directories currently installed.)
    amazon-ebs: Preparing to unpack .../software-properties-common_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking software-properties-common (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Preparing to unpack .../python3-software-properties_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking python3-software-properties (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Processing triggers for man-db (2.7.5-1) ...
    amazon-ebs: Processing triggers for dbus (1.10.6-1ubuntu3.4) ...
    amazon-ebs: Setting up python3-software-properties (0.96.20.9) ...
    amazon-ebs: Setting up software-properties-common (0.96.20.9) ...
==> amazon-ebs: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.
==> amazon-ebs: Step "StepProvision" failed, aborting...
Build 'amazon-ebs' errored: unexpected EOF

[umerebryx]

@anthonymag , for me the issue was fix after I freed up some memory. The worst part is packer doesn't give any indication the termination is because of OOM. You can try running by closing all your programs to see if the issue is RAM in your case. Hope this helps.

[anthonymag]

@umerebryx thanks for your response! I'm surprised as I'm giving the built instance 8GB, but I can increase this and try it out, unless you mean memory on the Packer host that is remotely controlling the build?

[umerebryx]

You have to free up memory on the host that is running the packer build command and not the remote host.

[geostant]

An interesting twist in the story...
I would love to know that all the issue was memory related - please keep us updated if that solves that issue, thanks.

[anthonymag]

I'm currently running the build directly from a Jenkins host, and will change it to run from a Jenkins worker/slave to have its own dedicated resources and will let you guys know if that fixes the issue. It might have failed from OOM whenever it was running alongside another build.

[anthonymag]

No dice. I gave it its own dedicated host with 8GB and I still get the same failure every other build or so...