Terraform: segmentation fault with remote-exec + wget

[cmosetick]

terraform version:
0.6.14 (installed via brew)
running on:
Darwin 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64

I have a simple bash script that I want remote-exec to run on a Ubuntu 16.04 (devel) server:
e.g.

#!/bin/bash
wget https://www.mydomain.com/download/client

via:
script = "files/doit.sh"

with TF_LOG=TRACE set, terraform seems to indicate that it's choked on the wget portion of the bash script.

From the trace:

...
aws_instance.Cmosetick-Nano (remote-exec): Saving to: 'client'

aws_instance.Cmosetick-Nano (remote-exec): /tmp/terraform_1298498081.sh: line 4: 
1473 Segmentation fault (core dumped) wget https://www.mydomain.com/download/client
2016/04/20 16:57:44 [DEBUG] terraform-provisioner-remote-exec: 2016/04/20 16:57:44 
remote command exited with '1': /tmp/terraform_1298498081.sh
...

where mydomain.com is an actual domain with a file that any wget can actually retrieve.

Please let me know where I can retrieve the core dump from.

[cmosetick]

Note that switching from wget to curl seems to work, and gets past the segfault in my project.

[apparentlymart]

Hi, @cmosetick! This looks hairy...

Given that it works when you switch to curl, I suspect the segmentation fault is actually coming from wget itself on the remote server. In that case, the core dump would be wherever that remote host is configured to write core dumps; this is not something Terraform is controlling or affecting.

Have you tried manually installing your script on the target server and running it, so see if the segmentation fault can be reproduced without Terraform in the mix?

[tyrostone]

@cmosetick I believe I was seeing a similar issue, and it was due to how wget prints output to the console. From investigation, it appeared that it logged not only to stdout but also stderr, which was confusing Terraform and causing the segfault. Adding the --quiet flag to wget resolved this.

[cmosetick]

@apparentlymart
Yes, I had already manually tried the whole process on the remote system with wget before I filed my report here.

@tyrostone
That is a good call on --quiet flag. I never got around to trying that out, just switched to curl.

I still felt that this was report/bug worthy because without the TRACE output turned on, this would probably "silently fail", and users would be unlikely to guess right off the bat why things were not working.

Also IMO, I don't think that wget's stdout and stderr default mechanisms combined with Terraform should cause anything to segfault. wget is one of the most commonly used core-utils. Maybe I should file a pull request to add something to the docs about this. Perhaps something in the remote-exec examples section would help.

[jen20]

Hi @cmosetick! As @apparentlymart says this segfault is happening on the remote server - so the location of the core file will depend on the configuration of the remote server. I don't believe we do anything particularly unusual with the standard input, output or error streams. Are you sure this fails silently without TRACE turned on? It would seem to me that a core dump should result in a provisioner failure and a tainted resource - if that isn't the case then we should probably consider this a bug.

[byrnedo]

I was seeing segfault with wget too. Apparently not just happening with terraform execs: http://ubuntuforums.org/showthread.php?t=2325265

[mitchellh]

Closing since not a TF issue.

[OJFord]

I appreciate this isn't Terraform's fault, but can anyone shed any light on _why_ it's an issue?

I can run the same script with/out an enclosing exec fine (no --quiet) - it's only an issue when run as a provisioner.

[bbriggs]

I'm curious too. I just had to send a patch to the Terraform examples on another project to work around this, but would love to know the root cause.

[jgrammen-agilitypr]

This is still occuring with terraform v0.10.8 and aws 1.2.

snippet of code, without the --quiet this fails with a seg fault

provisioner "remote-exec" {
                inline = [
                        "wget --quiet -O /tmp/puppetlabs-release-pc1-$(lsb_release -sc).deb https://apt.puppetlabs.com/puppetlabs-release-pc1-$(lsb_release -sc).deb",

aws_instance.master (remote-exec): Connected!
aws_instance.master (remote-exec): --2017-11-01 15:15:56--  https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb
aws_instance.master (remote-exec): Resolving apt.puppetlabs.com (apt.puppetlabs.com)...
aws_instance.master (remote-exec): 52.84.35.5, 52.84.35.206, 52.84.35.204, ...
aws_instance.master (remote-exec): Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.84.35.5|:443... connected.
aws_instance.master (remote-exec): HTTP request sent, awaiting response... 200 OK
aws_instance.master (remote-exec): Length: 13662 (13K) [application/x-debian-package]
aws_instance.master (remote-exec): Saving to: ‘/tmp/puppetlabs-release-pc1-xenial.deb’

aws_instance.master (remote-exec): Segmentation fault (core dumped)

If I login to the vm and run the temporary file that contains my remote exec commands (the script file terraform auto creates), it runs fine.

So something about the way terraform interacts with a wget is causing issues. It may not be a terraform issue but it certainly feels like it.

[jgrammen-agilitypr]

some more details, I added an addition line of "sleep 60", just to see if the wget was running too early in the boot process. but it still segfaults.

tracelog of terraform

aws_instance.master : Still creating... (2m0s elapsed)
2017/11/01 12:01:34 [TRACE] dag/walk: vertex "provisioner.remote-exec (close)", waiting for: "aws_instance.master "
2017/11/01 12:01:34 [TRACE] dag/walk: vertex "root", waiting for: "provisioner.remote-exec (close)"
2017/11/01 12:01:34 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.master "
2017/11/01 12:01:34 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.master "
2017/11/01 12:01:39 [TRACE] dag/walk: vertex "provisioner.remote-exec (close)", waiting for: "aws_instance.master "
2017/11/01 12:01:39 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.master "
2017/11/01 12:01:39 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.master "
2017/11/01 12:01:39 [TRACE] dag/walk: vertex "root", waiting for: "provisioner.remote-exec (close)"
aws_instance.master : Still creating... (2m10s elapsed)
aws_instance.master (remote-exec): --2017-11-01 16:01:49--  https://www.google.com/
aws_instance.master (remote-exec): Resolving www.google.com (www.google.com)...
aws_instance.master (remote-exec): 172.217.2.100, 2607:f8b0:400b:808::2004
aws_instance.master (remote-exec): Connecting to www.google.com (www.google.com)|172.217.2.100|:443... connected.
aws_instance.master (remote-exec): HTTP request sent, awaiting response...
aws_instance.master (remote-exec): 200 OK
aws_instance.master (remote-exec): Length: unspecified [text/html]
aws_instance.master (remote-exec): Saving to: ‘/tmp/file.log’

aws_instance.master (remote-exec): /tmp/file.l            0  --.-KB/s
aws_instance.master (remote-exec): Segmentation fault (core dumped)

snippet from instance /var/log/syslog

Nov  1 16:01:03 ip-10-3-10-147 systemd[1]: Started Execute cloud user/final scripts.
Nov  1 16:01:03 ip-10-3-10-147 systemd[1]: Reached target Cloud-init target.
Nov  1 16:01:03 ip-10-3-10-147 systemd[1]: Startup finished in 5.213s (kernel) + 1min 4.433s (userspace) = 1min 9.647s.
Nov  1 16:01:11 ip-10-3-10-147 systemd-timesyncd[527]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Nov  1 16:01:49 ip-10-3-10-147 kernel: [  115.983569] wget[1447]: segfault at 558150f6b000 ip 00007f8439fbdf08 sp 00007fffddd38658 error 6 in libc-2.23.so[7f8439e4c000+1bf000]

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.