Terragrunt: fatal error: unexpected signal during runtime execution

Created on 11 Oct 2016  路  27Comments  路  Source: gruntwork-io/terragrunt

Hi guys,

It's me again. Seems that #36 was fixed, because I was able to use 0.1.1 for several days without any issues, but today during "terragrunt plan" it crashed:

https://gist.github.com/antonbabenko/9eae6e5ded37633307d78894e1b6c833

Rerun fixes the problem and I continue.

Any ideas ?

bug help wanted

Most helpful comment

I posted a question on the go-nuts mailing list asking for help: https://groups.google.com/d/msg/golang-nuts/4OJz3jprF84/c8BdI0bkCwAJ

All 27 comments

Wow, this one is even weirder!

The stack trace is from this line in Terragrunt:

bytes, err := ioutil.ReadFile(configPath)

The docs for ReadFile don't mention any possibility of a panic. Digging further into the stack trace, it turns out the error is in the call to make in the readAll method of ioutil.go:

buf := bytes.NewBuffer(make([]byte, 0, capacity))

In other words, it's failing to allocate memory. Is it possible you were out of memory on your computer?

Other than that, it just seems like a bug in Go itself...

It is rather unlikely that my mac was out of memory that moment, but I will keep using it and see if problem persists.

@antonbabenko I did notice you had a /home/ubuntu path, but you say you're on a mac. Are your running terragrunt inside a container? Or in a VM?

Yeah, this is strange. I run it without container, no VM. Just downloaded release and placed it as /usr/local/bin/terragrunt

New fatal - https://gist.github.com/antonbabenko/4b1dff65695fb73065f090c71929bfc3

I am not sure if it is related, but last time I was running "apply" command 12 hours ago and it worked, but today it didn't work from the first attempt, but rerun worked as expected.

Also, does anyone know why there is /home/ubuntu in the error? I always use Mac OS.

We cross-compile all the binaries on CircleCI. That /home/ubuntu path most likely gets baked in from the CircleCI servers, somehow. I'm guessing it's just a red herring: i.e. those paths are baked into the binary to make reading stack traces easier, but the paths themselves aren't actually used, since you're not running the code from those source files anyway.

I'm having a similar problem actually. https://gist.github.com/cultavix/bcbda8d985e1618397df0678b2430757

I am using Terragrunt v0.1.3

The problem seems to go away if I just keep running it. Any idea where this could be coming from?

Hm, I'm stumped. What OS are you on?

Ok, now I've gotten this, too. Here's my output:

josh@Joshs-MacBook-Pro /r/g/d/i/m/vpc> terragrunt apply
[terragrunt] 2016/11/04 15:50:33 Remote state is already configured for backend s3
[terragrunt] 2016/11/04 15:50:33 Attempting to acquire lock for state file vpc-mgmt in DynamoDB
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0xef1b]

runtime stack:
runtime.throw(0x72dfe0, 0x2a)
    /usr/local/go/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.unlock(0x9b1560)
    /usr/local/go/src/runtime/lock_sema.go:107 +0x14b
runtime.(*mheap).alloc_m(0x9b1560, 0x20, 0x100000000, 0xb85208)
    /usr/local/go/src/runtime/mheap.go:492 +0x314
runtime.(*mheap).alloc.func1()
    /usr/local/go/src/runtime/mheap.go:502 +0x41
runtime.systemstack(0xc82005fee0)
    /usr/local/go/src/runtime/asm_amd64.s:307 +0xab
runtime.(*mheap).alloc(0x9b1560, 0x20, 0x10100000000, 0x20)
    /usr/local/go/src/runtime/mheap.go:503 +0x63
runtime.largeAlloc(0x3fe00, 0xc800000001, 0xc82002d4e8)
    /usr/local/go/src/runtime/malloc.go:766 +0xb3
runtime.mallocgc.func3()
    /usr/local/go/src/runtime/malloc.go:664 +0x33
runtime.systemstack(0xc82001b500)
    /usr/local/go/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1051

goroutine 12 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:245 fp=0xc82002d380 sp=0xc82002d378
runtime.mallocgc(0x3fe00, 0x497d40, 0xc800000001, 0x3e00)
    /usr/local/go/src/runtime/malloc.go:665 +0x9eb fp=0xc82002d458 sp=0xc82002d380
runtime.newarray(0x497d40, 0x3fe00, 0xc82002d4f8)
    /usr/local/go/src/runtime/malloc.go:798 +0xc9 fp=0xc82002d498 sp=0xc82002d458
runtime.makeslice(0x486ae0, 0x3fe00, 0x3fe00, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/slice.go:32 +0x165 fp=0xc82002d4e8 sp=0xc82002d498
bytes.makeSlice(0x3fe00, 0x0, 0x0, 0x0)
    /usr/local/go/src/bytes/buffer.go:198 +0x64 fp=0xc82002d538 sp=0xc82002d4e8
bytes.(*Buffer).ReadFrom(0xc82010b030, 0xb09b60, 0xc82002e1f0, 0x1fe00, 0x0, 0x0)
    /usr/local/go/src/bytes/buffer.go:170 +0xd3 fp=0xc82002d5e8 sp=0xc82002d538
io.copyBuffer(0xb0b588, 0xc82010b030, 0xb09b60, 0xc82002e1f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:374 +0x180 fp=0xc82002d6a8 sp=0xc82002d5e8
io.Copy(0xb0b588, 0xc82010b030, 0xb09b60, 0xc82002e1f0, 0x34e54c, 0x0, 0x0)
    /usr/local/go/src/io/io.go:350 +0x64 fp=0xc82002d700 sp=0xc82002d6a8
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
    /usr/local/go/src/os/exec/exec.go:236 +0x8b fp=0xc82002d780 sp=0xc82002d700
os/exec.(*Cmd).Start.func1(0xc82014f400, 0xc820199160)
    /usr/local/go/src/os/exec/exec.go:344 +0x1d fp=0xc82002d7b0 sp=0xc82002d780
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82002d7b8 sp=0xc82002d7b0
created by os/exec.(*Cmd).Start
    /usr/local/go/src/os/exec/exec.go:345 +0x967

goroutine 1 [select]:
net/http.(*Transport).getConn(0xc8200cc000, 0xc8200ce380, 0x0, 0xc820134e10, 0x5, 0xc8201359e0, 0x24, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:711 +0x4ef
net/http.(*Transport).RoundTrip(0xc8200cc000, 0xc8200ce380, 0xc8200cc000, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:311 +0x7e9
net/http.send(0xc8200ce380, 0xb09528, 0xc8200cc000, 0x0, 0x0, 0x0, 0xc8201358f0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0x9abe00, 0xc8200ce380, 0x0, 0x0, 0x0, 0x6b, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0x9abe00, 0xc8200ce380, 0x780bf8, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0x9abe00, 0xc8200ce380, 0x6730e0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:191 +0x1e4
github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/aws/corehandlers.glob.func2(0xc820144000)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:73 +0x55
github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc820144168, 0xc820144000)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:136 +0xc3
github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc820144000, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:271 +0x560
github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/service/dynamodb.(*DynamoDB).DescribeTable(0xc82002e188, 0xc82002e190, 0x9c9480, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/aws/aws-sdk-go/service/dynamodb/api.go:438 +0x4f
github.com/gruntwork-io/terragrunt/dynamodb.lockTableExistsAndIsActive(0x6dd830, 0x10, 0xc82002e188, 0xc82008ac60, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/dynamodb/dynamo_lock_table.go:30 +0xb9
github.com/gruntwork-io/terragrunt/dynamodb.createLockTableIfNecessary(0x6dd830, 0x10, 0xc82002e188, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/dynamodb/dynamo_lock_table.go:15 +0x5c
github.com/gruntwork-io/terragrunt/dynamodb.DynamoDbLock.AcquireLock(0xc820017901, 0x8, 0x681fe0, 0x9, 0x6dd830, 0x10, 0x168, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/dynamodb/dynamo_lock.go:55 +0x192
github.com/gruntwork-io/terragrunt/dynamodb.(*DynamoDbLock).AcquireLock(0xc820011580, 0x0, 0x0)
    <autogenerated>:1 +0xa4
github.com/gruntwork-io/terragrunt/locks.WithLock(0xb09dd0, 0xc820011580, 0xc8201074c8, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/locks/lock.go:24 +0x55
github.com/gruntwork-io/terragrunt/cli.runTerraformCommandWithLock(0xc820012dc0, 0xb09dd0, 0xc820011580, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:153 +0x1a0
github.com/gruntwork-io/terragrunt/cli.runApp(0xc820012dc0, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:92 +0x28f
reflect.Value.call(0x4c97c0, 0x780608, 0x13, 0x6691f0, 0x4, 0xc820107b70, 0x1, 0x1, 0x0, 0x0, ...)
    /usr/local/go/src/reflect/value.go:435 +0x120d
reflect.Value.Call(0x4c97c0, 0x780608, 0x13, 0xc820107b70, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/reflect/value.go:303 +0xb1
github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli.HandleAction(0x4c97c0, 0x780608, 0xc820012dc0, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:480 +0x2ee
github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli.(*App).Run(0xc820000f00, 0xc82000a1e0, 0x2, 0x2, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:241 +0xb05
main.main()
    /home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/main.go:19 +0x8e

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [chan receive]:
net/http.(*Transport).dialConn(0xc8200cc000, 0x0, 0xc820134e10, 0x5, 0xc8201359e0, 0x24, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:834 +0x9c2
net/http.(*Transport).getConn.func4(0xc8200cc000, 0x0, 0xc820134e10, 0x5, 0xc8201359e0, 0x24, 0xc820069c20)
    /usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
    /usr/local/go/src/net/http/transport.go:708 +0x262

goroutine 11 [syscall]:
syscall.Syscall6(0x7, 0x3fc6, 0xc820102d04, 0x0, 0xc8200973b0, 0x0, 0x0, 0xc820102ce0, 0x51600, 0xc820102cd8)
    /usr/local/go/src/syscall/asm_darwin_amd64.s:41 +0x5
syscall.wait4(0x3fc6, 0xc820102d04, 0x0, 0xc8200973b0, 0x90, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:34 +0x7f
syscall.Wait4(0x3fc6, 0xc820102d4c, 0x0, 0xc8200973b0, 0xc82002e210, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_bsd.go:129 +0x55
os.(*Process).wait(0xc820199440, 0x1e, 0x0, 0x0)
    /usr/local/go/src/os/exec_unix.go:22 +0x105
os.(*Process).Wait(0xc820199440, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/doc.go:49 +0x2d
os/exec.(*Cmd).Wait(0xc82014f400, 0x0, 0x0)
    /usr/local/go/src/os/exec/exec.go:396 +0x211
os/exec.(*Cmd).Run(0xc82014f400, 0x0, 0x0)
    /usr/local/go/src/os/exec/exec.go:262 +0x64
os/exec.(*Cmd).Output(0xc82014f400, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/exec/exec.go:432 +0x26c
crypto/x509.execSecurityRoots(0x8e725, 0x0, 0x0)
    /usr/local/go/src/crypto/x509/root_darwin.go:17 +0xe9
crypto/x509.initSystemRoots()
    /usr/local/go/src/crypto/x509/root_nocgo_darwin.go:10 +0x18
sync.(*Once).Do(0x9c94f8, 0x780400)
    /usr/local/go/src/sync/once.go:44 +0xe4
crypto/x509.systemRootsPool(0x0)
    /usr/local/go/src/crypto/x509/root.go:15 +0x2f
crypto/x509.(*Certificate).Verify(0xc820132480, 0xc8201359e0, 0x20, 0xc82018a8d0, 0x0, 0xecfaf07b9, 0x2e88cf0b, 0x9abf60, 0x0, 0x0, ...)
    /usr/local/go/src/crypto/x509/verify.go:241 +0x1de
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc820103e48, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/handshake_client.go:278 +0x6e9
crypto/tls.(*Conn).clientHandshake(0xc82018c000, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/handshake_client.go:213 +0x1737
crypto/tls.(*Conn).Handshake(0xc82018c000, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1033 +0xec
net/http.(*Transport).dialConn.func3(0xc82018c000, 0xc820150240, 0xc820160060)
    /usr/local/go/src/net/http/transport.go:828 +0x21
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:833 +0x986

goroutine 13 [syscall]:
syscall.Syscall(0x3, 0x9, 0xc8201a4000, 0x8000, 0xb85000, 0x0, 0x1ea00000000)
    /usr/local/go/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.read(0x9, 0xc8201a4000, 0x8000, 0x8000, 0x5d7368, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:972 +0x5f
syscall.Read(0x9, 0xc8201a4000, 0x8000, 0x8000, 0xc8201a4000, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:161 +0x4d
os.(*File).read(0xc82002e208, 0xc8201a4000, 0x8000, 0x8000, 0x1, 0x0, 0x0)
    /usr/local/go/src/os/file_unix.go:228 +0x75
os.(*File).Read(0xc82002e208, 0xc8201a4000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /usr/local/go/src/os/file.go:95 +0x8a
io.copyBuffer(0xb0b5b0, 0xc820196780, 0xb09b60, 0xc82002e208, 0xc8201a4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:380 +0x247
io.Copy(0xb0b5b0, 0xc820196780, 0xb09b60, 0xc82002e208, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:350 +0x64
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
    /usr/local/go/src/os/exec/exec.go:236 +0x8b
os/exec.(*Cmd).Start.func1(0xc82014f400, 0xc8201991e0)
    /usr/local/go/src/os/exec/exec.go:344 +0x1d
created by os/exec.(*Cmd).Start
    /usr/local/go/src/os/exec/exec.go:345 +0x967

I do notice that it happens intermittently and that if I run terragrunt apply again, the issue is resolved. I'm still not sure why this is happening, though.

I'm getting this everytime I run terragrunt now, after upgrading to 0.3.0.

https://gist.github.com/cultavix/18a1e11abf6c6a52a00a33c02a504f64

OS: Linux (16.04 Xubuntu), Linux (14.10 Ubuntu Server) and macOS (latest)

Maybe it's the way we are using it?

@cultavix Does that happen for every Terragrunt command (e.g. terragrunt plan, terragrunt apply, etc)? Could you share your .terragrunt file? Is the terraform executable still in your PATH?

I just ran v0.3.0 on my own (OS X) computer and didn't get an issue...

@brikis98 ok, so it's a fairly detailed answer..

.terragrunt file:

 cat <<EOF > .terragrunt
lock = {
  backend = "dynamodb"
  config {
    state_file_id = "${TF_VAR_remote_states_bucket}_${terraform_state_key}"
    aws_region = "${TF_VAR_remote_states_region}"
    table_name = "terragrunt_locks"
  }
}

remote_state = {
  backend = "s3"
  config {
    encrypt = "true"
    bucket = "${TF_VAR_remote_states_bucket}"
    key = "${terraform_state_key}"
    region = "${TF_VAR_remote_states_region}"
  }
}
EOF

As for running it, it's embedded into a script which does a number of things to organise our terraform code/infrastructure. But essentially, like this:

terragrunt plan \ $refresh \ $destroy \ -input=false \ $detailed_exitcode \ $infrastructure_layer_var_file \ $region_infrastructure_layer_var_file \ $arguments \ $LAYER_WORK_DIR

@cultavix Terragrunt does not support interpolation (${...}) or env vars at the moment, so the .terragrunt file you have will not work. I'm working on adding basic support for interpolation now for #26.

I'm still not sure why you got such a weird error message though, so that's something we should fix.

@cultavix @brikis98 - All ${TF_VAR_*} are replaced, so that produced file .terragrunt does not have any interpolation.

Who replaces the TF_VAR values? Terraform does that for .tf files, but nothing is doing that for .terragrunt...

Yes, I know. We replace those when we create .terragrunt file ( @cultavix copied source code from shell script instead of produced file, which exists before calling terragrunt). Here is the complete file:

lock = {
  backend = "dynamodb"
  config {
    state_file_id = "tf-states-dev_us-west-2_layer"
    aws_region = "eu-west-1"
    table_name = "terragrunt_locks"
  }
}

remote_state = {
  backend = "s3"
  config {
    encrypt = "true"
    bucket = "tf-states-dev"
    key = "us-west-2_layer"
    region = "eu-west-1"
  }
}

Ah, I see. So if you take that static file, put it into /tmp/.terragrunt, and run terragrunt apply with v0.3.0, you get a crash every time?

It works fine, it just gives you that error :)

It works fine, it just gives you that error

Sorry, I'm still confused :)

So if you run terragrunt plan on that completely static file that @antonbabenko pasted above, the command succeeds (i.e. shows you a plan), but also shows you an error message?

Let me assume, it fails with an error message at first run and then it works on second run. Same as I have been experiencing and reporting before.

My assumption is based on the fact that I have been working on the same project with @cultavix and we talked about it multiple times.

PS: I am using hub, which is also written in Go (source repo) and there I have similar weird crashes once every 2-3 days. Maybe there is something wrong with my/our installation of Go and/or brew on the Mac? Not sure how to investigate even.

PPS: terragrunt as well as hub crash once in a while (every 1-3 days) and second rerun always work.

@antonbabenko It could be environmental or even an issue with Go itself. That said, @cultavix earlier said:

I'm getting this everytime I run terragrunt now, after upgrading to 0.3.0.

Which doesn't sound like the intermittent issue you're describing...

You are right, I have missed word "everytime".

I posted a question on the go-nuts mailing list asking for help: https://groups.google.com/d/msg/golang-nuts/4OJz3jprF84/c8BdI0bkCwAJ

OK, it looks like the most likely explanation for this issue is that our CI job was building binaries with Go 1.6.x, and apparently, those binaries will not work with the latest version of OS X. I'm not on the latest OS X yet, which explains why I haven't seen these crashes!

We just updated the CI job to use Go 1.7.3. I've cut a new release and the binaries for it should be there in a few minutes: https://github.com/gruntwork-io/terragrunt/releases/tag/v0.7.0. Please try it out and let me know if it works better for you!

Great! I will be able to try this in couple weeks only.

Thanks, I'll update/test tomorrow :)

Was this page helpful?
0 / 5 - 0 ratings

Related issues

dmlemos picture dmlemos  路  3Comments

dzirg44 picture dzirg44  路  3Comments

mmorianos picture mmorianos  路  3Comments

geekifier picture geekifier  路  3Comments

goseeped picture goseeped  路  3Comments