Delve: Can't debug github.com/hashicorp/terraform

[sodre]

1. What version of Delve are you using (dlv version)?
   Delve Debugger
   Version: 0.11.0-alpha
   Build: 26ce16f1db3fdd6fdd915f2ec7d11e4bfe72baed
2. What version of Go are you using? (go version)?
   go version go1.6 darwin/amd64
3. What operating system and processor architecture are you using?
   OS X
4. What did you do?
   I tried to debug hashicorp/terraform application.

$ dlv debug github.com/hashicorp/terraform -- apply
Type 'help' for list of commands.
(dlv) c

1. What did you expect to see?
   I expected to debugger to be able to find the other plugins that Terraform depends.
2. What did you see instead?

Error configuring: 3 error(s) occurred:

* triton_firewall_rule.inet-to-bastion: provider triton couldn't be found
* triton_firewall_rule.bastion-to-vms: provider triton couldn't be found
* triton_machine.bastion: provider triton couldn't be found
Process 40980 has exited with status 1

[dlsniper]

Terraform uses plugins by executing them via shell. The plugins are different running processes than what binary you are launching. As such you need either to attach to the new process or execute the plugins with delve

[aarzilli]

@sodre If you compile github.com/hashicorp/terraform and run it what happens?

[sodre]

@aarzlli I thought that is what dlv debug github.com/hashicorp/terraform -- apply was doing for me.

@dlsniper you are correct. I added a runtime.Breakpoint() line to where I was trying to debug but that just caused the separate program to crash -- I assume because I was not fast enough to attach to it. I could put the code in an controlled loop near where I want to debug but that sounds like a hack. Also, wouldn't that mean that the plugin would have been compiled without the "delve" modifications?

I have been able to solve my original bug ( using print statements :) ) , so there is no rush. However, what should be delve's best practices for applications that behave like terraform?

[aarzilli]

@sodre I was trying to determine if the problem is the "provider triton couldn't be found" error or the fact that we don't follow execs.

If you have a way to start the plugins separately and point the main program to them, then you can do that. Otherwise using delve will require either changing the main program or changing the plugins. Delve does not currently follow execs and getting it to do it is a major feature.

[sodre]

@aarzilli got it! I'll see how that can be done.

[derekparker]

On Linux where we use ptrace(2) for process manipulation this can be achieved very easily by ensuring we always set the PTRACE_O_TRACEFORK option.

However, for OSX and Windows I am not so sure. Windows may have something for this in their debug APIs. For OSX, since we're dealing mostly with mach exceptions, and lower-level syscalls for process manipulation, we need a way to determine when the traced process has fork'ed so we can begin tracing the new child. After some quick research I couldn't find a way to get notifications upon a fork call in the traced process. We do always get a SIGTRAP exception when the traced process calls exec since we're ptraceing the process, however since ptrace(2) is so horribly limited on OSX, we don't have all the options / niceties that we do on Linux. I will keep trying to dig into this.

[sodre]

Derek,

Thanks for looking into this, would dtrace be a better equivalent in the OS X world?
A quick search on dtrace and fork on google brought up this link from sun.com (oracle.com)

https://community.oracle.com/thread/1921595?start=0&tstart=0

Patrick

On Apr 18, 2016, at 12:25 PM, Derek Parker [email protected] wrote:

On Linux where we use ptrace(2) for process manipulation this can be achieved very easily by ensuring we always set the PTRACE_O_TRACEFORK option.

However, for OSX and Windows I am not so sure. Windows may have something for this in their debug APIs. For OSX, since we're dealing mostly with mach exceptions, and lower-level syscalls for process manipulation, we need a way to determine when the traced process has fork'ed so we can begin tracing the new child. After some quick research I couldn't find a way to get notifications upon a fork call in the traced process. We do always get a SIGTRAP exception when the traced process calls exec since we're ptraceing the process, however since ptrace(2) is so horribly limited on OSX, we don't have all the options / niceties that we do on Linux. I will keep trying to dig into this.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub https://github.com/derekparker/delve/issues/496#issuecomment-211455885

[sjwl]

The limitation of tracing programs using exec is not just for the Terraform plugins, but also the binary itself.

Turns out the terraform binary also uses exec on itself. From it's main routine, it ultimately spawns another instance of itself inside a wrapper like so:

https://github.com/mitchellh/panicwrap/blob/master/panicwrap.go#L143

    // Build a subcommand to re-execute ourselves. We make sure to
    // set the environmental variable to include our cookie. We also
    // set stdin/stdout to match the config. Finally, we pipe stderr
    // through ourselves in order to watch for panics.
    cmd := exec.Command(exePath, os.Args[1:]...)

btw, I was able to figure this out thanks to delve, except once the process executes the line above, delve can no longer follow...

[sjwl]

I just returned from Hashicon and got a tip from the core Terraform team (they use delve too!). This is how I got debugging to work with Terraform:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "remotePath": "",
            "port": 2345,
            "host": "127.0.0.1",
            "program": "${workspaceRoot}",
            "env": {
                "cccf35992f8f3cd8d1d28f0109dd953e26664531": "7c28215aca87789f95b406b8dd91aa5198406750"
            },
            "args": ["plan", "/my_terraform_config_file_directory"],
            "showLog": true
        }
    ]
}

NOTE: this technique only works with Terraform core. Debugging the Terraform plug-ins is still an issue.

The key is setting the above environment variable. I won't even try to pretend explaining why this works, but you can find where I got the mysterious values from here: https://github.com/mitchellh/panicwrap/search?utf8=%E2%9C%93&q=DEFAULT_COOKIE_KEY

[huydinhle]

@sjwl hi, I saw your post, I am currently trying to work on terraform-core. Could you post the step, I don't know how to feed this json into my delve.

[vancluever]

@sjwl funny to see your post here! Thanks for leading me in the right direction!

@huydinhle it should be noted that the JSON above is a https://github.com/microsoft/vscode launch.json file, so if you are not using Visual Studio code then this might not apply to you.

I've been playing with this today on OS X and Linux, and my findings so far are:

• I didn't really have much success in OS X. Environment vars didn't pass in properly, and after hacking the source to not fork execution hung until I killed the internal plugins that TF runs for providers. I also swear I could not get variables to print properly either. This was using brew off the master recipe.
• Linux however was much, much easier to work with. Delve worked with Terraform pretty much exactly like it says it does on the tin. Environment vars pass in properly, which means that you don't need to pass in the panicwrap cookie variable, just simply run the following:

TF_FORK=0 dlv exec `which terraform` -- apply

TF_FORK=0 disables the panicwrap forking and just runs main in the current process. Debugging worked great, provider plugins don't work as @sjwl mentioned (although I'm wondering if you could circumvent this by using dlv test if you really needed to debug a provider), but other than that everything else works great.

Also, don't necessarily build the binary with make dev. go install -gcflags="-N -l" will build you the binary with the optimizations disabled which is necessary to print variables, and put it in your GOPATH. Alternatively you could probably using dlv debug with the appropriate gcflags passed in, if you're okay with the delays involved in building every time you run the debugger.

[derekparker]

@vancluever

Environment vars didn't pass in properly,

This should be fixed on master for OSX.

[farvour]

Are there any updates to this issue? SPecifically, how to debug custom plugin/providers?

[vancluever]

For @farvour and anyone else looking for an answer: This is quite a late reply, but I thought I'd drop in and say: providers are actually one of the _easier_ things to debug in Terraform with Delve!

The key to doing it is to debug it through Delve's test functionality, through an acceptance test. Write an acceptance test for the scenario that you want to test on, if one does not exist already.

Then, run:

TF_ACC=1 dlv test ./YOUR_PROVIDER_NAME -- -test.v -test.run=TestAccSomeResourceHere_basic

The key to remember is that since you have to pass your test arguments over to the actual test binary directly instead of proxying them with go test, prefixing your test-specific arguments with test. is necessary after the -- terminator.

In the debugger, set your breakpoints, continue, and debug away!

In some of the providers I have worked on, you will see a debugacc make target that helps with this a bit, although you will still need to supply the -test.run= to run a specific test:

debugacc: fmtcheck
    TF_ACC=1 dlv test $(TEST) -- -test.v $(TESTARGS)

[derekparker]

Going to close this out as it seems to be working now. Please reopen if that is not the case.

[EliiseS]

This is _quite_ an old thread and I'm probably screaming into the void, but I'm trying to debug a terraform provider acceptance test with vscode and delve. I'm trying to pass a -v argument to the test and failing miserably.

The debugger config:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${fileDirname}",
            "env": {
                "TF_ACC": "1"
            },
            "args": [
                "-test.v",
            ],
            "buildFlags": "-v -tags=all",
            "trace": "log",
            "envFile": "${workspaceRoot}/.env",
            "showLog": true
        }
    ]
}

Which results in the following error:

API server listening at: 127.0.0.1:37336
--- FAIL: TestAccResourceAuthorization_CRUD (0.00s)
    testing.go:555: Acceptance tests must be run with the -v flag on tests
FAIL
Process exiting with code: 0

Any help would be appreciated.

[aarzilli]

@polinasok is this something you would know?

[polinasok]

Hmm, I don't see anything obviously wrong with the configuration. It would be more bulletproof if you specified "test" mode explicitly, but based on the log, it seems to be resolving things correctly automatically.

Is it possible that the condition that prints the error, tests for more than just testing.Verbose() (example), so something else is not configured correctly?

vscode should log the delve command that it creates based on the configuration - "Running: /my/bin/dlv test ..." And then delve should log what it is going to run: "info layer=debugger launching process with args: ...". Do those look right?

[EliiseS]

Hey! Thanks for your replies @aarzilli @polinasok! I've managed to figure out a way to get it to work here: https://dev.to/eliises/debug-terraform-azuredevops-provider-with-vscode-c24

The only problem is that it doesn't work by clicking the debug test option in VSCode and I have no idea how to get it to work with that :(

[polinasok]

debug test doesn't currently use the configuration from launch.json - see https://github.com/golang/vscode-go/issues/855 for details