Terraform: [API / Library / Question] How to discover type data from provider plugins?

[awilkins]

With the new support for language-server in Atom, I found myself with the urge to write a Terraform language-server to make authoring TF files easier.

My reasoning went like this : Terraform is written in Go, so it would make sense to write a language-server in Go and mine the Terraform plugins for type information on resources etc.

I got as far as listing the names of resources that a TF plugin supports (and whether they are importable or not), but I couldn't work out for the life of me how to get e.g. the list of input attributes that a particular resource expects.

Is there something obvious I'm missing or is this information just really difficult to discover from the API (I understand that it's not necessary to reveal all this info to serialize things)

e.g. I can get a ResourceType[] from the .Resources() method of a Provider, but not any more info.

..... while perusing the source to construct this issue I come across : 183833affc456db915f1edfc4d1e871816d73db8

Is that the sort of thing I'd need?

[apparentlymart]

Hi @awilkins!

The commit you found there is part of some current work in progress to make the information you're looking for available to core Terraform. The initial motivation for it is to drive the new configuration parser (which uses schema to produce better error messages during parsing) but using it for editor integrations as you're thinking of here was another use-case we had in mind for the future.

I'm not sure what's the best way to get at this data with what's currently implemented, since it's currently loaded in a place that's convenient for the configuration parser but not so convenient for you to call it directly from some Go code of your own.

Calling directly into the plugin code itself is a reasonable place to start, but for "real-world" use I expect you'd instead want to talk to the installed plugins via the plugin protocol so that you don't need to recompile your program each time the schemas change.

I have a few hints here but I'm not 100% sure what might be required here and indeed these internal interfaces are subject to change as we continue to develop Terraform. (Building a language server _inside_ Terraform was on our longer-term ideas list, so if you get something working out of this we could adopt it into Terraform itself potentially, which would then cause us to maintain compatibility with it.)

• The plugin/discovery package has the machinery for searching for plugins, though unfortunately the details of which directories to search are private in both the main package and the command package. :confounded:
• Once you've found the plugin you want, you can get a client for it using plugin.Client.
• You can see in providerFactory an example of how to turn a plugin.Client into a terraform.ResourceProvider. (There's some other useful stuff to refer to in that file too.)
• Once you have that, you can call GetSchema on it to get the schema for the provider config itself and for zero or more resource types and data sources.

Unfortunately all of this comes with a big caveat today: we've not yet rebuilt any of the plugins with support for this new method, so calling this on the released providers (as of this writing) will result in an error. We'll be rebuilding the plugins with support for this as part of rolling out the opt-in experimental version of the revamped configuration language. In the mean time, if you compile the plugins yourself and update their vendored copies of Terraform to master you should find that they work as expected. (I've been doing this myself for testing purposes.)

The result of all this is types from configschema.

Hopefully once we get past the initial set of use-cases for the config language we can come back and smooth this out a bit and make simpler interfaces to look up provider schema for use-cases like yours. If you're feeling adventurous though, I'd love to give any help I can to get something working since having a language server for Terraform is a bit of a dream of mine and, as I say, one of the motivations for introducing this schema method in the first place. :grinning:

[awilkins]

Cool! I figured that might be the position.

One thought I have arrived at WRT a language server in Terraform is that it could benefit from some of the types being made more specific, my first thought was ID types ; many input attributes require an ID of a specific type (usually from a specific resource) but you'd not be able to determine which output attributes match that type from the present schemas where fields are mostly of TypeString.

I was figuring that I'd have to augment the existing schema with a layer of more metadata to make a language server maximally useful (I reckon typing ID expressions is probably the largest effort class task in authoring a Terraform file and you could probably do some really smart magic inferring which attribute you wanted by comparing the resource names of the attribute you're typing in with those of the resources with compatible outputs).

[apparentlymart]

Hi @awilkins,

One way of thinking about ids here could be to have something that represents something comparable to "foreign key" relationships in a database, saying e.g. values in subnet_ids on aws_instance correspond to values of id on aws_subnet, and with that foundation _in principle_ static analysis can be used to trace expressions that _indirectly_ refer to an AWS subnet id. (This is complicated, unfortunately, by passing values _between_ Terraform configurations using outside data stores or using terraform_remote_state.)

Terraform doesn't currently have any need for such data, and I think retrofitting it would be quite a lot of work, but perhaps this could be prototyped by maintaining a mapping outside of Terraform of a handful of the most interesting cases to see what sort of editor support can be built from such a thing, and that'll help inform whether the additional complexity in Terraform is worth the payoff to have more generalized support for this.

Currently Terraform has a special idea of an "id" for a resource that's primarily used for internal tracking, but in the long run we've been considering eliminating that in favor of just using normal attributes for this use-case, thus allowing us to more naturally represent situations where the id is actually a _tuple_ of different attributes rather than a single string. That's one reason why I'd suggest thinking of this as a generalized "foreign key"-type thing rather than thinking of it as being about ids in particular, though I'd acknowledge that this generalization will likely make the problem more complex to solve.

The first language-server-type usecases I'd thought about -- which I think are easier to implement with information Terraform already has -- were these:

• Auto-complete of the strings following resource and data in the header of resource and data blocks, for resource types.
• Auto-complete of attribute names in resource/module/etc block bodies.
• "Go to definition" for resource and attribute references in expressions.
• Auto-complete of variable/attribute names in interpolation expressions.
• "Find references" for resources, modules, specific attributes, etc.
• Hover over a name in an expression to see its type and, if available, current value from state. (the latter is tricky when using remote backends since it would involve reaching out to the remote service and may require credentials, so just the type might be more practical to start)
• Rename resources, etc and simultaneously update all references to them in the current module. (and, ideally, also in the state like terraform mv, but that's tricky if there are multiple workspaces in play and/or if you don't have remote state credentials available in the language server)

The new language parser was designed to retain enough information to be able to build out most of this stuff with some further effort (efficient implementation of some will likely require walking the tree and caching results, etc).

Auto-completing (or otherwise helping with) attribute _values_ is honestly not something I'd thought too much about, and indeed it is a harder thing to deal with. I'm curious to see what it might look like (from a user's perspective) in practice.

[tintoy]

@awilkins just tried this myself, and then discovered this issue :)

Are you still working on this? Are you by any chance building it as an LSP language service? If so, would be good to compare notes (not that I've got very far yet 😁)

[awilkins]

Hi there.... I had another go along the lines above - tried building my own copy of the AWS provider (it's the one I care about most).

The problem I ran into then was something I understand is a Golang "quirk" ; vendor namespaces - the classes from the vendored copy of the core libraries pulled into the provider don't have the same namespace as the actual core library, so you can't pass instances from one to the other?

I confess it got a bit much for my old noggin used to the likes of Java where the classpath is a PITA but does at least name classes predictably and I couldn't find docs telling me what the "right" way to get around that is.

[minamijoyo]

Hi there,
I'm interested in the language server for Terraform and trying to call GetSchema , but it failed.

The problem looks the same as what @awilkins is mentioned to.
Types of vendored library in provider are not same as the others.

A reproduction code is as follows:

```main.go
package main

import (
"fmt"
"go/build"

    "github.com/davecgh/go-spew/spew"
    "github.com/hashicorp/terraform/plugin"
    "github.com/hashicorp/terraform/plugin/discovery"
    "github.com/hashicorp/terraform/terraform"
    _ "github.com/zclconf/go-cty/cty"

)

func main() {
// find provider plugins
gopath := build.Default.GOPATH
pluginDirs := []string{gopath + "/bin"}
pluginMetaSet := discovery.FindPlugins("provider", pluginDirs)
spew.Dump(pluginMetaSet)

    plugins := make(map[string]discovery.PluginMeta)
    for plugin := range pluginMetaSet {
            name := plugin.Name
            plugins[name] = plugin
    }

    // initialize aws plugin
    client := plugin.Client(plugins["aws"])
    rpcClient, err := client.Client()
    if err != nil {
            panic(err)
    }

    raw, err := rpcClient.Dispense(plugin.ProviderPluginName)
    if err != nil {
            panic(err)
    }
    provider := raw.(terraform.ResourceProvider)

    // invoke GetSchema
    req := &terraform.ProviderSchemaRequest{
            ResourceTypes: []string{"aws_security_group"},
            DataSources:   []string{},
    }
    res, err := provider.GetSchema(req)

    if err != nil {
            panic(fmt.Sprintf("%+v", err))
    }

    spew.Dump(res)

}

The terraform core and the AWS provider are current master.

[terraform@master]$ git rev-parse HEAD
18975d72704796def799bdae053c8c9ae6f5c92c

[terraform-provider-aws@master]$ git rev-parse HEAD
558487c96c6548b58729ab434c367cbb48b2b254
[terraform-provider-aws@master]$ go install
[terraform-provider-aws@master]$ ls -la $GOPATH/bin/terraform-provider-aws
-rwxr-xr-x 1 minamijoyo staff 115546724 1 5 21:16 /Users/minamijoyo/bin/terraform-provider-aws

[20180105]$ go run main.go
2018/01/05 21:31:29 [DEBUG] checking for provider in "/Users/minamijoyo/bin"
2018/01/05 21:31:29 [WARNING] found legacy provider "terraform-provider-aws"
(discovery.PluginMetaSet) (len=1) {
(discovery.PluginMeta) {
Name: (string) (len=3) "aws",
Version: (discovery.VersionStr) (len=5) "0.0.0",
Path: (string) (len=44) "/Users/minamijoyo/bin/terraform-provider-aws"
}: (struct {}) {
}
}
2018-01-05T21:31:29.013+0900 [DEBUG] plugin: starting plugin: path=/Users/minamijoyo/bin/terraform-provider-aws args=[/Users/minamijoyo/bin/terraform-provider-aws]
2018-01-05T21:31:29.016+0900 [DEBUG] plugin: waiting for RPC address: path=/Users/minamijoyo/bin/terraform-provider-aws
2018-01-05T21:31:29.038+0900 [DEBUG] plugin.terraform-provider-aws: plugin address: timestamp=2018-01-05T21:31:29.037+0900 address=/var/folders/xb/25_mwq6d3bj1ycr70ng1f09h0000gn/T/plugin801159030 network=unix
panic: reading body error decoding cty.Type: gob: name not registered for interface: "github.com/terraform-providers/terraform-provider-aws/vendor/github.com/zclconf/go-cty/cty.primitiveType"

goroutine 1 [running]:
main.main()
/Users/minamijoyo/work/tmp/20180105/main.go:48 +0x5f7
exit status 2
```

The sample code invokes GetSchema API in the AWS provider via go-plugin , which uses gob encoding/decoding,
and the provider has vendored type of go-cty , which uses gob.Register.

https://github.com/zclconf/go-cty/blob/48ce95f3a00f37ac934ff90a62e377146f9428e1/cty/types_to_register.go#L48

So we can't gob decode the vendored cty.primitiveType .

I think gob.RegisterName may be worth considering, but this may not be type-safe.

I've found a related issue of the go-plugin .
https://github.com/hashicorp/go-plugin/issues/16

I guess the same problem will be present in the terraform core. (The vendored go-cty in the provider is not the same as the terraform core's one)

@apparentlymart Can you decode the GetSchema response from the provider correctly?

[minamijoyo]

@apparentlymart Thank you for fixing go-cty to use gob.RegisterName.
It works fine for me 😄
I've just submitted a pull request to update vendored go-cty in the Terraform core.
https://github.com/hashicorp/terraform/pull/17055

[apparentlymart]

Hi @minamijoyo! I actually independently ran into this issue while working on something else, so it was just a coincidence that I happened to fix just after you encountered it, but I'm glad the fix worked out for you too!

I was planning to update the vendoring as part of some other work but we can probably update it separately now as you proposed... I'll just do a quick check to make sure there aren't any implications of that which might cause trouble for existing plugin binaries.

[minamijoyo]

@apparentlymart Hah, even if we actually independently found the same issue, anyway, thank you for working on this!!

[minamijoyo]

FYI: I wrote a third-party tool of Terraform, named tfschema, which gets resource type definitions dynamically from Terraform providers via go-plugin protocol.

https://github.com/minamijoyo/tfschema

[apparentlymart]

Hi all,

A long time after this discussion, we added the terraform providers schema -json command, which returns the schema for all of the providers in the current configuration in a JSON format.

The format of this is not yet documented because it was quite a late addition to the Terraform v0.12.0 scope, but hopefully it's reasonably self-explanatory if you're already familiar with the result of calling the GetSchema function as we discussed before.

This command is intended as a public interface, so I'd suggest using it where possible rather than calling directly into Terraform or into the providers themselves. For integration into an editor, you may wish to cache the (often quite large) output of this command in memory, and then monitor for changes in the .terraform/plugins directory to know when to flush that cache and re-read new schemas.