Skaffold: [Deploy] Add Terraform Deployer

[bzub]

I'm working on a Deployer implementation for Terraform modules and would like to get early input for the design. Initial proposal follows. Questions and comments are highly encouraged!

Skaffold Terraform Deployer

Uses terraform cli to deploy Kubernetes resources via Terraform module(s).

Use Cases / Goals

• Rapid development of apps in an environment where Terraform is used significantly to manage Kubernetes clusters and resources.
• Infrastructure and environment details can be consumed from a remote Terraform state and incorporated in an app's development/deployment dynamically.
• Apps developed with Skaffold + Terraform Deployer can be tested and then released as Terraform modules for use in more complex Terraform environments.

Design

Scope

• I believe the Skaffold Terraform Deployer will be relatively simple, providing a thin layer of automation on top of simply running terraform apply and terraform destroy on one or more root modules.
• Terraform modules, and variables settings fed to said modules, would be the primary (only?) configurations available at the Skaffold level/config.
• Dynamic values managed by Skaffold like image tags would be passed into Terraform modules as variable inputs.

Skaffold Config

Best represented with an example:

deploy:
  terraform:
    modules:
    - path: tf/module1
      variables:
        variable_name: value
    - path: tf/module2

In the example above two Terraform modules backed by two separate Terraform states will be used by the Skaffold project. This config results in (roughly) the following CLI commands being run by Skaffold:

• tf/module1

terraform [apply|delete] -state=~/tf/states/module1/terraform.state -var 'variable_name=value' tf/module1

• tf/module2

terraform [apply|delete] -state=tf/module2/terraform.state tf/module2

[balopat]

Hi @bzub, thank you for opening this. We are thinking about how Skaffold could help in setting up the infrastructure around a deployment. However the Deployer interface does not seem like the best place to start in my mind, it was meant to be for a Kubernetes (maybe Knative in the future) specific application deployment that for example should be triggered every time there is a source code change in skaffold dev. Were you thinking about Kubernetes modules only in Terraform? Doing infrastructure work on every code change seems excessive.

[bzub]

Hi @balopat, I'm glad you brought this up. My use-case is limited to deploying Kubernetes resources via the Kubernetes Terraform Provider. Any examples of using this proposed deployer would reflect that theme.

That said, I had not intended on enforcing this as a restriction in the deployer. It is possible to use the terraform --target flag to limit use to only kubernetes resources, however that quickly becomes fragile due to the possibility to use multiple instances of the kubernetes provider via aliases.

So in the end I thought this Skaffold deployer would assume the user is deploying Kubernetes resources via Terraform configs. If users go beyond this assumption, it will not get in their way. Since this deployer should be a fairly thin wrapper over Terraform I think it will be rare for a problem with Terraform or a provider other than Kubernetes to be conflated as a Skaffold issue.

[bzub]

From a design perspective, deploying other infrastructure with this deployer could actually be fairly intuitive. Assuming this deployer tracks what Kubernetes resources it creates accurately, and deletes those resources from a cluster accurately, when a user does a skaffold deploy again their non-Kubernetes resources would not be re-applied, since Terraform will detect that only the Kubernetes specific resources are not in their desired state.

[balopat]

Hi @bzub - for now I would put this idea on hold and close this issue. We thought about this, and are not convinced that this belongs to the scope of Skaffold. If we see a lot of demand, we can reopen it.

[Hkyx]

Hi @balopat ,

I got the same pretty usecase than Bzub but I understand the mindest behind terraform is not the same as skaffold is doing. So question, how to do work at hashicorp to get your live debugging into a kubernetes but keeping deploying with TF ?

[vladdy]

I have another use case where the app instance goes beyond kubernetes as it requires some AWS resources (i.e, KMS keys, IAM roles for service accounts). Having Terraform as a deployer or some kind of hook to run it along with existing deployers would make it possible to use skaffold in such development requirements.

[MoSattler]

Same here - a terraform deployer would be immensely helpful