Terraform-provider-kubernetes: Terraform Kubernets provider pipe "|" in AKS daemonset not formatted properly in yaml

[a-m5]

I am trying to add our certificates to an AKS cluster using a DS, but Terraform translates the pipe incorrectly into the yaml. We are doing this a bit odd, and not leveraging other options such as a config map because Docker for some reason wont restart with those ways. We have been on multiple calls with Microsoft, and doing the DS this way, seems to be the only way that will work. What happens, is Terraform single quotes the pipe in the yaml, which then causes the script to error out. I already have a ticket open with Hashi on this, but was advised to create an issue as well.

Terraform Version and Provider Version

Terraform version: 0.13.2
Kubernetes provider version: 1.12.0

Affected Resource(s)

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
resource "kubernetes_daemonset" "cert" {
metadata {
name = "nginx-ca"
namespace = "kube-system"
labels = {
k8s-app = "nginx-ca"
}
}

spec {
selector {
match_labels = {
name = "nginx-ca"
}
}
strategy {
type = "RollingUpdate"
}
template {
metadata {
labels = {
name = "nginx-ca"
}
}

spec {
node_selector = {
"kubernetes.io/os" = "linux"
}

restart_policy = "Always"
dns_policy = "ClusterFirst"
host_pid = "true"

container {
image = "nginx"
name = "nginx-ca"
image_pull_policy = "IfNotPresent"
command = ["nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid", "--", "sh", "-c", "|", "rm -rf /etc/docker/certs.d/root.crt", "curl http://certpath.... -o /etc/docker/certs.d/root.crt", "cp -a /etc/docker/certs.d/root.crt /etc/ssl/certs/root.pem", "update-ca-certificates", "systemctl restart docker", "systemctl status docker", "while true; do sleep 1000; done"]

security_context {
privileged = true
}

volume_mount {
name = "etc-docker"
mount_path = "/etc/docker/certs.d"
}
volume_mount {
name = "etc-ssl"
mount_path = "/etc/ssl/certs"
}
}

volume {
name = "etc-docker"
host_path {
path = "/etc/docker/certs.d"
type = "Directory"
}
}
volume {
name = "etc-ssl"
host_path {
path = "/etc/ssl/certs"
type = "Directory"
}
}

}
}
}
}

Debug Output

Panic Output

Expected Behavior

What should have happened?
Terraform should create the daemonset, which adds the certificates and restarts docker.

Actual Behavior

What actually happened?
Terraform creates the DS, but formats the command wrong, specifically around the pipe, which causes the commands to error.

Steps to Reproduce

Important Factoids

How Terraform is formatting the yaml after creation:

containers:
- command:
- nsenter
- --target
- "1"
- --mount
- --uts
- --ipc
- --net
- --pid
- --
- sh
- '|'
- -c
- rm -rf /etc/docker/certs.d/root.crt
- curl http://certpath... -o /etc/docker/certs.d/root.crt
- cp -a /etc/docker/certs.d/root.crt /etc/ssl/certs/root.pem
- update-ca-certificates
- systemctl restart docker
- systemctl status docker
- while true; do sleep 1000; done

How the yaml should be formatted for the DS to work:

containers:
- command:
- nsenter
- --target
- "1"
- --mount
- --uts
- --ipc
- --net
- --pid
- --
- sh
- -c
- |
rm -rf /etc/docker/certs.d/root.crt
curl http://certpath... -o /etc/docker/certs.d/root.crt
cp -a /etc/docker/certs.d/root.crt /etc/ssl/certs/root.pem
update-ca-certificates
systemctl restart docker
systemctl status docker
while true; do sleep 1000; done

I have also tried formatting the Terraform with the pipe as so "put | things | with | pipes | here" and received the same result.

References

• GH-1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jrhouston]

Ah, I see what the problem is.

Terraform doesn't actually deal with Kubernetes YAML at all — it translates the HCL config into Go types made available through client-go, which in turn serializes them to JSON to send directly to the API, so there's no YAML involved. The | which, at first glance, makes it looks like you are doing a pipe is actually a YAML convention for supplying a multi-line string to an attribute. HCL, and therefore Terraform, doesn't use this convention.

The solution here is to use the heredoc syntax to supply everything that is meant to come after sh -c, like this:

command  = [
  "sh", "-c", <<EOT
command 1 
command 2
command 3
  EOT
]

[a-m5]

@jrhouston This did the job, thank you!

[jrhouston]

@a-m5 No problem! Going to close this then. 😄

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!