Atlantis: Handling passwords

[justinhauer]

How are people handling passwords being sent into the .tf file with Atlantis? Example: I want to create a database in Azure, and in the .tf file for this I need to set the username and password for the database - I obviously don't want this in Git history. What way should this be handled? Perhaps a Vault integration, which could pull the values from vault into Terraform at runtime?

[lkysow]

Hi Justin, do the answers here help? https://github.com/runatlantis/atlantis/issues/471
There is already a vault provider for Terraform that can accomplish what you're asking.

[justinhauer]

I've got some additional questions on this, and perhaps the way I'm think about this, you can provide some best practices feedback:

The way you mention in #471 would work:

provider "vault" {
 ...
}
data "vault_generic_secret" "secret" {
  path = "path"
}
provider "vsphere" {
  password       = "${data.vault_generic_secret.secret["password"]}"
}

.... For setting up a password before hand, but think of this from an end user's perspective with the scenario above, they'll have to set that password in vault prior to the TF run. Can you think of any type of scenario where a random string could be generated for username/pw, and be output into a vault wrapper, for people to pick up their credentials after the fact? Is that a thing that's possible? This might be more of a terraform question.

[tedwardd]

Would something like this work for you?

resource "random_string" "password" {
  length = 16
  special = true
}
provider "vault" {
  ...
}
data "vault_generic_secret" "secret" {
path = "path"
data_json = <<EOT
  {
    password = "${random_string.password.result}"
  }
}
provider "vsphere" {
  password = "${data.vault_generic_secret.secret["password]}"
}

I'm not 100% sure but I think the value of ${random_string.password.result} might come out in the plan output but maybe using sensitive = true would work here. I've not tested it, though.

[justinhauer]

I've gotta set up the vault provider today, let me see if I can test this ^ out and see the result. As a feature request for the project, I think it would be a value add to publish things like this on how to handle secrets with different scenarios when using Atlantis/Terraform

[lkysow]

I've gotta set up the vault provider today, let me see if I can test this ^ out and see the result. As a feature request for the project, I think it would be a value add to publish things like this on how to handle secrets with different scenarios when using Atlantis/Terraform

Absolutely, I've been meaning to add an FAQ section to the docs.

[justinhauer]

Do you have any documentation on deploying the vault auth piece with your helm chart? since this is in a stateful set and the vault/k8s documentation is in a deployment, I would assume if anyone wants to use Atlantis and have secret management, the main way of deploying via helm charts would no longer work, as you now have to add in these vault side cars: https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s
Again this isnt your fault, just integrating the hashicorp secrets management pieces with atlantis seems to be getting into the weeds for the average user...

[lkysow]

Sorry I do not have documentation on this. As far as I know, you're the first user to be doing this.

I agree it's getting into the weeds but I'm guessing the "average user" either use their cloud provider's secret store, or use kubernetes secrets to inject credentials, or injects a long-lived vault token so they don't need the sidecar.

[justinhauer]

When I get this running, I'll make the statefulset generic and do a PR if you want to use it as an example

[justinhauer]

Got it running, you can close out the issue. Let me know if you want an example of the helm deployment's statefulset + vault pieces to integrate

[tedwardd]

@justinhauer It looks like either I may have missed some of the context around what you were trying to do here or that you pivoted to a modified solution (your mentions of helm have me confused). I would very much like more details and context around what you're doing here if you don't mind sharing. If this ticket is not the appropriate place to do so, you're welcome to message me on the Atlantis slack (@ tedward).

[lkysow]

Got it running, you can close out the issue. Let me know if you want an example of the helm deployment's statefulset + vault pieces to integrate

Posting them here would be great!

[justinhauer]

Sure, here's a recap:
1) I wanted to deploy atlantis. I initially used helm:

helm install -f values.yaml atlantis --tiller-namespace=my-namespace --name=name-for-atlantis-statefulset

2)
That deployed successfully - but there was no documentation around Azure credential to use terraform with an Azure provider in your documentation, so I added in env-vars into a container image based off your container image. With this, I was able to deploy resources to Azure.
3)
There was no documentation I saw on your site regarding sending in credentials to a terraform document, or how you handle any integrations with a credential provider of any sort. My organization stood up vault recently so I looked into the integrations with that. This was needed in the event of:

• I need to create a resource that needs credentials provided to it (think of a master account for a db, server)
• I want to assign randomly generated credentials for a resource (database or vm lets say), and I need a secure way for a customer/end user to pick up these credentials

This can be figured out in the .tf file by using the terraform vault provider docs to craft your module/.tf file.

4) In order to integrate with terraform, the statefulset that gets created through your helm chart has to be modified with the following:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ template "atlantis.fullname" . }}
  labels:
    app: {{ template "atlantis.name" . }}
    chart: {{ template "atlantis.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  {{- with .Values.statefulSet.annotations }}
  annotations:
{{ toYaml . | indent 4 }}
  {{- end }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "atlantis.name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "atlantis.name" . }}
        release: {{ .Release.Name }}
      {{- with .Values.podTemplate.annotations }}
      annotations:
{{ toYaml . | indent 8 }}
      {{- end }}
    spec:
      # Authenticate to vault using kubernetes method to get a token from JWT
      serviceAccountName: {{ template "atlantis.serviceAccountName" . }}
      securityContext:
        fsGroup: 1000
      volumes:
      - name: vault-token
        emptyDir:
          medium: Memory
      - name: vault-config
        configMap:
          name: atlantis-vault-autoauth
      - name: atlantis-consul-template
        configMap:
          name: atlantis-consul-template
      - name: shared-data
        emptyDir: {}
      - name: entrypoint
        configMap:
          name: atlantis-entrypoint
          defaultMode: 0744
      {{- if .Values.tlsSecretName }}
      - name: tls
        secret:
          secretName: {{ .Values.tlsSecretName }}
      {{- end }}
      {{- range $name, $_ := .Values.serviceAccountSecrets }}
      - name: {{ $name }}-volume
        secret:
          secretName: {{ $name }}
      {{- end }}
      {{- if .Values.gitconfig }}
      - name: gitconfig-volume
        secret:
          secretName: {{ template "atlantis.fullname" . }}-gitconfig
      {{- end }}
      {{- if .Values.aws }}
      - name: aws-volume
        secret:
          secretName: {{ template "atlantis.fullname" . }}-aws
      {{- end }}
      containers:
        - name: vault-agent-auth
          image: vault
          volumeMounts:
            - name: vault-config
              mountPath: /etc/vault
            - name: vault-token
              mountPath: /home/vault
          env:
            - name: VAULT_ADDR
              value: {your vault server address}
            - name: VAULT_SKIP_VERIFY
              value: '1'
          args:
            - agent
            - -config=/etc/vault/vault-agent-config.hcl
            # - -log-level=debug
        - name: consul-template
          image: hashicorp/consul-template:alpine
          volumeMounts:
            - name: vault-token
              mountPath: /home/vault
            - name: atlantis-consul-template
              mountPath: /etc/consul-template
            - name: shared-data
              mountPath: /etc/secrets
          env:
            - name: HOME
              value: /home/vault
            - name: VAULT_ADDR
              value: {your vault address}
          args:
            - -config=/etc/consul-template/consul-template-config.hcl
            # - -log-level=debug

        # Use the generated secret file to load into env and run desired app
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command:
            - /bin/sh
          {{- if .Values.gitconfig }}
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "cp /etc/secret-gitconfig/gitconfig /home/atlantis/.gitconfig && chown atlantis /home/atlantis/.gitconfig"]
          {{- end}}
          args:
            - /etc/entrypoint/entrypoint.sh
          {{- if .Values.allowRepoConfig }}
            - --allow-repo-config
          {{- end }}
          ports:
          - name: atlantis
            containerPort: 4141
          env:
          {{- range $key, $value := .Values.environment }}
          - name: {{ $key }}
            value: {{ $value | quote }}
          {{- end }}
          {{- if .Values.logLevel }}
          - name: ATLANTIS_LOG_LEVEL
            value: {{ .Values.logLevel | quote}}
          {{- end }}
          {{- if .Values.tlsSecretName }}
          - name: ATLANTIS_SSL_CERT_FILE
            value: /etc/tls/tls.crt
          - name: ATLANTIS_SSL_KEY_FILE
            value: /etc/tls/tls.key
          {{- end }}
          - name: ATLANTIS_DATA_DIR
            value: /atlantis-data
          - name: ATLANTIS_REPO_WHITELIST
            value: {{ toYaml .Values.orgWhitelist }}
          - name: ATLANTIS_PORT
            value: "4141"
          {{- if .Values.atlantisUrl }}
          - name: ATLANTIS_ATLANTIS_URL
            value: {{ .Values.atlantisUrl }}
          {{- else if .Values.ingress.enabled }}
          - name: ATLANTIS_ATLANTIS_URL
            value: http://{{ .Values.ingress.host }}
          {{- end }}
          {{- if .Values.github }}
          - name: ATLANTIS_GH_USER
            value: {{ required "github.user is required if github configuration is specified." .Values.github.user }}
          - name: ARM_CLIENT_SECRET
            valueFrom:
              secretKeyRef:
                name: azurecreds
                key: ARM_CLIENT_SECRET
          - name: ARM_SUBSCRIPTION_ID
            valueFrom:
              secretKeyRef:
                name: azurecreds
                key: ARM_SUBSCRIPTION_ID
          - name: ARM_TENANT_ID
            valueFrom:
              secretKeyRef:
                name: azurecreds
                key: ARM_TENANT_ID
          - name: ARM_CLIENT_ID
            valueFrom:
              secretKeyRef:
                name: azurecreds
                key: ARM_CLIENT_ID
          - name: ATLANTIS_GH_TOKEN
            valueFrom:
              secretKeyRef:
                name: {{ template "atlantis.fullname" . }}-webhook
                key: github_token
          - name: ATLANTIS_GH_WEBHOOK_SECRET
            valueFrom:
              secretKeyRef:
                name: {{ template "atlantis.fullname" . }}-webhook
                key: github_secret
          {{- if .Values.github.hostname }}
          - name: ATLANTIS_GH_HOSTNAME
            value: {{ .Values.github.hostname }}
          {{- end }}
          {{- end}}
          {{- if .Values.gitlab }}
          - name: ATLANTIS_GITLAB_USER
            value: {{ required "gitlab.user is required if gitlab configuration is specified." .Values.gitlab.user }}
          - name: ATLANTIS_GITLAB_TOKEN
            valueFrom:
              secretKeyRef:
                name: {{ template "atlantis.fullname" . }}-webhook
                key: gitlab_token
          - name: ATLANTIS_GITLAB_WEBHOOK_SECRET
            valueFrom:
              secretKeyRef:
                name: {{ template "atlantis.fullname" . }}-webhook
                key: gitlab_secret
          {{- if .Values.gitlab.hostname }}
          - name: ATLANTIS_GITLAB_HOSTNAME
            value: {{ .Values.gitlab.hostname }}
          {{- end }}
          {{- end}}
          {{- if .Values.bitbucket }}
          - name: ATLANTIS_BITBUCKET_USER
            value: {{ required "bitbucket.user is required if bitbucket configuration is specified." .Values.bitbucket.user }}
          - name: ATLANTIS_BITBUCKET_TOKEN
            valueFrom:
              secretKeyRef:
                name: {{ template "atlantis.fullname" . }}-webhook
                key: bitbucket_token
          {{- if .Values.bitbucket.baseUrl }}
          - name: ATLANTIS_BITBUCKET_BASE_URL
            value: {{ .Values.bitbucket.baseUrl }}
          - name: ATLANTIS_BITBUCKET_WEBHOOK_SECRET
            valueFrom:
              secretKeyRef:
                name: {{ template "atlantis.fullname" . }}-webhook
                key: bitbucket_secret
          {{- end }}
          {{- end }}
          {{- if .Values.requireApproval }}
          - name: ATLANTIS_REQUIRE_APPROVAL
            value: "true"
          {{- end }}
          {{- if .Values.requireMergeable }}
          - name: ATLANTIS_REQUIRE_MERGEABLE
            value: "true"
          {{- end }}
          {{- if .Values.livenessProbe.enabled }}
          livenessProbe:
            httpGet:
              path: /healthz
              port: 4141
              scheme: {{ .Values.livenessProbe.scheme }}
            initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
            periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
            timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
            successThreshold: {{ .Values.livenessProbe.successThreshold }}
            failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
          {{- end }}
          {{- if .Values.readinessProbe.enabled }}
          readinessProbe:
            httpGet:
              path: /healthz
              port: 4141
              scheme: {{ .Values.readinessProbe.scheme }}
            initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
            periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
            timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
            successThreshold: {{ .Values.readinessProbe.successThreshold }}
            failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
          {{- end }}
          volumeMounts:
          - name: shared-data
            mountPath: /etc/secrets
          - name: entrypoint
            mountPath: /etc/entrypoint
          - name: atlantis-data
            mountPath: /atlantis-data
          {{- range $name, $_ := .Values.serviceAccountSecrets }}
          - name: {{ $name }}-volume
            readOnly: true
            mountPath: /etc/{{ $name }}
          {{- end }}
          {{- if .Values.gitconfig}}
          - name: gitconfig-volume
            readonly: true
            mountPath: /etc/secret-gitconfig
          {{- end }}
          {{- if .Values.aws}}
          - name: aws-volume
            readonly: true
            mountPath: /home/atlantis/.aws
          {{- end }}
          {{- if .Values.tlsSecretName }}
          - name: tls
            mountPath: /etc/tls/
          {{- end }}
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      {{- if .Values.imagePullSecrets }}
      imagePullSecrets:
        - name: {{ .Values.imagePullSecrets }}
      {{- end }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}
  volumeClaimTemplates:
  - metadata:
      name: atlantis-data
    spec:
      accessModes: ["ReadWriteOnce"] # Volume should not be shared by multiple nodes.
      {{- if .Values.storageClassName }}
      storageClassName: {{ .Values.storageClassName }} # Storage class of the volume
      {{- end }}
      resources:
        requests:
          # The biggest thing Atlantis stores is the Git repo when it checks it out.
          # It deletes the repo after the pull request is merged.
          storage: {{ .Values.atlantisDataStorage }}


---
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: atlantis-vault-autoauth
  namespace: atlantis
data:
  vault-agent-config.hcl: |
    pid_file = "/home/vault/pidfile"

    auto_auth {
      method "kubernetes" {
        mount_path = "auth/your_auth_path"
        config = {
          role = "team-{your-teams-name}-dev"
        }
      }
      sink "file" {
        config = {
          path = "/home/vault/.vault-token"
        }
      }
    }

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: atlantis-consul-template
  namespace: atlantis
data:
  consul-template-config.hcl: |
    vault {
      vault_agent_token_file = "/home/vault/.vault-token"
      retry {
        backoff = "1s"
      }
      ssl {
        verify = false
      }
    }

    template {
      destination = "/etc/secrets/values.sh"
      contents = <<EOH
      export VAULT_ADDR="{your vault address}"
      export VAULT_TOKEN="{{"{{"}} file "/home/vault/.vault-token" {{"}}"}}"
      EOH
    }

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: atlantis-entrypoint
  namespace: {your namespace}
data:
  entrypoint.sh: |
    source /etc/secrets/values.sh
    /usr/local/bin/docker-entrypoint.sh server

[justinhauer]

Forgot @lkysow that the docs for the vault integration are here: https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s

[tedwardd]

Also wanted to leave a minor follow-up to my previous comment. In digging in to this for something else I was working on, it does look like random_string suppresses it's output as of this PR: https://github.com/terraform-providers/terraform-provider-random/pull/18

[justinhauer]

Right, you need to send that random string somewhere though or you wouldn't know what it is :)

[vvbogdanov87]

The thread is closed but I want to add one more option here. Since we are talking about running Atlantis on k8s, the secret data can be taken from k8s secret.

provider "kubernetes" {}

data "kubernetes_secret" "password" {
  metadata {
    name = dbsecret
  }
}
resource "aws_db_instance" "default" {
  name                   = var.name
  username               = var.username
  password               = data.kubernetes_secret.password.data.password
  ...
}

[kubernetes_secret data source](https://www.terraform.io/docs/providers/kubernetes/d/secret.html_

[wsams]

I'm struggling with a similar issue. I want to use Atlantis to provision a Test and Production instance of Vault.

I've used envFrom to inject the VAULT_ADDR and VAULT_TOKEN environment variables, and planing to switch to AppRole. But how do I provide the Vault credentials necessary for the Vault provider to function securely. And provide Vault-Test vs Vault-Production credentials to Atlantis securely.

My server-side repo config looks like this:

repos:
  - id: /.*/
    apply_requirements: [approved,mergeable]
    allowed_overrides: [workflow]
    allow_custom_workflows: false
workflows:
  test:
    plan:
      steps:
        - init:
            extra_args: [-backend-config=vars/test.backend.tfvars]
        - plan:
            extra_args: [-var-file=vars/test.tfvars]
    apply:
      steps:
        - apply:
            extra_args: [-var-file=vars/test.tfvars]
  production:
    plan:
      steps:
        - init:
            extra_args: [-backend-config=vars/production.backend.tfvars]
        - plan:
            extra_args: [-var-file=vars/production.tfvars]
    apply:
      steps:
        - apply:
            extra_args: [-var-file=vars/production.tfvars]

And my Terraform project atlantis.yaml looks like this (although I feel like I'm overusing projects to mean test/prod - I only have a single Vault terraform project):

version: 3
projects:
- name: vault-test
  dir: .
  workflow: test
  workspace: test
  terraform_version: v0.13.5
- name: vault-production
  dir: .
  workflow: production
  workspace: production

I'm also wanting to use a Consul-Test backend for Vault-Test state, and a Consul-Prod backend for Vault-Prod state.

When I submit a PR, a plan is generated in the test workspace using the test workflow which swaps out tfvars. And similar for production. I would then have to run separate apply commands in GitHub with the correct -p value. For example, apply test, confirm all is good, then later apply production, then close the PR. But nothing enforces that workflow, and running just apply would apply to test and production at the same time.

What I can't figure out is how to toggle the backend and Vault test vs production credentials. Is there a clean way to do this?

Another thought was to have two Atlantis instances and set up a webhook for each so that when a PR is submitted Atlantis-test uses the Consul-Test backend and has the Vault-Test environment variables set (or AppRole for auth).

In a nutshell, I have a single Terraform config project and I want to use Atlantis to apply that configuration on a Vault-Test instance and have it's state stored in a Consul-Test backend. And if that looks good, also have Atlantis apply the configuration on a Vault-Prod instance with state in a Consul-Prod backend. And pass the Vault/Consul credentials to Atlantis securely.

Anyone have any suggestions?

Thanks

PS. Here's a screenshot showing how it ran the plan. (Note, the backend is hard-coded to Consul-Test and a single Vault instance. Can't figure out how pass credentials to the backend and Vault instances securely when there are multiple backends and Vault instances. I don't want tfvars in the project.)