Terraform: Terraform remote-exec and file are using private IP addresses for connection

Created on 28 Apr 2016  ·  11Comments  ·  Source: hashicorp/terraform

Terraform Version

Terraform v0.6.15

Affected Resource(s)

  • aws_instance

    Terraform Configuration Files

/*******************************
* Kubernetes build server
********************************/
variable "env_file" {
    default = "k8s-bootstrap.env"
}
variable "instancefile" {
    default = ".instance.id"
}
variable "ssh_key" {
    default = "xxx-key.pem"
}

resource "aws_instance" "k8s-bootstrap" {
    ami = "${var.images.us-east-1}"
    instance_type = "t2.medium"
    subnet_id = "subnet-xxxxxx"
    vpc_security_group_ids = [ "sg-xxxxxxx","sg-yyyyyy" ]
    tags {
        Name = "k8s-bootstrap"
    }
    key_name = "deployment-key"
    provisioner "local-exec" {
        command = "echo export K8S_ENV_NAME=\\\"10point1\\\" AWS_S3_REGION=\\\"US Standard\\\" >${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export KUBE_AWS_ZONE=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.availability_zone}\\\" >>${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export AWS_S3_BUCKET=\\\"kubernetes-artifacts-'$K8S_ENV_NAME'\\\" >>${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export NUM_NODES=3 KUBE_OS_DISTRIBUTION=\\\"jessie\\\" >>${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export INSTANCE_PREFIX=\\\"'$K8S_ENV_NAME'\\\" KUBE_AWS_INSTANCE_PREFIX=\\\"'$K8S_ENV_NAME'\\\" >>${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export AWS_SSH_KEY='$HOME/.ssh/deployment_key_rsa' >>${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export SERVICE_CLUSTER_IP_RANGE=\\\"10.0.0.0/16\\\" CLUSTER_IP_RANGE=\\\"10.244.0.0/16\\\" MASTER_IP_RANGE=\\\"10.246.0.0/24\\\" >>${var.env_file}"
    }
    provisioner "local-exec" {
        command = "echo export KUBE_SUBNET_CIDR=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.cidr_block}\\\" >>${var.env_file}"
    }
    provisioner "remote-exec" {
        inline = [
           "echo ${aws_instance.k8s-bootstrap.id} >${var.instancefile}",
           "apt-get install awscli s3cmd",
           "s3cmd --access_key=${var.access_key} --secret_key=${var.secret_key} --ssl get s3://k8s-env/${aws_instance.k8s-bootstrap.id}",
           "mkdir .ssh",
           "chmod 600 .ssh",
           "s3cmd --access_key=${var.access_key} --secret_key=${var.secret_key} --ssl get s3://k8s-env/${var.ssh_key} .ssh/id_rsa",
           "s3cmd --access_key=${var.access_key} --secret_key=${var.secret_key} --ssl -r get s3://k8s-env/${var.account_id} ~/.aws",
           "git clone [email protected]:10point1/k8s_build_script.git",
           "cd k8s_build_script",
           "chmod 755 k8s_env.sh",
           "./k8s_env.sh ~/${aws_instance.k8s-bootstrap.id} up >k8s_installation.log 2>&1"
        ]
    }
}
resource "aws_eip" "k8s-bootstrap-IP" {
    //network_interface = "${aws_network_interface.k8s-bootstrap-net.id}"
    instance = "${aws_instance.k8s-bootstrap.id}"
    vpc = true
}
resource "aws_security_group" "ssh-in" {
  name = "ssh-in"
  description = "Allow http(s) and ssh traffic"
  vpc_id = "vpc-1ff9087a"
  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = [ "0.0.0.0/0" ]
  }
  tags {
    Name = "ssh-in"
  }
}
resource "aws_s3_bucket" "k8s-env" {
    bucket = "k8s-env"
    acl = "private"

    tags {
        Name = "k8s-env"
        Environment = "Dev"
    }
}
resource "aws_s3_bucket_object" "object" {
    bucket = "${aws_s3_bucket.k8s-env.bucket}"
    key = "${aws_instance.k8s-bootstrap.id}"
    source = "${var.env_file}"
    etag = "${md5(file("${var.env_file}"))}"
}

Debug Output

https://gist.github.com/polasekr/239d987e061f11b8104360c04fc8d0e9

Panic Output

n/a

Expected Behavior

As shown in the output remote-exec provisioner is trying to connect to 172.16.1.84 (private IP address) via ssh while my expectation is that it should be using EC2 instance public IP address for this connection. I checked the documentation but I haven't found a way to control what IP address is used for the connection. This is crucial, since you are very likely building your AWS infrastructure from outside AWS cloud. I also checked the instance created by Terraform and it doesn't have public IP address yet. It appears that public IP allocation is happening after build completion.

Actual Behavior

Instance build never gets completed.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Important Factoids

References

documentation provisioneremote-exec
Source
picture polasekr

Most helpful comment

Hi @polasekr!

The connection block is the intended way to control how the provisioners connect to your instance(s).

You can use interpolation with the special self object to access the properties of the instance being provisioned, including the public IP address, like this:

connection {
  host = "${self.public_ip}"
  # ... (and any other settings you need)
}

Terraform tries to provide some reasonable defaults, but in general I find that I almost always use a connection block for one reason or another. For example, authentication generally requires connection block attributes too.

picture apparentlymart on 28 Apr 2016
👍21

All 11 comments

Hi @polasekr!

The connection block is the intended way to control how the provisioners connect to your instance(s).

You can use interpolation with the special self object to access the properties of the instance being provisioned, including the public IP address, like this:

connection {
  host = "${self.public_ip}"
  # ... (and any other settings you need)
}

Terraform tries to provide some reasonable defaults, but in general I find that I almost always use a connection block for one reason or another. For example, authentication generally requires connection block attributes too.

apparentlymart picture apparentlymart on 28 Apr 2016
👍21

Hi Martin

Thank you for the prompt response.
I tried what you suggested, but now provisioner is trying to establish a connection with a host which is undefined (empty). I again checked the AWS instance created and it didn't have public IP address. 

resource "aws_instance" "k8s-bootstrap" {
    ami = "${var.images.us-east-1}"
    instance_type = "t2.medium"
    subnet_id = "subnet-d9b3989f"
    vpc_security_group_ids = [ "sg-f9c8d281","sg-a8a073cd" ]
    //user_data = "${file("userdate-k8s-bootstrap.sh")}"
    tags {
        Name = "k8s-bootstrap"
    }
    key_name = "deployment-key"
    provisioner "local-exec" {
        command = "echo export K8S_ENV_NAME=\\\"${var.k8s_env_name}\\\" AWS_S3_REGION=\\\"US Standard\\\" >${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export KUBE_AWS_ZONE=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.availability_zone}\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export AWS_S3_BUCKET=\\\"kubernetes-artifacts-'$K8S_ENV_NAME'\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export NUM_NODES=3 KUBE_OS_DISTRIBUTION=\\\"jessie\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export INSTANCE_PREFIX=\\\"'$K8S_ENV_NAME'\\\" KUBE_AWS_INSTANCE_PREFIX=\\\"'$K8S_ENV_NAME'\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export AWS_SSH_KEY='$HOME/.ssh/deployment_key_rsa' >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export SERVICE_CLUSTER_IP_RANGE=\\\"10.0.0.0/16\\\" CLUSTER_IP_RANGE=\\\"10.244.0.0/16\\\" MASTER_IP_RANGE=\\\"10.246.0.0/24\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export KUBE_SUBNET_CIDR=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.cidr_block}\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }

    provisioner "local-exec" {
        command = "echo export KUBE_SUBNET_CIDR=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.cidr_block}\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export KUBE_SUBNET_CIDR=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.cidr_block}\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "local-exec" {
        command = "echo export KUBE_SUBNET_CIDR=\\\"${aws_subnet.k8s-172-31-1-0-s-subnet-1a.cidr_block}\\\" >>${var.env_file}-${var.k8s_env_name}.env"
    }
    provisioner "remote-exec" {
        inline = [
          "echo ${aws_instance.k8s-bootstrap.id} >${var.instancefile}",
          "apt-get install awscli s3cmd",
          "s3cmd --access_key=${var.access_key} --secret_key=${var.secret_key} --ssl get s3://k8s-env/${aws_instance.k8s-bootstrap.id}",
          "mkdir .ssh",
          "chmod 600 .ssh",
          "s3cmd --access_key=${var.access_key} --secret_key=${var.secret_key} --ssl get s3://k8s-env/${var.ssh_key} .ssh/id_rsa",
          "s3cmd --access_key=${var.access_key} --secret_key=${var.secret_key} --ssl -r get s3://k8s-env/${var.account_id} ~/.aws",
          "git clone [email protected]:10point1/k8s_build_script.git",
          "cd k8s_build_script",
          "chmod 755 k8s_env.sh",
          "./k8s_env.sh ~/${aws_instance.k8s-bootstrap.id} up >k8s_installation.log 2>&1"
        ]
        connection {
            user = "ubuntu"
            private_key = "${file("/home/robert/.ssh/deployment.pem")}"
            host = "${self.public_ip}"
        }
    }
}

robert@razer-laptop:~/terraform_10point1$ terraform apply
aws_s3_bucket.k8s-env: Refreshing state... (ID: k8s-env)
aws_vpc.kubernetes-vpc-172-31-0-0: Refreshing state... (ID: vpc-acc42dcb)
aws_iam_user.devops-admin: Refreshing state... (ID: devops-admin)
aws_security_group.ssh-in: Refreshing state... (ID: sg-f9c8d281)
aws_iam_user.devops-svc: Refreshing state... (ID: devops-svc)
aws_iam_policy.ReadOnly_policy_for_all_objects: Refreshing state... (ID: arn:aws:iam::266344612010:policy/ReadOnly_policy_for_all_objects)
aws_iam_access_key.devops-svc-accesskey: Refreshing state... (ID: AKIAI3UOIQLHAGYD6VYA)
aws_iam_policy_attachment.readonly-access-attachement: Refreshing state... (ID: readonly-access-attachement)
aws_iam_group_membership.admins: Refreshing state... (ID: admin)
aws_iam_access_key.devops-admin-accesskey: Refreshing state... (ID: AKIAJ3565PWT2C7ORXPQ)
aws_internet_gateway.kubernetes-vpc-172-31-0-0-igw: Refreshing state... (ID: igw-cbba23af)
aws_subnet.k8s-172-31-1-0-s-subnet-1a: Refreshing state... (ID: subnet-03c6a25b)
aws_vpc_peering_connection.peer-infra-k8s-172-31-0-0: Refreshing state... (ID: pcx-d69926bf)
aws_security_group.ssh-http-in: Refreshing state... (ID: sg-f8d7cd80)
aws_s3_bucket_object.object: Refreshing state... (ID: k8s-bootstrap-10point1)
aws_eip.k8s-bootstrap-IP: Refreshing state... (ID: eipalloc-f6bd2890)
aws_route_table.K8s-route-table-172-31-1-0: Refreshing state... (ID: rtb-462c4e21)
aws_main_route_table_association.k8s-31-main-route-table: Refreshing state... (ID: rtbassoc-1b4c8c7d)
aws_instance.k8s-bootstrap: Creating...
  ami:                               "" => "ami-6610390c"
  availability_zone:                 "" => "<computed>"
  ebs_block_device.#:                "" => "<computed>"
  ephemeral_block_device.#:          "" => "<computed>"
  instance_state:                    "" => "<computed>"
  instance_type:                     "" => "t2.medium"
  key_name:                          "" => "deployment-key"
  placement_group:                   "" => "<computed>"
  private_dns:                       "" => "<computed>"
  private_ip:                        "" => "<computed>"
  public_dns:                        "" => "<computed>"
  public_ip:                         "" => "<computed>"
  root_block_device.#:               "" => "<computed>"
  security_groups.#:                 "" => "<computed>"
  source_dest_check:                 "" => "1"
  subnet_id:                         "" => "subnet-d9b3989f"
  tags.#:                            "" => "1"
  tags.Name:                         "" => "k8s-bootstrap"
  tenancy:                           "" => "<computed>"
  vpc_security_group_ids.#:          "" => "2"
  vpc_security_group_ids.2909197702: "" => "sg-f9c8d281"
  vpc_security_group_ids.3864581758: "" => "sg-a8a073cd"
aws_instance.k8s-bootstrap: Still creating... (10s elapsed)
aws_instance.k8s-bootstrap: Still creating... (20s elapsed)
aws_instance.k8s-bootstrap: Still creating... (30s elapsed)
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export K8S_ENV_NAME=\"10point1\" AWS_S3_REGION=\"US Standard\" >k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export KUBE_AWS_ZONE=\"us-east-1a\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export AWS_S3_BUCKET=\"kubernetes-artifacts-'$K8S_ENV_NAME'\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export NUM_NODES=3 KUBE_OS_DISTRIBUTION=\"jessie\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export INSTANCE_PREFIX=\"'$K8S_ENV_NAME'\" KUBE_AWS_INSTANCE_PREFIX=\"'$K8S_ENV_NAME'\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export AWS_SSH_KEY='$HOME/.ssh/deployment_key_rsa' >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export SERVICE_CLUSTER_IP_RANGE=\"10.0.0.0/16\" CLUSTER_IP_RANGE=\"10.244.0.0/16\" MASTER_IP_RANGE=\"10.246.0.0/24\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export KUBE_SUBNET_CIDR=\"172.31.1.0/24\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export KUBE_SUBNET_CIDR=\"172.31.1.0/24\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export KUBE_SUBNET_CIDR=\"172.31.1.0/24\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'local-exec'...
aws_instance.k8s-bootstrap (local-exec): Executing: /bin/sh -c "echo export KUBE_SUBNET_CIDR=\"172.31.1.0/24\" >>k8s-bootstrap-10point1.env"
aws_instance.k8s-bootstrap: Provisioning with 'remote-exec'...
aws_instance.k8s-bootstrap (remote-exec): Connecting to remote host via SSH...
aws_instance.k8s-bootstrap (remote-exec):   Host:
aws_instance.k8s-bootstrap (remote-exec):   User: ubuntu
aws_instance.k8s-bootstrap (remote-exec):   Password: false
aws_instance.k8s-bootstrap (remote-exec):   Private key: true
aws_instance.k8s-bootstrap (remote-exec):   SSH Agent: true
aws_instance.k8s-bootstrap (remote-exec): Connecting to remote host via SSH...
aws_instance.k8s-bootstrap (remote-exec):   Host:
aws_instance.k8s-bootstrap (remote-exec):   User: ubuntu
aws_instance.k8s-bootstrap (remote-exec):   Password: false
aws_instance.k8s-bootstrap (remote-exec):   Private key: true
aws_instance.k8s-bootstrap (remote-exec):   SSH Agent: true
aws_instance.k8s-bootstrap: Still creating... (40s elapsed)
aws_instance.k8s-bootstrap (remote-exec): Connecting to remote host via SSH...
aws_instance.k8s-bootstrap (remote-exec):   Host:
aws_instance.k8s-bootstrap (remote-exec):   User: ubuntu
aws_instance.k8s-bootstrap (remote-exec):   Password: false
aws_instance.k8s-bootstrap (remote-exec):   Private key: true
aws_instance.k8s-bootstrap (remote-exec):   SSH Agent: true
aws_instance.k8s-bootstrap (remote-exec): Connecting to remote host via SSH...
aws_instance.k8s-bootstrap (remote-exec):   Host:
aws_instance.k8s-bootstrap (remote-exec):   User: ubuntu
aws_instance.k8s-bootstrap (remote-exec):   Password: false
aws_instance.k8s-bootstrap (remote-exec):   Private key: true
aws_instance.k8s-bootstrap (remote-exec):   SSH Agent: true
^CInterrupt received. Gracefully shutting down...
^CTwo interrupts received. Exiting immediately. Note that data
loss may have occurred.
polasekr picture polasekr on 28 Apr 2016

havent tried this my self but maybe try using the eip public_ip instead ${aws_eip.k8s-bootstrap-IP.public_ip} in the connection block. From the documentation "If you are using an aws_eip with your instance, you should refer to the EIP's address directly and not use public_ip, as this field will change after the EIP is attached."

HX-Rd picture HX-Rd on 2 May 2016

@polasekr sorry I didn't notice in your first message that you are using an aws_eip resource to assign an elastic IP to the instance.

You suspected correctly that the aws_eip resource won't be handled until the aws_instance resource is completed, and provisioning is part of creating the resource.

I think you have a few options here to work around this:

  • Use associate_public_ip_address on your aws_instance to allocate it a regular (non-elastic IP) public IP address during provisioning, which I think will then change to the EIP once it's handled.
  • Move your provisioner and connection blocks into the aws_eip resource rather than the aws_instance resource, so that the provisioning will happen only after the Elastic IP has attached, and will use the public_ip attribute of aws_eip instead of the EC2 instance.

The latter is probably the pattern we should describe in the docs, since I think it's the most robust workaround even though it feels a little strange to be "provisioning" an elastic IP address.

apparentlymart picture apparentlymart on 2 May 2016
👍10

@apparentlymart

Thank you for your help. The solution of moving provisioner to aws_eip worked like a charm.

polasekr picture polasekr on 2 May 2016

I've re-tagged this as documentation to turn this ticket into a request to document that workaround. :)

Thanks!

apparentlymart picture apparentlymart on 3 May 2016

@apparentlymart moving provisioner and connection to the aws_eip resource did not work for me, but moving them to aws_eip_association did.

andrewrynhard picture andrewrynhard on 10 Jan 2017

In case any other newbs like myself end up here, this worked for me:

resource "aws_instance" "bastion" {
  instance_type          = "t3.micro"
  ami                    = "${var.amis["ecs"]}"
  vpc_security_group_ids = ["${aws_security_group.bastion.id}"]
  key_name               = "us-east-2-root"
  subnet_id              = "${aws_subnet.public.*.id[0]}"

  tags = {
    Name = "${var.name}-bastion"
  }
}

resource "aws_eip" "bastion" {
  connection {
    user = "ec2-user"
    host = "${aws_eip.bastion.public_ip}"
  }

  instance = "${aws_instance.bastion.id}"

  tags = {
    Name = "${var.name}-bastion-public"
  }

  provisioner "remote-exec" {
    inline = ["sudo yum update -y"]
  }
}

i think what i didnt realize in the past 24 hours of reading about terraform is that a resource can refer to its own computed attributes in the way the connection -> host attr does with host = "${aws_eip.bastion.public_ip}".

chris-brace picture chris-brace on 12 Mar 2019

Facing similar issue with azure now.
"${self.public_ip}" not working
Neither
${azurerm_public_ip.publicIPresource.ip_address}

ukreddy-erwin picture ukreddy-erwin on 12 Jun 2020

I am going to close this issue due to inactivity and age.

If there is still a question, I recommend the the community forum, where there are far more people available to help. If there is a bug or you would like to make a feature request, please open a new issue and fill out the template.
Thanks!

mildwonkey picture mildwonkey on 27 Aug 2020

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashibot[bot] picture hashibot[bot] on 13 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

amaczuga picture amaczuga  ·  124Comments
dupuy picture dupuy  ·  61Comments
glenjamin picture glenjamin  ·  112Comments
ncraike picture ncraike  ·  77Comments
jszwedko picture jszwedko  ·  77Comments