Terraform: Recurring plan issue with aws_elastic_beanstalk_environment and aws:autoscaling:launchconfiguration:SecurityGroups setting

[ninjaMog]

Hi there,

Terraform Version

Using Terraform v0.6.14

Affected Resource(s)

aws_elastic_beanstalk_environment

Expected Behavior

When applying custom security groups to instances in a beanstalk using aws_elastic_beanstalk_environment and the "aws:autoscaling:launchconfiguration:SecurityGroups" setting, subsequent plans should show no changes required.

Actual Behavior

Subsequent plans show a change to this setting is needed

Steps to Reproduce

1. Include the following in your beanstalk configuration:

   setting {
   namespace = "aws:autoscaling:launchconfiguration"
   name = "SecurityGroups"
   value = "my_securitygroup_a,my_securitygroup_b"
   }

2. terraform apply
3. terraform plan

You'll see output in the plan like:

setting.1137948120.name:      "SecurityGroups" => ""
setting.1137948120.namespace: "aws:autoscaling:launchconfiguration" => ""
setting.1137948120.value:     "my_securitygroup_a,my_securitygroup_b,awseb-e-mbtpf9p2hy-stack-AWSEBSecurityGroup-OMD0ATLU3CZ8" => ""
...
setting.3267732739.name:      "" => "SecurityGroups"
setting.3267732739.namespace: "" => "aws:autoscaling:launchconfiguration"
setting.3267732739.value:     "" => "my_securitygroup_a,my_securitygroup_b"

Important Factoids

awseb-e-mbtpf9p2hy-stack-AWSEBSecurityGroup-OMD0ATLU3CZ8 will be a security group that AWS EB adds to the instance to allow ingress rules from the loadbalancer & apply rules specified by
aws:autoscaling:launchconfiguration:SSHSourceRestriction

References

This seems a similar issue to the aws:elb:loadbalancer:ManagedSecurityGroup setting issue that was fixed with https://github.com/hashicorp/terraform/pull/4691

I believe terraform needs to ignore the security group added by AWS when doing a compare.

[artburkart]

Not sure if this is considered spam, but I'm experiencing the same issue with ELBSubnets (aws:ec2:vpc) and ELBScheme (aws:ec2:vpc).

[dharrisio]

@ninjaMog @artburkart Thanks for bringing up these issues! There are a few issues like this with different option settings for Elastic Beanstalk. The problem being that the AWS API returns some values in a different format than the submitted value.

The fix will probably be similar to #4691. I should be able to address this one in the next few days, I'll update here with the PR once I have that ready.

[dharrisio]

@ninjaMog This should be fixed once #6491 is merged.

[dharrisio]

@artburkart I wasn't able to reproduce the issue with the ELBSubnets. Do you have an Terraform plan that creates the issue? This is what I was using to try and recreate the issue.

provider "aws" {
  region = "us-east-1"
}

resource "aws_elastic_beanstalk_application" "tftest" {
  name        = "tf-test-migrations"
  description = "tf-test-desc"
}

resource "aws_elastic_beanstalk_environment" "tfenvtest" {
  name                = "tf-test-recurring"
  application         = "${aws_elastic_beanstalk_application.tftest.name}"
  cname_prefix        = "tf-test"
  solution_stack_name = "64bit Amazon Linux 2016.03 v2.1.0 running Go 1.4"

  setting {
    namespace = "aws:ec2:vpc"
    name      = "VPCId"
    value     = "vpc-xxxxxxx"
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBSubnets"
    value     = "subnet-xxxxxxx,subnet-xxxxxxx"
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBScheme"
    value     = "internal"
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "Subnets"
    value     = "subnet-xxxxxxx,subnet-xxxxxxx"
  }
}

[artburkart]

@dharrisio - sorry, I missed your notification somehow (been happening a lot lately 😞).

I'll try to put together an example for you as soon as I can, but first I'll update my terraform install and see if it's still an issue.

[artburkart]

@dharrisio

I just upgraded my terraform install. I'm still seeing the issue.

Here's my resource config:

resource "aws_elastic_beanstalk_application" "app" {
  name = "${aws_security_group.app.name}"
  description = "This is a description"
}

resource "aws_elastic_beanstalk_environment" "app-env" {
  name = "${aws_security_group.app.name}"
  application = "${aws_elastic_beanstalk_application.app.name}"
  tier = "Worker"
  solution_stack_name = "64bit Amazon Linux 2016.03 v2.1.0 running Docker 1.9.1"

  setting = {
    namespace = "aws:elasticbeanstalk:command"
    name = "BatchSize"
    value = "30"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:command"
    name = "BatchSizeType"
    value = "Percentage"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:command"
    name = "DeploymentPolicy"
    value = "Rolling"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sns:topics"
    name = "Notification Endpoint"
    value = "[email protected]"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "ErrorVisibilityTimeout"
    value = "2"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "HttpPath"
    value = "/event-message"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "WorkerQueueURL"
    value = "https://sqs.us-east-1.amazonaws.com/account_id/${var.sqs_queue_name}"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "VisibilityTimeout"
    value = "300"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "HttpConnections"
    value = "10"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "InactivityTimeout"
    value = "299"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:sqsd"
    name = "MimeType"
    value = "application/json"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:environment"
    name = "ServiceRole"
    value = "aws-elasticbeanstalk-service-role"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:environment"
    name = "EnvironmentType"
    value = "LoadBalanced"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:application"
    name = "Application Healthcheck URL"
    value = "/health"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:healthreporting:system"
    name = "AWSEBHealthdGroupId"
    value = "some-id-of-the-thing"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:healthreporting:system"
    name = "SystemType"
    value = "enhanced"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:healthreporting:system"
    name = "HealthCheckSuccessThreshold"
    value = "Ok"
  }

  setting = {
    namespace = "aws:ec2:vpc"
    name = "Subnets"
    value = "${var.private_subnet_d_id},${var.private_subnet_c_id},${var.private_subnet_e_id}"
  }

  setting = {
    namespace = "aws:ec2:vpc"
    name = "VPCId"
    value = "${var.vpc_id}"
  }

  setting = {
    namespace = "aws:ec2:vpc"
    name = "ELBSubnets"
    value = "${var.private_subnet_d_id},${var.private_subnet_c_id},${var.private_subnet_e_id}"
  }

  setting = {
    namespace = "aws:ec2:vpc"
    name = "ELBScheme"
    value = "internal"
  }

  setting = {
    namespace = "aws:ec2:vpc"
    name = "AssociatePublicIpAddress"
    value = "false"
  }

  setting = {
    namespace = "aws:autoscaling:launchconfiguration"
    name = "SecurityGroups"
    value = "${aws_security_group.app.id}"
  }

  setting = {
    namespace = "aws:autoscaling:launchconfiguration"
    name = "IamInstanceProfile"
    value = "${aws_iam_instance_profile.app.name}"
  }

  setting = {
    namespace = "aws:autoscaling:launchconfiguration"
    name = "InstanceType"
    value = "m3.medium"
  }

  setting = {
    namespace = "aws:autoscaling:launchconfiguration"
    name = "EC2KeyName"
    value = "launcher"
  }

  setting = {
    namespace = "aws:autoscaling:updatepolicy:rollingupdate"
    name = "RollingUpdateEnabled"
    value = "false"
  }

  setting = {
    namespace = "aws:elasticbeanstalk:healthreporting:system"
    name = "ConfigDocument"
    value = <<EOF
{"Version":1,"CloudWatchMetrics":{"Instance":{"ApplicationRequestsTotal":60},"Environment":{"ApplicationRequests5xx":60,"ApplicationRequests4xx":60,"ApplicationRequests2xx":60}}}EOF
  }
}

I'm using a lot of variables and it's pretty massive, so it's not entirely clear what the culprit is. Once I've applied, I get these results in my subsequent plan:

...
    setting.id.name:      "" => "ELBSubnets"
    setting.id.namespace: "" => "aws:ec2:vpc"
    setting.id.value:     "" => "subnet-id1,subnet-id2,subnet-id3"
    setting.id.name:      "" => "ELBScheme"
    setting.id.namespace: "" => "aws:ec2:vpc"
    setting.id.value:     "" => "internal"
    setting.id.name:      "" => "AWSEBHealthdGroupId"
    setting.id.namespace: "" => "aws:elasticbeanstalk:healthreporting:system"
    setting.id.value:     "" => "some-id-of-the-thing"
    wait_for_ready_timeout:       "" => "10m"
...

I guess I'm running into issues in more places than I initially thought. And maybe it's not relevant to this ticket, but ConfigDocument always indicates that a change is required, unless I minify the JSON as I did in the example above.

[dharrisio]

@artburkart Thanks for the details, I'll take a closer look at this soon.

[dzhus]

Still happens in Terraform v0.6.16:

In my aws_elastic_beanstalk_environment.ingester I have a section to add an extra security group to EC2 instances:

  # For accessing RDS
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name = "SecurityGroups"
    value = "${aws_security_group.ingester_app.name}"
  }

Running terraform apply changes the environment every time due to changes to SecurityGroups being in the plan:

~ aws_elastic_beanstalk_environment.ingester
    setting.1487836754.name:      "" => "SecurityGroups"
    setting.1487836754.namespace: "" => "aws:autoscaling:launchconfiguration"
    setting.1487836754.value:     "" => "ingester_app"

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.