Terraform-provider-aws: aws_kinesis_firehose_delivery_stream doesn't support error_output_prefix for s3_backup_configuration

[mannbiher]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.11

• provider.aws v1.58.0
  

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream
  extended_s3_configuration
  s3_backup_configuration

Terraform Configuration Files

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "bucket" {
  bucket_prefix = "tf-test-bucket"
  acl           = "private"
}

resource "aws_iam_role" "firehose_role" {
  name = "firehose_test_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role" "lambda_iam" {
  name = "lambda_iam"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_lambda_function" "lambda_processor" {
  filename      = "lambda.zip"
  function_name = "firehose_lambda_processor"
  role          = "${aws_iam_role.lambda_iam.arn}"
  handler       = "exports.handler"
  runtime       = "nodejs8.10"
}

resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
  name        = "cli-kinesis-firehose-extended-s3-test-stream"
  destination = "extended_s3"

  extended_s3_configuration {
    role_arn   = "${aws_iam_role.firehose_role.arn}"
    bucket_arn = "${aws_s3_bucket.bucket.arn}"

    prefix              = "firehose/ts=!{timestamp:yyyyMMdd}/"
    error_output_prefix = "error-firehose/!{firehose:error-output-type}/ts=!{timestamp:yyyyMMdd}/"

    s3_backup_mode = "Enabled"

    s3_backup_configuration {
      role_arn   = "${aws_iam_role.firehose_role.arn}"
      bucket_arn = "${aws_s3_bucket.bucket.arn}"
      prefix     = "raw/ts=!{timestamp:yyyyMMdd}/"
    }

    processing_configuration {
      enabled = "true"

      processors {
        type = "Lambda"

        parameters {
          parameter_name  = "LambdaArn"
          parameter_value = "${aws_lambda_function.lambda_processor.arn}:$LATEST"
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/mannbiher/072f8edefbdb27885f565c94ccaa990c

Panic Output

None

Expected Behavior

Resource aws_kinesis_firehose_delivery_stream is created.

Actual Behavior

1 error(s) occurred:

• aws_kinesis_firehose_delivery_stream.extended_s3_stream: 1 error(s) occurred:

• aws_kinesis_firehose_delivery_stream.extended_s3_stream: error creating Kinesis Firehose Delivery Stream: InvalidArgumentException: Ths supplied prefix(es) do not satisfy the following constraint: ErrorOutputPrefix cannot be null or empty when Prefix contains expressions
  status code: 400, request id: e112aa64-a711-1de6-b5cb-037352f6f25c
  

Steps to Reproduce

1. terraform apply

Important Factoids

Looking at Kinesis Firehose API documentation, S3BackupConfiguration supports ErrorOutputPrefix property. It seems ErrorOutputPrefix need to be set along with Prefix option. Firehose delivery stream creation through AWS cli gives the same error if ErrorOutputPrefix is not specified. Terraform provider doesn't have this property for s3_backup_configuration.

References

• #0000

[coofercat]

Just to say, this also affects un-enhanced S3 streams too (terraform 0.11.13, aws provider 2.2):

resource "aws_kinesis_firehose_delivery_stream" "s3_stream" {
  name        = "FirehoseToS3"
  destination = "s3"

  s3_configuration {
    role_arn           = "${aws_iam_role.firehose_to_s3.arn}"
    bucket_arn         = "${aws_s3_bucket.bucket.arn}"
    compression_format = "GZIP"
    prefix             = "data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"
}

returns:

* aws_kinesis_firehose_delivery_stream.s3_stream: error creating Kinesis Firehose Delivery Stream: InvalidArgumentException: Ths supplied prefix(es) do not satisfy the following constraint: ErrorOutputPrefix cannot be null or empty when Prefix contains expressions
    status code: 400, request id: ddb77615-96e8-196a-88ba-288ceac8d2b0

You can't specify an ErrorOutputPrefix in s3_configuration as it's not a supported attribute (and not shown in the documentation).

I created the Firehose without the prefix, and then used the console to edit it to add one. Adding just the prefix (no error prefix) says "You can't include expressions in the prefix unless you also specify an error prefix.", thus it looks like it's a required attribute when it has certain contents.

[cdaniluk]

Is this still an issue? This works for me:

resource "aws_kinesis_firehose_delivery_stream" "this" {
  name        = "test"
  destination = "extended_s3"

  extended_s3_configuration {
    bucket_arn         = aws_s3_bucket.this.arn
    error_output_prefix = "error/!{firehose:error-output-type}/ts=!{timestamp:yyyyMMdd}/"
    prefix              = "cloudtrail/${local.account_id}/year=!{timestamp:YYYY}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}"
    role_arn            = aws_iam_role.this.arn
    }
  }

  kinesis_source_configuration {
    kinesis_stream_arn = aws_kinesis_stream.this.arn
    role_arn           = aws_iam_role.this.arn
  }
}

[Obsecurus]

@cdaniluk your reference above is for the normal S3 output not the S3 source record backup folder (prior to Lambda transformation or conversion to parquet/ORC)

Are there any updates on this issue? I'm getting ready to wrap our firehose streams in CI with terraform and this currently forces a 10-15 manual task any time terraform apply runs. I suppose a workaround could be running aws firehose update-destination for all of the streams but will require a good bit of JSON templating, etc.