Terraform: aws_s3_bucket_notification multiple notifications for bucket

[fuCtor]

Try configure multiple notification for bucket. But provider override it by last definition.
In state file I have:

  "aws_s3_bucket_notification.bucket_notify.0": {
      "type": "aws_s3_bucket_notification",
      "depends_on": [
          "aws_s3_bucket.general",
          "aws_sns_topic.exchange"
      ],
      "primary": {
          "id": "test-bucket",
          "attributes": {
              "bucket": "test-bucket",
              "id": "test-bucket",
              "lambda_function.#": "0",
              "queue.#": "0",
              "topic.#": "1",
              "topic.0.events.#": "1",
              "topic.0.events.3356830603": "s3:ObjectCreated:*",
              "topic.0.filter_prefix": "chunks",
              "topic.0.filter_suffix": "",
              "topic.0.id": "s3-chunks-notify",
              "topic.0.topic_arn": "arn:aws:sns:XXX:s3-chunks-create-topic"
          }
      }
  },
  "aws_s3_bucket_notification.bucket_notify.1": {
      "type": "aws_s3_bucket_notification",
      "depends_on": [
          "aws_s3_bucket.general",
          "aws_sns_topic.exchange"
      ],
      "primary": {
          "id": "test-bucket",
          "attributes": {
              "bucket": "test-bucket",
              "id": "test-bucket",
              "lambda_function.#": "0",
              "queue.#": "0",
              "topic.#": "1",
              "topic.0.events.#": "1",
              "topic.0.events.3356830603": "s3:ObjectCreated:*",
              "topic.0.filter_prefix": "trace",
              "topic.0.filter_suffix": "",
              "topic.0.id": "s3-trace-notify",
              "topic.0.topic_arn": "arn:aws:sns:XXX:s3-trace-create-topic"
          }
      }
  },

Config file contains:

resource "aws_s3_bucket_notification" "bucket_notify" {
  count = "${var.topic_count}"
  bucket = "${aws_s3_bucket.general.id}"
  topic {
    id = "s3-${lookup(var.topics, count.index)}-notify"
    topic_arn = "${element(aws_sns_topic.exchange.*.arn, count.index)}"
    events = ["s3:ObjectCreated:*"]
    filter_prefix = "${lookup(var.topics, count.index)}"
  }
}

As you can see, this resource have same ID.

When I try add ID key for resource, receive error:

    Errors:
      * aws_s3_bucket_notification.bucket_notify #0: : invalid or unknown key: id
      * aws_s3_bucket_notification.bucket_notify #1: : invalid or unknown key: id

Version: Terraform v0.6.16

[fuCtor]

Solution:

resource "aws_s3_bucket_notification" "bucket_notify" {
  bucket = "${aws_s3_bucket.general.id}"
  topic {
    id = "s3-${lookup(var.topics, 0)}-notify"
    topic_arn = "${aws_sns_topic.exchange.0.arn}"
    events = ["s3:ObjectCreated:*"]
    filter_prefix = "${lookup(var.topics, 0)}"
  }

  topic {
    id = "s3-${lookup(var.topics, 1)}-notify"
    topic_arn = "${aws_sns_topic.exchange.1.arn}"
    events = ["s3:ObjectCreated:*"]
    filter_prefix = "${lookup(var.topics, 1)}"
  }
}

It is not obvious from the documentation. Only from source.

[catsby]

This is for notifications from a single bucket to go to many topics, correct? I apologize that the documentation wasn't clearer, do you think an example like the one you've shared would be sufficient on our docs page?

[catsby]

Going to close for now though, thanks!

[fuCtor]

Yes, it's correct. I think it will be useful for other.
But I think that will be more comfortable allow to use first case with count attribute. Because it's clearly and safe from mistake.

For example: we have 10 SNS topics and 10 S3 notification. Without count atteribute we must copy-paste same code 10 times and when we change something we must check other parts. It's potential place for mistakes.

Would be great if Terraform will be maintain both versions.

[bschwind]

For anyone as confused as I was about adding multiple events from one bucket to one queue, here is a working JSON config syntax. What wasn't obvious from the docs is that queue can point to an array of event configurations, but once I realized that, everything worked!

{
    "aws_s3_bucket_notification": {
        "s3-upload-events": {
            "bucket": "${aws_s3_bucket.app-bucket.id}",
            "queue": [
                {
                    "events": [
                        "s3:ObjectCreated:*"
                    ],
                    "filter_prefix": "raw_images/",
                    "id": "RawImageUpload",
                    "queue_arn": "${aws_sqs_queue.s3-upload-notifier.arn}"
                },
                {
                    "events": [
                        "s3:ObjectCreated:*"
                    ],
                    "filter_prefix": "raw_videos/",
                    "id": "RawVideoUpload",
                    "queue_arn": "${aws_sqs_queue.s3-upload-notifier.arn}"
                },
                {
                    "events": [
                        "s3:ObjectCreated:*"
                    ],
                    "filter_prefix": "resized_images/",
                    "id": "ResizedImageAdded",
                    "queue_arn": "${aws_sqs_queue.s3-upload-notifier.arn}"
                },
                {
                    "events": [
                        "s3:ObjectCreated:*"
                    ],
                    "filter_prefix": "converted_videos/",
                    "id": "ConvertedVideoAdded",
                    "queue_arn": "${aws_sqs_queue.s3-upload-notifier.arn}"
                }
            ]
        }
    }
}

[seanjoo]

I have a use case where two teams share a bucket but they want to add and manage their own notifications and topics. The topics can't not be combined under one notification object as each team has its own TF template. As a result, when one team destroys and recreates its topics, it destroys the other team's topics as well.

Is this a use case that can not be supported by the current implementation? and would it be ever supported?

[mitchellh]

I'm going to close this out. You found a solution and you're also hitting on a feature request we have elsewhere for being able to duplicate config blocks (basically: primitive loops). So, your request is still valid, but is covered elsewhere, and you otherwise found a way to make this work.

[antonfelich]

I had the same use case as @seanjoo it would be nice to see as an option to support multiple terraform templates creating bucket notifications against a shared bucket without overwriting each other.

In the meantime, I just made a second bucket.

[jufemaiz]

@mitchellh any chance you could link to the "primitive loops" feature request that covers this use case?

https://www.google.com.au/search?q=terraform+%22primitive+loops%22 doesn't really hold much.

Our use case is essentially a storage module (DRY Terraform) with the ability to pass in subscriptions as a variable with a known format. It was working fine while we only had one… but then we had two… and I can't seem to find a way to easily interpolate the base variables into the needed Terraform JSON for this.

[acobaugh]

Just ran into this as well. Sounds like our use case is the same as @seanjoo and @antonfelich. We want to be able to have one bucket with arbitrarily defined SQS queue notifications (multiple queue ARN and filter combinations so). This is to reduce the number of separate S3 buckets we need, as there's a limit per account.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.