Terraform-provider-aws: aws_s3_bucket_object: support for directory uploads

[ddcprg]

Allow recursively upload of directories to S3.

Terraform Version

All

Affected Resource(s)

• aws_s3_bucket_object

Terraform Configuration Files

resource "aws_s3_bucket_object" "directory" {
  bucket = "your_bucket_name"
  key    = "base_s3_key"
  source = "path/to/directory"
}

Expected Behavior

The resource should upload all files in path/to/directory under s3://your_bucket_name/base_s3_key/, e.g. given my-dir with the following structure:

   |- file_1
   |- dir_a
   |     |- file_a_1
   |     |- file_a_2
   |- dir_b
   |     |- file_b_1
   |- dir_c

One would end with the following S3 objects:

s3://your_bucket_name/base_s3_key/file_1`
s3://your_bucket_name/base_s3_key/dir_a/file_a_1`
s3://your_bucket_name/base_s3_key/dir_a/file_a_2`
s3://your_bucket_name/base_s3_key/dir_b/file_b_1`

Actual Behavior

Directory uploads are not supported

[simondiep]

For when you don't know all the files you want to upload, is the best alternative at the moment to have a null resource provisioner run the AWS CLI to upload your directory?

resource "null_resource" "remove_and_upload_to_s3" {
  provisioner "local-exec" {
    command = "aws s3 sync ${path.module}/s3Contents s3://${aws_s3_bucket.site.id}"
  }
}

[bassmanitram]

@simondiep That works (perfectly I might add - we use it in dev) if the environment in which Terraform is running has the AWS CLI installed. However, in "locked down" environments, and any running the stock terraform docker, it isn't (and in SOME lockdowns, the local-exec provisioner isn't even present) so a solution that sits inside of Terraform would be more robust.

Personally I'd like to see the "source" support URLs, including S3 and GIT URLs (with some sort of authentication scheme too) ... and even the fetching of a zip file from a URL and uploading the _content_ of the zip to the bucket.

THAT we currently do (in our less locked-down environments) as follows:

resource "null_resource" "upload_to_s3" {
  provisioner "local-exec" {
    command = <<EOF
curl --output /tmp/bucket-content.zip -u ${var.zip_file_user}:${var.zip_file_password} -O ${var.zip_file_url} &&
ls -ltr /tmp &&
mkdir -p /tmp/bucket-content &&
cd /tmp/bucket-content &&
unzip -o /tmp/bucket-content.zip &&
aws s3 sync . s3://${var.bucket_id} &&
cd .. &&
rm -rf /tmp/bucket-content

EOF
  }
}

[lonnblad]

I have a need for this as well and have been using the solution @simondiep described for a few months.

I am new to Terraform and was interested in how providers and Terraform works internally, so I went and created a new resource, the aws_s3_bucket_directory, it only supports some basic configuration, just to get something started.

example from localhost

example from terraform plan

+ module.base.module.web.aws_s3_bucket_directory.monkeys
    id:                            <computed>
    bucket:                        "my_bucket"
    etag:                          <computed>
    files.#:                       "2"
    files.1029329135.content_type: "application/javascript"
    files.1029329135.etag:         "1ea8de965fd20f5db9274c5a855d6e04"
    files.1029329135.source:       "build/index.js"
    files.1029329135.target:       "monkey/index.js"
    files.935355054.content_type:  "text/html; charset=utf-8"
    files.935355054.etag:          "64f42acd82c71664feba0c2f972ee408"
    files.935355054.source:        "build/index.html"
    files.935355054.target:        "monkey/index.html"
    source:                        "build"
    target:                        "monkey"

It is still a work in progress, but will create a PR if it would be interesting.

[tomelliff]

@lonnblad It might be worth raising a WIP PR and linking back to this issue so that people can see progress and offer code review on it, particularly if this is your first contribution to the AWS provider.

[lonnblad]

@tomelliff Yes, thank you, I will do so.

[yardensachs]

This is also important for aws_transfer_server.
A SSH user cannot access the SFTP/s3 bucket without the an existing folder structure.

So if you have a user called "dave" and a home folder "home"

You must have /home/dave/ created so dave could access the sftp service.

[yardensachs]

I was able to do this:

resource "aws_s3_bucket_object" "home_folder" {
  bucket  = "${aws_s3_bucket.this.id}"
  key     = "/home/"
  content = "/"
}

[yardensachs]

@ddcprg

This works:

resource "aws_s3_bucket_object" "this" {
  bucket  = "mybucket"
  key     = "/base_s3_key/dir_a/file_a_1/"
  content = "/"
}

[deitch]

@yardensachs that creates the directory, but does not recursively upload the files underneath it.

[lonnblad]

@ddcprg @tomelliff added a WIP PR if you want to have a look

[jonathanhle]

If it helps, my setup is a little different - needed to assume some roles, so I ended up using local-exec with a py script to upload the directories: https://gist.github.com/jonathanhle/6e327c827b2694bc3103f835984d6ed4.

Looking forwards to aws_s3_bucket_directory moving forwards...would love to get rid of my local-execs.

[rishiloyola]

Bump 😄

[bflad]

Hi folks 👋

Just to provide an update here, Terraform v0.12.8 will include a new fileset() function, which will accept a path and glob pattern as inputs and output a set of matching path names. This can be combined with the resource for_each functionality available in Terraform v0.12.6 and later to dynamically generate existing Terraform resources based on directories of files.

# Not implemented yet -- Terraform v0.12.8 design sketch
# Functionality and syntax may change during development

# Given the file structure from the initial issue:
# my-dir
#    |- file_1
#    |- dir_a
#    |     |- file_a_1
#    |     |- file_a_2
#    |- dir_b
#    |     |- file_b_1
#    |- dir_c
# And given the expected behavior of the base_s3_key prefix in the initial issue

resource "aws_s3_bucket_object" "example" {
  for_each = fileset(path.module, "my-dir/**/file_*")

  bucket = aws_s3_bucket.example.id
  key    = replace(each.value, "my-dir", "base_s3_key")
  source = each.value
}

This issue will be updated again when Terraform v0.12.8 releases. 👍

[bflad]

Hello again, just writing in that the fileset() function is now available via Terraform v0.12.8, released yesterday.

Here's a full example. 😄

Given the following file layout:

.
├── main.tf
├── subdirectory1
│   ├── anothersubdirectory1
│   │   └── anothersubfile.txt
│   ├── subfile1.txt
│   └── subfile2.txt
└── subdirectory2
    └── subfile3.txt

And the following Terraform configuration:

terraform {
  required_providers {
    aws = "2.26.0"
  }
  required_version = "0.12.8"
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "test" {
  acl           = "private"
  bucket_prefix = "fileset-testing"
}

resource "aws_s3_bucket_object" "test" {
  for_each = fileset(path.module, "**/*.txt")

  bucket = aws_s3_bucket.test.bucket
  key    = each.value
  source = "${path.module}/${each.value}"
}

output "fileset-results" {
  value = fileset(path.module, "**/*.txt")
}

Terraform successfully maps this file structure into S3:

$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.test will be created
  + resource "aws_s3_bucket" "test" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = (known after apply)
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = "fileset-testing"
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }
    }

  # aws_s3_bucket_object.test["subdirectory1/anothersubdirectory1/anothersubfile.txt"] will be created
  + resource "aws_s3_bucket_object" "test" {
      + acl                    = "private"
      + bucket                 = (known after apply)
      + content_type           = (known after apply)
      + etag                   = (known after apply)
      + id                     = (known after apply)
      + key                    = "subdirectory1/anothersubdirectory1/anothersubfile.txt"
      + server_side_encryption = (known after apply)
      + source                 = "./subdirectory1/anothersubdirectory1/anothersubfile.txt"
      + storage_class          = (known after apply)
      + version_id             = (known after apply)
    }

  # aws_s3_bucket_object.test["subdirectory1/subfile1.txt"] will be created
  + resource "aws_s3_bucket_object" "test" {
      + acl                    = "private"
      + bucket                 = (known after apply)
      + content_type           = (known after apply)
      + etag                   = (known after apply)
      + id                     = (known after apply)
      + key                    = "subdirectory1/subfile1.txt"
      + server_side_encryption = (known after apply)
      + source                 = "./subdirectory1/subfile1.txt"
      + storage_class          = (known after apply)
      + version_id             = (known after apply)
    }

  # aws_s3_bucket_object.test["subdirectory1/subfile2.txt"] will be created
  + resource "aws_s3_bucket_object" "test" {
      + acl                    = "private"
      + bucket                 = (known after apply)
      + content_type           = (known after apply)
      + etag                   = (known after apply)
      + id                     = (known after apply)
      + key                    = "subdirectory1/subfile2.txt"
      + server_side_encryption = (known after apply)
      + source                 = "./subdirectory1/subfile2.txt"
      + storage_class          = (known after apply)
      + version_id             = (known after apply)
    }

  # aws_s3_bucket_object.test["subdirectory2/subfile3.txt"] will be created
  + resource "aws_s3_bucket_object" "test" {
      + acl                    = "private"
      + bucket                 = (known after apply)
      + content_type           = (known after apply)
      + etag                   = (known after apply)
      + id                     = (known after apply)
      + key                    = "subdirectory2/subfile3.txt"
      + server_side_encryption = (known after apply)
      + source                 = "./subdirectory2/subfile3.txt"
      + storage_class          = (known after apply)
      + version_id             = (known after apply)
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_s3_bucket.test: Creating...
aws_s3_bucket.test: Creation complete after 2s [id=fileset-testing20190905121318114700000001]
aws_s3_bucket_object.test["subdirectory2/subfile3.txt"]: Creating...
aws_s3_bucket_object.test["subdirectory1/subfile1.txt"]: Creating...
aws_s3_bucket_object.test["subdirectory1/anothersubdirectory1/anothersubfile.txt"]: Creating...
aws_s3_bucket_object.test["subdirectory1/subfile2.txt"]: Creating...
aws_s3_bucket_object.test["subdirectory2/subfile3.txt"]: Creation complete after 0s [id=subdirectory2/subfile3.txt]
aws_s3_bucket_object.test["subdirectory1/subfile2.txt"]: Creation complete after 0s [id=subdirectory1/subfile2.txt]
aws_s3_bucket_object.test["subdirectory1/subfile1.txt"]: Creation complete after 0s [id=subdirectory1/subfile1.txt]
aws_s3_bucket_object.test["subdirectory1/anothersubdirectory1/anothersubfile.txt"]: Creation complete after 0s [id=subdirectory1/anothersubdirectory1/anothersubfile.txt]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

fileset-results = [
  "subdirectory1/anothersubdirectory1/anothersubfile.txt",
  "subdirectory1/subfile1.txt",
  "subdirectory1/subfile2.txt",
  "subdirectory2/subfile3.txt",
]

For any bug reports or feature requests with fileset() functionality, please file issues upstream in Terraform. Otherwise for general questions about this functionality, please reach out on the community forums. Enjoy! 🎉

[ajorpheus]

I can confirm that this is working fine with v0.12.8
Thank you very much !

[shaiss]

confirmed this works with the newer aws provider
terraform { required_providers { aws = "2.26.0" } required_version = "0.12.8" }

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!