Sceptre: Configure s3 bucket for templates

[cornerman]

I would like to add a lifecycle configuration to the template bucket, because it does not make sense for me to keep old cloudformation templates forever.

Currently, I can only set the template_bucket_name and it will be created automatically if it does not exist. To influence the bucket configuration right now, I could either change the bucket via aws-cli once after it is created or create the bucket myself in a separate cloudformation stack prior to the sceptre deployment.

But I would love to integrate this step this within my sceptre deployment. Does it make sense to add a stack-group config for lifecycle configuration of the template bucket, so it can be done automatically? Or could we aim for a more generic approach where I can provide a cloudformation template for creating the template bucket, like template_bucket_stack: !stack_output some/bucket.yaml::BucketName?

[ngfgrant]

Hey @cornerman

I don't see any problems with adding a config key such as template_bucket which would be passed into creating the bucket or updating it. It would be neat to support something like:

template_bucket:
  name: bucketA
  region: eu-west-1
  acl: private
  lifecycle_configuration:
      - Expiration:
             Days: 10
   ...

if we added support for the key template_bucket and kept template_bucket_name key with the current default behaviour then that would be backward compatible. If config_reader finds template_bucket then it would take precedence over template_bucket_name.

This would also resolve https://github.com/cloudreach/sceptre/pull/728 in a more backward-compatible friendly way. I'm probably going to close out #728 in favour of this suggested implementation - not because #728 is a bad idea but because I don't know what we would end up breaking in others setups with going straight to private buckets, probably best to give it as an option.

[craighurley]

Something worth considering along with this is enabling server side encryption by default as mentioned in #491

got to keep ticking those encryption boxes to keep security folks happy :)

[ngfgrant]

@craighurley absolutely! Well remembered. Could easily incorporate encryption preferences into this new way of specifying config. Think I will close out #491 in favour of this ticket but will make sure to allow setting of encryption here.

[cornerman]

This would also resolve #728 in a more backward-compatible friendly way.

@ngfgrant Agreed, configuring the bucket directly is more generic and enables even more use-cases.

I like the idea of a template_bucket directive in the config and it would cover a lot of configurations :+1:

Just one question: In my setup, I am deploying to two different accounts. Therefore I switch between two different profile and template_bucket_name values in my config, having one private template bucket in each account. So, my default config.yaml on the top-level fixes the profile and template_bucket_name to a specific value and in two places in the hierarchy, I am switching the account. And in both places, I have to reference the same template-bucket, which means I am repeating the naming convention there at two places. With template_bucket, I would need to repeat the whole bucket configuration.

I see three solutions: (1) I restructure my stack-groups, such that I have a different top-level folder for each account in order to define the profile/template_bucket only once there or (2) somehow allow https://github.com/cloudreach/sceptre/issues/729 to reference other stack_group_config values or (3) introduce some kind of template_bucket_path, where I could configure my bucket in a separate file/stack.

What do you think?

[cornerman]

I thought about my setup more and I think the proposal with template_bucket is great. But we should not require a name in the template_bucket, we could use the name from template_bucket_name. This would allow me to define the template_bucket once in the top-most stack-group and override the template_bucket_name in stack-groups beneath, where I switch accounts and need a different bucket-name.

[cornerman]

Would it be possible to somehow make the template_bucket_name a function of the selected AWS profile?

I have code like this in stack-groups that are deployed to different accounts, because I need to keep the bucket name unique:

profile: {{ var.AWS_Profile }}
template_bucket_name: My-{{ var.AWS_Profile }}-cloudformation-templates

What do guys do in these situations? Have cross-account access to one bucket or have a bucket in each account?

[ngfgrant]

I think we are going to the leave the template_bucket_name as it is for v2. I don't want to change the expected behaviour of template_bucket_name. I think we just document that you can either use template_bucket_name or template_bucket in your stacks. If you specify the name in template_bucket then it is the same as specifying the name in template_bucket. Tbh I would probably be up for dropping template_bucket_name in the next major release.

[cornerman]

@ngfgrant I would keep the template_bucket_name, as mentioned, because it allows us to separate the bucket definition (including lifecycle rules and acl) from the name and make it reusable.

I think, it is possible to do this in a backwards compatible way. If there is only template_bucket_name, just do what we do now. If there is a template_bucket, use this definition to create the bucket and use the name from template_bucket_name if no explicit name is provided.

[zaro0508]

@cornerman We have many AWS accounts and we share templates across accounts. We don't rely on sceptre to create the template_bucket_name. Our strategy has been to deploy a base set of CFN templates to each account which creates the template_bucket_name. This allows us to establish multiple S3 repositories of templates which then allows any project to reference any other project's templates.