Sceptre: Ability to have template_path to be S3 paths

[emiliogarcia]

Ability to support template_path not to be just a local file but also a file hosted in S3.

[theseanything]

Makes sense. Curious, can I ask how come the templates are getting uploaded separately?

[emiliogarcia]

The templates are uploaded into S3 like if they were artifacts on a previous step. Then templates can be consumed by following a standard in the notation of S3 location.

eg: s3:://templates-repo/module1/v1.3/template.yaml

[theseanything]

Right, but whats the reasoning to uploading the templates separately? Are you not able to store the templates locally in a git repo and let sceptre deal with the S3 upload?

[rdkr]

In our use case, the templates are stored in a git repo and are tested and released to S3 as part of a CD pipeline to ensure they are known to be working and can be safely used elsewhere. This means we already have the templates in S3 and must download them so that sceptre can upload them again.

[theseanything]

@rdkr, why not make the git repo where you store your templates - the Sceptre project repo? Personally, code is far better managed in a git repository than S3.

But otherwise I'm happy for someone to propose a PR with this functionality. I think the path should be stated like
template_object: bucket_name/key/to/object.ext

(Maybe not template_object as the attribute name - open to suggestions)

[mechtron]

@theseanything Another use case for loading templates from S3 is working around CloudFormation's 51200 byte limit on templateBodys. If this feature existed, we could:

1. Generate a JSON CloudFormation template with sceptre generate-template ..
2. Upload generated template to S3
3. Deploy it with sceptre launch-env --s3-template s3:://templates/template.json (or equivalent)

[jamesroutley]

@mechtron If you specify the template_bucket_name parameter in your environment config, we handle the generation, upload and deployment from S3 for you.

[mechtron]

@jamesroutley Excellent thanks so much!

[zaro0508]

@jamesroutley I see that templates can be automatically uploaded to S3 but it doesn't seem like the docs mention that sceptre can deploy templates referenced directly in an S3 bucket. What would you specify for the template_path if you want to reference a bucket, something like s3://org.acme.mybucket/foo-template.yaml?

[leonrodenburg]

Is there any news on this? I would be happy to create a PR that would add support for something like s3://<bucket-name>/<template> in the template_path. In our case we want it because we have multiple microservices that are developed by separate teams. The 'golden' templates are managed by a central infra team. Every microservice has its own sceptre config directory as part of its repository, next to the code of the service itself. When a new service is started, team members have to copy-paste a lot of templates from a central location, without any versioning guarantees whatsoever. They might also be tempted to tweak the templates, which can turn out disastrous.

It would be really cool to be able to develop, test and deploy a set of versioned templates to an S3 bucket that teams can reference.

[zaro0508]

@leonrodenburg The sceptre community would mostly definitely welcome a PR for this. A more general discussion regarding this issue is in #213