Sceptre: Support for templates outside Sceptre project directory

[theseanything]

Currently Sceptre only supports templates located within the current working Sceptre project directory. Providing support to specify a template outside the project directory would allow more flexibility when managing templates. An external templates folder would allow them to be shared between multiple Sceptre projects locally. Additionally the external folder be managed in a separate git repository for sharing and distribution of templates.

Possible Implementation:

Having a external_template_folder property in config.yaml or the proposed sceptre.yaml file. Then all references of template_path in stack configs are relative to that folder. If the external_template_folder property is absent then Sceptre defaults to the templates folder within the current project folder.

[mike-rsi]

Will external_template_folder also be definable/overridable in stack configs? Within the same project, it would be nice if certain stacks could use shared templates (i.e. from external directories) while other stacks use project-specific templates from within the Sceptre project directory. Or maybe even easier, have you thought about just supporting ../ in values for the template_path and not needing a new configuration option to specify a path for external templates?

A less critical thought: if you go with the new configuration option, would the key external_template_directory be a better choice than external_template_folder, seeing as "folder" is typically a GUI-related term?

[johnculkin]

+1 to 'it would be nice if certain stacks could use shared templates (i.e. from external directories) while other stacks use project-specific templates"

[theseanything]

Good point @mike-rsi. Agree with directory instead of folder.

Maybe could have different naming to deal with referencing either local or external directory:

template_path: networking/vpc.yaml

or

external_template_path: networking/vpc.yaml

[theseanything]

Another issue with re-usability is that external_template_directory probably won't be the same for a sceptre project on different local machines. So thinking it might be sensible to make this property set a system-wide level i.e. have an environment_variable SCEPTRE_TEMPLATE_DIR.

[b-t-g]

@theseanything, I've thought about it a little more and I think there may be a usability issue with just having an environment variable to denote a directory containing the desired template.

Consider the following two SCEPTRE_TEMPLATE_DIRs:

.
├── repo1
│   ├── ec2.yaml
│   └── vpc.yaml
└── repo2
    ├── elb.yaml
    └── other_ec2.yaml

and

.
├── project1
│   ├── repo1
│   │   ├── ec2.yaml
│   │   └── vpc.yaml
│   └── repo2
│       ├── elb.yaml
│       └── other_ec2.yaml
└── project2
    └── possibly_other_repos

I'm imaging the case when someone first joins a project and naively clones the repos into their SCEPTRE_TEMPLATE_DIR (the first external template directory) and someone who is possibly on multiple projects (the second external template directory) and organizes the SCEPTRE_TEMPLATE_DIR more carefully.

In this example, where would Sceptre know where to look for the second external directory when we say external_template_directory: ec2.yaml or whatever the desired syntax would be?

Enforcing a consistent style of local directory organization (which ostensibly shouldn't affect functionality) seems a bit heavy-handed to me. Or, worse yet, what if we leave the organization up to teams and different teams have conflicting structures.

Possible solution: Since this feature is supposed facilitate reuse and sharing of templates, there ought to be a source of truth for the templates being shared. Thus, why not just point to that source of truth, store the result locally somewhere, and then have Sceptre proceed as normal from there?

[b-t-g]

If someone is trying to implement this functionality where both repositories are version controlled in git, then submodules or subtree merging may be of interest to you depending on your specific workflow. For Mercurial there is also subrepositories and mixed submodules/subrepositories are supported either via a compatibility layer (as in git-hg) or natively (as in Mercurial)

[whistler]

Another possible solution could be to update template path to support fetching templates from a URL.
e.g.

template_path: https://raw.githubusercontent.com/username/templates/master/vpc.yaml

This way you could support both project-specific templates in the templates directory as well as shared templates stored on git, S3 or an artefact repository.

[jpedro]

The template_path could be resolved from a comma separated list of directories not just a single one, optionally defined in config.yml defaulting to the current ./templates. The code would then resolve the template_path according that list, first-found first-used basis.

Ansible already does this for roles and this allow a hierarchy to be defined, with base templates (shared via a git submodule, a git repo, a common folder location or a remote http location) and local templates overriding locally the shared ones if the order dictate so.

[zaro0508]

How about the idea of having Sceptre support resolvers for the template_path? Then it can be left to others to contribute resolvers for this. I would image it would work similar to what's already implemented for parameters, something like this..

Get template from local template folder:

template_path: !local vpc.yaml

Get template from github:

template_path: !github https://raw.githubusercontent.com/username/templates/master/vpc.yaml

Get template from S3 bucket:

template_path: !S3 https://s3.amazonaws.com/my-bucket/templates/shared/vpc.yaml

another S3 option:

template_path: 
  !S3
    bucket: my-bucket
    path: templates/shared
    object: vpc.yaml

[craighurley]

I have encountered a setup where a single repository contains multiple workloads, each with their own sceptre directory. E.g.

├── workload-a
│   ├── config
│   └── templates
└── workload-b
    ├── config
    └── templates

I'm not a fan of that setup because it limits the ability to easily reuse templates and launch en masse. If it were possible to tell sceptre to use an external template folder, the structure could be changed to this, which would be much better for template reuse:

├── workload-a
│   └── config
├── workload-b
│   └── config
└── templates

[adnanhb]

I can look into resolving this task, can somebody assign this task to me?

[zaro0508]

hey @adnanhb i believe this issue is resolved by PR https://github.com/Sceptre/sceptre/pull/864 . How about just help review and test that one?

[campanellaf]

Any updates on this? Is it possible to refer to a shared folder within the same repo but outside the project?

projects
├── project-a-development
│ ├── config
└── project-a-test
├── config
└── templates

In V1, I was able to refer to project-a-test/templates from project-a-development by using:

template_path: ../project-a-test/templates/template.json

Is it still possible to achieve this in V2? If it is possible, how?

[campanellaf]

Anyone?