Sceptre: Allow hooks to calculate their arguments using resolvers

[sodle]

I want to be able to use a resolver (i.e. a CloudFormation stack output or a local environment variable) to calculate the arguments to a hook. For example:

# After creating our Aurora database, restore it from a SQL dump
# using the hostname and port number returned by the stack
hooks:
   after_create:
    - !cmd "mysql -h !stack_output OurStack::DBHost -P !stack_output OurStack::DBPort < db.sql"

# After updating our EC2 instance, curl the webserver to ensure that it is still up
# using the public IP returned by the stack
hooks:
  after_update:
    - !cmd "curl !stack_output OurStack::WebserverIP"

Currently, these resolvers are not evaluated by the hook; instead, their literal text is executed, causing the command to fail.

[ngfgrant]

Thanks for the request @sodle. Will see if someone can pick this soon.

[m1keil]

Hey @ngfgrant, I'm interested in similar feature as well and willing to commit some time to investigate.
It seems like two design decisions:

1. All current resolvers only refer to other stacks, should we have a resolver that is able to refer to the outputs of "this" stack? In which case, this resolver should only be allowed to run in the after_* hooks.
2. Should we really use Resolvers here? Maybe a new construct such as "Outputs" can be used here instead? And these Outputs will be accessible via the hooks. Outputs already accessible via the describe-stack-outputs command..
3. Maybe start simple by just providing custom_hooks with access to stack outputs?

[ChristophShyper]

Any progress on that?
Maybe just give us access to parameters or sceptre_user_data inside hooks?

[ngfgrant]

@m1keil Option 3 sounds like a very good start and we could move quickly on it?

[m1keil]

@ngfgrant I can give it a go. Been some time since I looked at this issue. I'll update in few days.
BTW, we are talking about v2?

[ChristophShyper]

Thanks guys.

[ngfgrant]

Yeh V2 only please

[m1keil]

@Krzysztof-Szyper-Epam btw I believe you already have access to parameters or sceptre_user_data inside a hook (in v2).

It's all accessible via self.stack.paramers or self.stack.sceptre_user_data.
@ngfgrant I think that the docs are currently wrong?:

Hooks may have access to `argument`, `stack_config`, `stack_group_config` and
`connection_manager` as object attributes. For example `self.stack_config`.

Should be probably something along the lines of:

Hooks have access to Stack Config via `self.stack`. For example, parameters dict is accessible via `self.stack.parameters`.

[ChristophShyper]

@m1keil That's great. I'll test it when I have the time.
Honestly, I'm still using 1.3.4 because I inherited it that way and just adjusting infra stacks. Haven't had time to update it yet.

So something like that should work?

sceptre_user_data:
  db_endpoint: !stack_output_external my-other-sceptre-project-mysql::DBEndpointAddress
hooks:
  before_update: !cmd "echo {{  self.stack.sceptre_user_data.db_endpoint }}"

[connec]

In Sceptre 1.x we used a trick whereby we would add a custom field to the root of the configuration (e.g. hook_data: ...) which would contain resolvers. Because the field was 'unknown' to Sceptre it wouldn't try to actually do the resolving, but you could do it yourself in the hook by accessing the stack config field and assigning it to a resolvable property, e.g.:

# stack.yaml
...
hook_data:
  upload:
    s3_bucket_name: !stack_output_external platform-{{ environment_path.0 }}-infrastructure::LambdaCodeS3Bucket

...
from sceptre.resolvers import ResolvableProperty
...

class DeployCode(Hook):
    resolved_hook_config = ResolvableProperty('resolved_hook_config')

    def run(self):
        self.resolved_hook_config = self.stack_config['hook_data']['deploy_code']

I've not tried to migrate anything like that to Sceptre 2.x, but the general approach could be something that could be worked into an implementation in the library (also iirc only stack_output_external would work here, but I can't remember if that's true or why 😄).

[myoung34]

I'm currently using v2 (newbie here) but i dont see any way to get outputs during a hook.

self.stack.sceptre_user_data is empty dict {}

my yaml:

template_path: openvpn.py
parameters:
  VpcId: !stack_output_external vpc-v2::VpcId
  Subnet: !stack_output_external v2-dev::privateSubnetZone1
  KeyName: test
  InstanceType: t2.micro
hooks:
  after_create:
    - !kms_grant ....

[outwarped]

At the moment hook node type is ScalarNode only.

loader.construct_scalar(node)

I have experimented with allowing other node types (sequences, maps) and fully utilise constructor arguments:

def __init__(self, *args, **kwargs):

Obviously this change allows all sorts of types of objects appearing in the arguments (scalar only produces single string) including objects constructed by other resolvers, which can be a major concern, since this exposes a lot of data.
The good news is that the change should be backward compatible as the scalar nodes model does not change.

Example:
hooks: before_update: !cmd - echo - !stack_output s3.yaml::S3BucketName

[ngfgrant]

@myoung34

I'm currently using v2 (newbie here) but i dont see any way to get outputs during a hook.

self.stack.sceptre_user_data is empty dict {}

my yaml:

template_path: openvpn.py
parameters:
  VpcId: !stack_output_external vpc-v2::VpcId
  Subnet: !stack_output_external v2-dev::privateSubnetZone1
  KeyName: test
  InstanceType: t2.micro
hooks:
  after_create:
    - !kms_grant ....

You've not specified a sceptre_user_data key

try add

...
sceptre_user_data:
  - welcome: hello world

For everyone else

Regarding hooks - you can access everything on a stack including its dependencies with

# my_hook.py
...

self.stack.sceptre_user_data
self.stack.parameters
self.stack.dependenices # gives full stack objects with complete stack config of dependencies

I can definitely see that having this functionality would be useful but I am not sure it is a wise thing to add in terms of maintainability and potential for debugging chaos. We do plan to rewrite how resolvers work so that they are a bit nicer to maintain and at that point we might look at doing this.

Not settled on anything just yet but those are some thoughts.

[hreeder]

An approach to this was resolved in #140 but only ever made it into v1.5.0. I'll be looking to submit a PR to resolve this regression in master soon.

[alexharv074]

I am trying to pass the outputs of Sceptre stack_output_external resolvers to hooks and getting surprising failures, e.g.:

hooks:
  before_create:
  - !cmd "./network/hooks/validate_cidrs.sh \
      -p {{ var.PrimaryCIDRBlock }} \
      -a {{ var.PublicSubnetCIDRa }},{{ var.PublicSubnetCIDRb }},{{ var.PublicSubnetCIDRc }} \
      -b {{ var.PrivateSubnetCIDRa }},{{ var.PrivateSubnetCIDRb }},{{ var.PrivateSubnetCIDRc }}"

And in my vars:

PublicSubnetCIDRa: '!stack_output_external gitops-test-vpc::PublicSubnetCIDRa'
PublicSubnetCIDRb: '!stack_output_external gitops-test-vpc::PublicSubnetCIDRb'
PublicSubnetCIDRc: '!stack_output_external gitops-test-vpc::PublicSubnetCIDRc'
PrivateSubnetCIDRa: '!stack_output_external gitops-test-vpc::PrivateSubnetCIDRa'
PrivateSubnetCIDRb: '!stack_output_external gitops-test-vpc::PrivateSubnetCIDRb'
PrivateSubnetCIDRc: '!stack_output_external gitops-test-vpc::PrivateSubnetCIDRc'

This is resulting in a failure:

subprocess.CalledProcessError: Command './network/hooks/validate_cidrs.sh -p 10.90.0.0/19 -a !stack_output_external gitops-test-vpc::PublicSubnetCIDRa,!stack_output_external gitops-test-vpc::PublicSubnetCIDRb,!stack_output_external gitops-test-vpc::PublicSubnetCIDRc -b !stack_output_external gitops-test-vpc::PrivateSubnetCIDRa,!stack_output_external gitops-test-vpc::PrivateSubnetCIDRb,!stack_output_external gitops-test-vpc::PrivateSubnetCIDRc' returned non-zero exit status 1.

I guess I am running into the same issue as this one?

[sql-sith]

Me too, plus one, and all that. 👍

I bet there is a way to access stack outputs from inside a hook fired after that stack is created, but if there is, I can't figure it out from the docs. If someone knows how to do this, can you get me started?

[labmanjoe]

There is a crude way to do this.

Basically, your hook calls a wrapper script instead of the actual command you want to call.

The wrapper script calls sceptre --output yaml list outputs <stack group> and parses the output to get the needed stack output, and then calls the actual command you want to call with this value.

I think this will work on all versions of sceptre, though I have only tested it on the version I'm using.

Note depending on how you split up your stacks/stack groups, you may need to have reflective property names defined in your stack config to pass to the wrapper script. e.g. a property stack_group_hack: production so that you can pass that to the wrapper, which then uses it when calling sceptre list outputs (but likely appending specific substacks to the value).
(Though, there may be some secret value that already knows this, but I don't know it.)