Serverless-application-model: Cannot create StepFunctions with CloudFormation from external definition file

[gitorrin]

Hi,

We are trying to deploy stepfunctions with CloudFormation, and we'd like to reference the actual stepfunction definition from an external file in S3.

Here's how the template looks like:

StepFunction1: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: !Ref StepFunction1SampleName
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn
      DefinitionString:  
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'

However, this doesn't seem to be supported, as we are getting error
Property validation failure: [Value of property {/DefinitionString} does not match type {String}]

We are doing something similar for APIs, referencing the actual API definition from an external swagger file, and that seems to be working fine.

Example:

SearchAPI:
    Type: "AWS::Serverless::Api"
    Properties:
      Name: myAPI
      StageName: latest
      DefinitionBody: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:            
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/ApiGateway/myAPI/swagger.yaml'

[jlhood]

Hi thanks for reaching out!

The reason the include transform works for AWS::Serverless::Api is because the yaml/JSON in the DefinitionBody parameter gets passed through to the Body property of the AWS::ApiGateway::RestApi resource type, which supports a yaml or JSON object embedded in the template itself.

The DefinitionString property of the AWS::StepFunctions::StateMachine resource type expects a string containing a JSON definition of the statemachine. However, the include transform attempts to embed the statemachine definition as an embedded yaml/JSON object rather than a string containing the JSON statemachine definition.

I don't know of a way to use the include transform to accomplish this. Here are some options I can think of:

1. Embed the statemachine as a string in the template.
2. Use a template parameter to hold the statemachine definition and pass the statemachine in as the parameter value when creating/updating the stack.
3. Use a templating system to construct the template at build-time so you can store/develop the statemachine in a separate file, but still embed it into the template before passing it to CloudFormation.

[gitorrin]

Thank you, @jlhood, for taking the time to reply.

1. The template can become quite large if there's more than a few stepfunctions defined.
2. It could work, but we are working in an automated integration & delivery manner.
3. It sounds similar to what we were trying to achieve, with Transform/Include and it proofs to have some limitations.

Since I have Enterprise Support plan as well, I asked for support through that channel also.
We got advice through that channel, that it is possible to use Transform/Include with StepFunction definitions, in CloudFormation.

The trick is to escape the StepFunction DefinitionString property, and include the actual property, DefinitionString, in the external CloudFormation referenced file. Escaping only the stepfunction definition string would fail, CloudFormation complaining that the referenced Transform/Include template, is not a valid yaml/json.
Here's how it looks like:
Template:

StepFunction1: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: !Ref StepFunction1SampleName
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn      
      Fn::Transform:
        Name: AWS::Include
        Parameters:
          Location: 
            Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'

External stepfunction definition file:

{
    "DefinitionString" : {"Fn::Sub" : "{\r\n  \"Comment\": \"A Retry example of the Amazon States Language using an AWS Lambda Function\",\r\n  \"StartAt\": \"HelloWorld\",\r\n  \"States\": {\r\n    \"HelloWorld\": {\r\n      \"Type\": \"Task\",\r\n      \"Resource\": \"arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}\",      \r\n      \"End\": true\r\n    }\r\n  }\r\n}"}
}

Now, although this solves the problem, it's a bit more difficult to maintain the StepFunction definition, in this form, in source control.

So I've thought about using a CloudFormation custom resource backed by a lambda function. The lambda function would deal with the actual StepFunction DefinitionString escaping part.

Here's how it looks like:
Template:

StepFunctionParser:
    Type: Custom::AMIInfo
    Properties:
      ServiceToken: myLambdaArn
      DefinitionString: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunctionX/definition.json'   
  StepFunctionX: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: StepFunction1SampleNameX
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn      
      DefinitionString: !GetAtt StepFunctionParser.DefinitionString

External StepFunction definition file:

{
  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Task",
      "Resource": {"Fn::Sub" : "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}" },
      "End": true
    }
  }
}

Here's a https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-custom-resources-lambda-lookup-amiids.html for creating AWS Lambda-backed Custom Resources.

There's still a problem with this.
Transform/Include converts external template boolean properties into string properties.
Therefore, DefinitionString

"DefinitionString": {
            "States": {
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
                    **"End": true**
                }
            },
            "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
            "StartAt": "HelloWorld"
        }

becomes

"DefinitionString": {
            "States": {
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": _realLambdaFunctionArn_,
                    **"End": "true"**
                }
            },
            "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
            "StartAt": "HelloWorld"
        }

CloudFormation then complains about the StepFunction definition not being valid:

Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: Expected value of type Boolean at /States/HelloWorld/End' 

Is this a CloudFormation Transform/Include issue? Can someone from AWS give a statement on this?

@jlhood , could you please reopen the case, or move it where you think it fits best?
Thank you.

[jlhood]

I'm glad you contacted support about this. Their suggestion to include DefinitionString in your included template is a good option I hadn't thought of. Building on that idea, if you use yaml instead of JSON, you might be able to get the best of both worlds (being able to store your definition in a separate file, but also having it be readable) using yaml's support for multiline strings via the > or | character and the !Sub intrinsic function. You could try something like this:

DefinitionString: !Sub |
  {
    "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
    "StartAt": "HelloWorld",
    "States": {
      "HelloWorld": {
        "Type": "Task",
        "Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
        "End": true
      }
    }
  }

Note: I closed this issue because it's not a SAM-specific issue, but rather a general CloudFormation one. I think StackOverflow is a more appropriate place to ask this question (tag your question with amazon-cloudformation). If my above suggestion works for you, please still post the question to StackOverflow and then answer it with the solution that worked for you so others can benefit from it as well.

[gitorrin]

@jlhood, I implemented your recommendation and that it's a good improvement.
Thank you, again, for taking the time to think & reply about this.
The AWS Support team is investigating the possible Transform issue.

[gitorrin]

One thing about the yaml example. I think short syntax for Fn::Sub - !Sub, is not supported inside Transform templates.
The following snippet works:

DefinitionString: 
  Fn::Sub: |
    {
      "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
      "StartAt": "HelloWorld",
      "States": {
        "HelloWorld": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
          "End": true
        }
      }
    }

References:
"We do not currently support using shorthand notations for YAML snippets." Documentation

[jlhood]

I'm so glad we were able to find a solution for you! I still encourage you to post your question to StackOverflow and then answer it with this answer so others can benefit.