Aws-cdk: Toolkit: programmatic usage

[hassankhan]

Hi there, thanks for creating this package! I was just wondering if there's any way of triggering CDK CLI commands programatically? This would really help with respect to integrating with existing tools and workflows.

[rix0rrr]

Hi @hassankhan,

The aws-cdk NodeJS package does expose some library functions that can be used from JavaScript/TypeScript projects directly. However, it's not a polished experience at the moment, because our attention has been focused more towards the construct library so far. You're very welcome to try it and tell us if you run into any issues, though. As a last resort, you can always shell out to the command-line tool.

I'm also curious: could you tell us something about the kinds of integration that you're thinking of?

[hassankhan]

Hi @rix0rrr,

Thanks for the info! In an ideal world, I'd love to see something like:

import CDK from '@aws-cdk/cdk';

import MyApp from './MyApp';

const cdk = new CDK({ app: '', versionReporting: false });

cdk.deploy(new MyApp())
  .then(() => {
    return remove(new App());
  })

With regards to integration, essentially, our aim was to define resources such as Lambdas and DynamoDB tables once in our application code, and generate valid snippets from them and insert them into a template for deployment. For example, a theoretical DynamoDB model would look something like this (using dynamodb-data-mapper):

@table('pets')
class Pet {
    @hashKey()
    id: string;

    @rangeKey({defaultProvider: () => new Date()})
    createdAt: Date;

    @attribute()
    isCat?: boolean;
}

The table schema would then be read from this class as part of the deployment process, converted into its constituent Construct and added to the App/Stack. We could use the same approach for other resource types such as SNS Topics and Lambda functions.

[rix0rrr]

Okay, thanks. I definitely see the value of the first one, especially in defining one-off tasks like EMR jobs or integ tests.

For the second one: you can already do this today, even without tool integration.

If you define you data model in a library, you can use this library both in your runtime code as well as introspect it in your infrastructure code. Your build step would produce both a CloudFormation template as well as a JAR, and during deployment you deploy both together.

[rui-ktei]

+1
I mentioned this in a duplicate ticket as well.

[BDQ]

I'd like this too, in my case I'd like to wrap a deploy of an ECS cluster so I can scale it manually after the deploy is done (I create it with desiredCount: 0 so the CloudFormation creating is quicker).

[dsilvasc]

Additional use cases from https://github.com/awslabs/aws-cdk/issues/1133 - synthesizing stacks from an IDE debugger and embedding into JVM build systems (like Gradle) without having to shell out to a CLI that depends on node.js.

[mmoulton]

My team is currently programmatically invoking CDK to support dynamic composition of constructs to create stacks based on the answers to question we ask in a custom CLI tool we wrote. We are currently primarily using it to spin up either SAM apps, or static sites using codepipeline, codebuild, rds, vpc, ssm params, s3 buckets, cloudfront, certs, zones, etc.

I'd very much like an officially supported way of accomplishing this. Currently we are doing some very dirty things to mimic the nature of the CDK CLI and the fact it executes the cdk app multiple times to meet dependencies.

If the CDK team does not have plans to officially support this, I'd very much appreciate not making it harder to achieve this as the notes in https://github.com/awslabs/aws-cdk/issues/2044 and https://github.com/awslabs/aws-cdk/issues/2016 suggest.

[KarthickEmis]

Is there anyway of triggering AWS CDK ClI (CDK synth , CDK deploy) commands pro-grammatically in typescript ?

[auser]

I found a way to handle this. I'll create an example repo (hopefully) to demonstrate how I accomplished this process, but the short version looks like this:

Create an AppStack instance and set the synthesizer to the output of app.run():

import { AppStacks } from 'aws-cdk/lib/api/cxapp/stacks'
import { Configuration } from 'aws-cdk/lib/settings';
// ...
const out = this._app.run();        
const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

const appStacks = new AppStacks({
    aws: this.sdk,
    configuration: configuration,
    synthesizer: async () => out
})

Using the AppStack instance, create an instance of the CdkToolkit:

const appStacks = await this.getAppStacks();

const provisioner = new CloudFormationDeploymentTarget({
    aws: this.sdk,
})
const cli = new CdkToolkit({ appStacks, provisioner });

In order to get the credentials, we'll need to muck around at a low-level with the AWS-SDK:

import { debug } from '@aws-rocket/core';
import AWS = require('aws-sdk')
import { SDK } from 'aws-cdk/lib/api/util/sdk';
import { Mode } from 'aws-cdk/lib/api/aws-auth/credentials'

async function getCredentialsConfig(sdk: SDK): Promise<any> {
    const region = await sdk.defaultRegion()
    const defaultAccount = await sdk.defaultAccount()
    const credentials = await (sdk as any).credentialsCache.get(defaultAccount, Mode.ForReading)

    return {
      region,
      credentials
    }
  }

export const awsCredentialsMiddleware = async (argv: any) => {
    debug(`Setting profile to: ${argv.profile}`)
    if (argv.profile) {
        const sdk = new SDK({
            profile: argv.profile
        })
        const credentials = await getCredentialsConfig(sdk)
        AWS.config.update(credentials)
        AWS.config.setPromisesDependency(Promise);
        argv.AWS = AWS
        argv.SDK = sdk
    }
    return argv
}

Finally, this all comes together using the actual deploy like so:

const appStacks = await this.getAppStacks();
const allStacks = await appStacks.listStacks()
const allStackNames = allStacks
    .map((s: cxapi.CloudFormationStackArtifact) => s.name)

const cli = await this.getCdkToolkit()
try {
    const res = await cli.deploy({
        stackNames: allStackNames,
        // `requireApproval` is so that the user never needs to 
        // accept anything for the deploy to actually occur
        requireApproval: RequireApproval.Never,
    })
    console.log('deploy result ->', res);
} catch (e) {
    console.error(`Error deploying`, e);
    process.exit(-1)
}

Hope this helps and hopfully this becomes a part of the actual library. I'll try to submit a PR if I can, but no promises. If anyone else has any better way of handling this, I'd love suggestions.

[jd-carroll]

This is all very interesting and hopeful that it will get implemented 🤞

@auser - Were you ever able to put together a repo / gist with all of this information? I follow the logic but I'm not sure how I follow how to assemble all of the pieces. Is each code snippet supposed to be in the same file and if so does each code section follow one an other? I can guess, but I also don't know where the this._app and this._argv come from.

@fulghum / @shivlaks - It looks like there is some movement towards an implementation. Do feel programmatic access might be limited in some areas or is the goal to have a full API access the same you would have on command line?

More or less, do you think something like the following would be supported:

import AWS from 'aws-sdk';
import { deploy } from 'aws-cdk';
import MyStack from './my-stack';

const cfnOutput = await deploy(new MyStack());
console.log(`Successfully deployed stack:\n${cfnOutput}`);

const s3FileAndDetails = { /* ... */ };
const s3 = new AWS.S3();
await s3.putObject(s3FileAndDetails).promise();

Also, instead of needing to run the configuration like @auser has above:

const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

It would be great if we could skip the command line altogether and just pass in a json object with relevant configuration. (Thereby eliminating the need for a cdk.json, but one could exist with overrides)

// Use factory function to allow reading config overrides in async
const configuration = await Configuration.create({ /* json from cdk.json */ })

[ryan-mars]

If you want to synth programmatically to the directory of your choice...

// Let's say you want it synthed to ./build
const outdir = path.join(process.cwd(), 'build') 
// App's constructor let's you specify the outdir 
const app = new cdk.App({outdir})
new ApiLambdaCrudDynamoDBStack(app, 'ApiLambdaCrudDynamoDBExample')
app.synth()

If you want to inspect what was produced app.synth() returns a CloudAssembly which can be interrogated with getStack('your stack name')

I haven't tried deploying programmatically yet.

[jd-carroll]

When implementing this feature, would it make sense to do it in the context of aws-sdk-js-v3?
https://github.com/aws/aws-sdk-js-v3

[ralovely]

(One of) the premise of CDK being "programmatic CloudFormation", it opens the way for fully programmable infrastructure, potentially even starting with the account creation
(and that's awesome, we (building an "Ops as a Service" product) have been hoping for this for quite some time, even looking into developing our own solution).
Creating a full, real-life infrastructure requires more features than CloudFormation can offers, even considering CustomResources.
Some of those will require the SDK ; some others need to talk to 3rd party services ; fetch data before ; output/notify after.

Being able to use CDK not so much as a toolkit/cli, not as a complete solution, but as a lib, as a way to integrate the infrastructure-control layer into a programm would be great. As great as CDK promise to be, it's not a complete solution, probably can't, and shouldn't even try to be one.
Of course, one can always shell out, wrap around CDK CLI, but that's not ideal.

It ties in with a few other GitHub issues:

• Support for app.postDeploy() for post-deploy functions
• CLI: way to get outputs from deployed stack
• Stacks and/or Contructs should emit events
• Document how to use stack parameters (to some degree)

[jfaixo]

Another use case : when your CDK stack creates Cognito UserPool Client Apps, it generates client id and secrets that need to be injected in your clients. If your client is a webapp in S3, that means you have to do steps in this order:

• create userpool and clients, get id values
• build webapp with updated id values
• deploy webapp to S3

If it was possible to deploy programmatically, it could be possible to have the whole process written in the same tool (for example nodejs). Otherwise (as of now I do this way), you have to rely to some other external tool to call the CLI multiple times and schedule these actions (like a bash script, or other nodejs script through process exec..).

[abierbaum]

I just found a nice little helper that wraps some of this here: https://github.com/dmitrii-t/cdk-util May be worth a look for people that want to do this type of thing. Definitely can't wait to see it in the core API with first class support.

[ryan-mars]

@jfaixo

If it was possible to deploy programmatically

It is.

import * as s3 from "@aws-cdk/aws-s3";
import * as s3deploy from "@aws-cdk/aws-s3-deployment"; // <<--- this right here
import * as cdk from "@aws-cdk/core";

export class WebStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const websiteBucket = new s3.Bucket(this, "WebsiteBucket", {
      websiteIndexDocument: "index.html",
      publicReadAccess: true,
      versioned: true
    });

    new s3deploy.BucketDeployment(this, "DeployWebsite", {
      sources: [s3deploy.Source.asset("./build")], // <<--- folder to upload
      destinationBucket: websiteBucket
    });
  }
}

[abierbaum]

Unfortunately I have found that the code and helpers I can find online are all out of date with the latest version of the CDK. How are people using the CDK in the case where you have a command line tool that needs to dynamically allocate and manage resources on AWS?

[sukhwant]

I am looking to use it from JAVA.

[zyrorl]

I've been able to sort of get this working programatically but have discovered that the cli.deploy etc functions return void, undefined, boolean and 0/1 values instead of useful values like cloudformation stack arns, outputs etc.

[shivlaks]

from @benwainwright

Currently, I can get a list of stacks that will be generated by my project in the order they should be deployed (this is important; see below) by running the command cdk list
It would be useful if the output from this command could be exposed as an api in a way that doesn't require full synthesis of the templates under the hood.

full use case in #8436

[Guitarkalle]

This would be wonderful. Now it feels like CI integration is a pain.. You have to parse output for changeset names and stack ids after deploying to do any more serious CI flow. Using the synthesizing & deploy parts of the tool programmatically and also having the CLI able to output in a JSON format instead of just console logs with necessary information would be perfect

[moshir]

Hi,
any update on this one ?
I'm trying to automate the creation of AWS resources from a lambda function, and would be great to have a programmatic API in addition to the cdk CLI.
Alternatively, is there any way to simply get the cloudformation template string generated by synth(), and then pass it to boto3 cloudformation API ?

[rossng]

This would be very useful for my use case: dynamically deploying MediaLive broadcast pipelines. I don't really see shelling out to the CLI as a viable option.