Aws-cdk: Exporting/importing across CDK apps

[rix0rrr]

exporting/importing works wonders inside a CDK app, but we don't have a good solution across CDK apps.

Case in point: VPCs, which are an ID and the IDs of a variable number of subobjects.

This will come up if multiple teams are running their CDK apps inside the same VPC; the 3 components here would probably be in their own code repositories and sharing between those is not necessarily something we want to force people to do.

Better would be to do do export/import along some identifier, probably, or use a context provider (but would the latter work in a CI/CD context?)

cc @eladb, I want to tackle this soonish.

[rix0rrr]

I think the solution should be to define some sort of "parameter pack" that has an identifier. In effect, the mechanism would be something like this:

// App 1
const props = vpc.export();
stack.publishParameterPack('MyVpc', props);

// Will publish something like:
// MyVpc_VPCID = vpc-12354
// MyVpc_AZs = us-east-1a,us-east-1b, us-east-1c
// MyVpc_SubnetIds = s-123,s-456,s-12435
// ... etc

// App2
const props = stack.readParameterPack('MyVpc');
const vpc = VpcNetwork.import(this, 'MyVpc', props);

Obviously the function names and mechanisms here are obtuse and hard to discover, so we should make that more streamlined.

[rix0rrr]

Better syntax (and I think actually what people would expect) would be this:

// App 1
vpc.export('MyVpc');

// App2
const vpc = VpcNetwork.import(this, 'MyVpc');  // Not great because it ties export ID to construct ID

const vpc = VpcNetwork.import(this, 'MyVpc', 'MyVpc');  // Not great because what's with the stutter?

const vpc = VpcNetwork.import('MyVpc'); // Best. Possible if we don't make the imported thing a construct
// But not always possible to not make the imported thing a construct, sometimes it needs to construct other 
// resources.

const vpc = VpcNetwork.import(this, 'MyVpc', {
  exportName: 'MyVpc'      // A fair compromise?
});  

[rix0rrr]

We can also make ImportedVpc (or whatever) a non-invisible class, and if people want to construct one using straight-up hardcoded VPC ids and whatever, they can instantiate it directly.

[eladb]

Something to consider: the use case of importing a construct through a set of concrete values must be idiomatic, and not 2nd class (Bucket.import("bucketName")). We could have two different "imports" but we should support both and consider both in this design.

Also - can you share some thoughts on implementation? Do you plan to use SSM/stack-import-export, etc? Will this work across regions/accounts?

[eladb]

I like the export(“boom

[eladb]

Accidental

[rix0rrr]

In-app references

So in my mind, the ideal situation is that inside the same app, you don't even have to do export()/import(). The system has enough information to do this for you, so why wouldn't it?

Of course, as I've been trying to implement it in my spare time, I'm running into the fact that this is HARD to implement at the construct level, specifically because of lazy evaluation:

new cdk.Token(() => otherStack.queue.queueArn);

The use of otherStack.queue.queueArn should lead to an Export being added in otherStack, but we'll only know that at rendering time, at which time it's too late to mutate otherStack. There are two ways to deal with this:

• Introduce a definite moment in time during which lazy tokens have their values resolved.
• Make Lazy tokens not take an arbitrary function, but make them more of a placeholder so that we can do something in reaction to the value becoming available. I.e:

lazyToken.setValue(otherStack.queue.queueArn);

• Work at the CloudFormation template level (and lose the benefits from most of the class model and the fact that we have object references).

I'd prefer the second one, honestly.

Implementation

Even for cross-region access, I think the correct implementation is to put "Outputs" on a stack. The only difference is how we consume them:

• Same account ∧ same region: use Fn::ImportValue.
• Different region: use custom resource
• Different account: use custom resource + assume Role

For now I was only planning to do the first case. The other ones can be left as an extension for later.

Direct and packed imports

My proposal was to do the following:

Bucket.import(this, 'Bucket', {
   exportName: 'ABC'
});

new ImportedBucket(this, 'Bucket', {
   bucketName: 'XYZ'
});

But something tells me you'd prefer this (:wink:):

Bucket.import(this, 'Bucket', {
   exportName: 'ABC'
});

Bucket.import(this, 'Bucket', {
   bucketName: 'XYZ'
});

With a runtime check on the presence of arguments.

[eladb]

In-app references

I agree that in-app references should be implicit, and I think it would be enormously valuable to properly model __cross references__ at the construct-level. As you hinted above, I believe we made a mistake and conflated the concepts of _lazy evaluation_ and _cross references_ (via magical "tokens").

The first thing we would need is to encode the source of a cross reference into the token. I think that's something that the cdk.Construct class needs to support (something like this.lazyString(value | function) => string).

As for resolution, it's not only an export that needs to be added, it's also that the value in the consumer side would need to be Fn::ImportValue instead of e.g. Fn::GetAtt, so the token would resolve to different things depending on the relationship.

I wonder if the right way is to add a post-synthesis phase which allows users to reflect on the synthesized output (alongside metadata generated about cross-references for example) and mutate it.

Implementation

Sounds reasonable

Direct and packed imports

I was thinking of something like:

Bucket.importFromAnotherStack(exportName);
// or
Bucket.importFromConcreteValues({ bucketName: ...});

(names pending)

[rix0rrr]

The first thing we would need is to encode the source of a cross reference into the token.

Don't know whether I'd want Construct to have methods on Construct specifically for this. I've been introducing a new anchor parameter to CloudFormationTokens (the only type of Tokens that could possibly care about care about their stack origin). And it doesn't need to be passed for all tokens, but it does need to be passed for:

• {Ref}
• {Fn::GetAtt}

These are easy, since all places where these are instantiated are generated by cfn2ts. We can have that pass an additional argument this in every instantiating call.

Less nice are these:

• {AWS::StackId}, {AWS::Region}, {AWS::Partition}, {AWS::AccountId}, and a couple of others.

They also return an intrinsic that depends on the source stack, and especially {AWS::Region,Partition,AccountId}, when used in constructing an ARN and passing the ARN around need to be scoped to the source stack. The ARN might be passed cross-region or cross-account and it's important that the {AWS::AccountId} keeps on referring to the source stack account ID, not the account ID of the consuming stack. These will muck up the call sites a bit, but I see no other way than to pass another anchor construct so the tokens are attached to stacks. We can keep the ugliness confined to the insides of the L2 layer.

Tokens that don't need an anchor:

• {Fn::Split}, {Fn::Select}, {Fn::Base64}, ...

the token would resolve to different things depending on the relationship.

Yes. The same Token object may be consumed in two different places, and may need to yield a different output for both resolve() calls.

I've been solving this by adding context to the resolve() call, passing the stack in which the current resolution is taking place.

Substituting is also something that a StackElement object could ultimately do, because that's where Tokens are ultimately materialized and so a StackElement could crawl its properties for x-references and treat them differently.

I wonder if the right way is to add a post-synthesis phase which allows users to reflect on the synthesized output

I fear any work done at the template level is going to be incomplete, because IDs across stacks can conflict and at that point we've lost the information that would allow us to tell objects apart (the object identity tells us which actual resource we were referring to, but we can't use that anymore at the template level).

Two templates with a cross-stack reference, in which we output the same {Ref} value for the Token regardless of whether it's a same-stack or cross-stack reference:

# Stack1
Resources:
    ResA: 
        Type: AWS::Something

# Stack2
Resources:
    ResA: 
        Type: AWS::Something
    ResB:
        Type: AWS::OtherThing
        Properties:
            # Which one of these refers to the same-stack ResA
            # and which refers to the other-stack ResA?
            Calls: { Ref: ResA }
            Uses: { Ref: ResA }

Metadata:
    "Stack2/ResB" has a cross-stack reference to "Stack1/ResA"

We can start working around this with more levels of indirection, but it starts to become convoluted.

[rix0rrr]

Here's what we do for the cross-stack references:

We use the pre-synthesis moment to crawl all stackelements and call resolveCrossStackReferences(), which is going to resolve their properties, discover all cross-stack references, and replace them with references to Outputs on the other stacks.

[rix0rrr]

Import

Bucket.importFromAnotherStack(exportName);

Some imported resources still need to instantiate new constructs, so they must be constructs themselves. For example:

const loadBalancer = ApplicationLoadBalancer.import(...);
loadBalancer.addListener(...);  // Works, creates a new `ApplicationListener` object.

To make this work, ImportedApplicationLoadBalancer needs to be a Construct, and so needs a (parent, id) pair as argument. So importing would look like:

ApplicationLoadBalancer.import(this, 'LoadBalancer', 'ExportName');

First of all:

• Don't like the two strings next to each other.
• It breaks the pattern that we have in most cases where we go (this, string, props).

So I'd prefer for this to be:

ApplicationLoadBalancer.import(this, 'LoadBalancer', {
  export: 'ExportName'
});

So how about:

Resource.import(parent, id, props);
Resource.importDirect(parent, id, props);

Export

I've considered doing export like this:

class Bucket {
  public export(exportName?: string): BucketRefProps;
}

In a bid to reduce API friction in migration. Existing code will continue to work (using the return value), but people can supply a name for cross-app reuse.

However, I think it's a step in the wrong direction to give people two ways to do the same thing, especially when one of the ways will work in all situations and the other will only work in one of the two situations.

There's no downside to exporting/importing by name in a same-app situation, but it's not possible to export/import by props object in a cross-app situation.

So why not force everyone to refactor to exporting/importing by name?

So the API will be:

class Bucket {
  public export(exportName: string): void;
}

[rix0rrr]

After some discussion, decided to park the larger design question in favor of a context provider for VPCs.

[rix0rrr]

Mental note: we should change the API of VpcNetwork to make it easier to Fn::ImportValue subnets.

So that means changing:

vpc.subnets(SubnetType.Public).map(s => subnetId)
// TO
vpc.subnetIds(SubnetType.Public)

So that we can Fn::ImportValue a list of strings.

[eladb]

I've spent a lot of time on designing and started the implementation of a solution, and eventually realized a solution that's only based on CloudFormation imports/exports is too fringy right now.

Let's list the use cases for referencing external resources in the CDK:

1. Reference a resource that resides in another stack in the same app (and environment)
   
   
   
   • Covered by @rix0rrr implicit reference masterpiece.
   
   
2. Reference a resource that already exists in your environment by a simple attribute Name/ARN
   
   
   
   • Going to be covered by "from" static methods Bucket.fromArn, etc (see #2273)
   
   
3. Reference composite resources which already exists in your environment (e.g. a VPC)
   
   
   
   • We plan to implement a .fromAttributes method for composite resources (similar to .fromArn but takes multiple attributes) (see #2273)
   
   
   • For VPCs, we also have a "lookup" method which allows you to specify the VPC ID and it will query the environment and import this VPC details.
   
   
   • We can implement "lookup" methods as needed for other common use cases
   
   
4. Export simple resources across stack in the same environment
   
   
   
   • Users can simply use CfnOutput and Fn.importValue with one of the from methods.
   
   

There are other use cases that are NOT covered by this change such as:

1. Reference resources across accounts/regions within the same app
   
   
   
   • Should be covered for specific use cases (e.g. CodePipeline) by @skinny85 physical names work
   
   
2. Reference resources across accounts/regions across apps
   
   
   
   • There are some ideas on how to enable this generally, but perhaps physical names would be sufficient.
   
   

So, you ask, why do we actually need this serialization/deserialization thing? Well, the initial use case we can planned to solve with serialization/deserialization is:

1. Reference composite resources exported by __another__ CDK app, within the same environment (account/region)

What does that mean? Let's say CDK App A defines a VPC and wants to "publish" it so other apps that run __in the same environment (account/region)__, this mechanism will let users "export" the entire VPC construct (including all it's details) under some CFN export name and then any app that runs within the same account/region will be able to import it and use it.

We don't have strong signal from customers that this is something they urgently need. In most cases, the VPC use case can be addressed using the lookup approach, and in the rare case where someone would need to implement something like import complex constructs across apps within the same account, they can always use fromAttributes and bind them to Fn.importValue. It's just going to be a bunch of glue logic, but nobody will be blocked.

The other aspect of this work is the general serialization/deserialization capability it brought with it, and that might have future value in general. For example, we could use this to marshal resources from construction time to the runtime space, save/load from SSM parameter store, etc.

Given this situation, we'll punt this work for now. I believe that at some point we will implement a serialization pattern/scheme for constructs, but we should have a more compelling use case before we spend this energy across the entire framework.

[rix0rrr]

Reference _composite_ resources exported by another CDK app, within the same environment (account/region)

We don't have strong signal from customers that this is something they urgently need.

We do have this request actually from ECS, where a very common user story is have team A manage the VPC+Cluster, and teams B-E run individual services on that single cluster.

You would want to give each team an individual repository with an individual CDK app, but they all need to import the complex objects VPC+Cluster, managed by team A in a different app.

In a CI/CD environment with multiple production stages, a single static ARN is not enough because the ARN will be different for every environment.

Yes, team A could vend a { region -> ARNs } lookup table to run the Vpc.fromAttributes() off of, but it would be infinitely nicer if the CDK could just automatically do this.

A custom lookup might be able to solve this problem, but it's not clear how to reference the intended resources at all.

Cluster.lookup(); // Which one, what if there are multiple?

Cluster.lookupByName('x');  // What if the name is automatically generated? I now need to statically define it which brings other problems with CFN. We need an indirection.

Cluster.lookupByNameFoundInSsmps('/SharedClusterName'); // Guess that works but it's a lot of ad-hoc machinery

Also, lookups require updates to both toolkit and construct library, which breaks compatibility.

[peterjuras]

What is the outcome here or the suggested way to tackle these?

I'd like to define a database in a separate CDK app and reference it in other apps. What would be the best way to achieve this?

[kevin-lindsay-1]

@peterjuras I have a workaround for this, the crux is that we use Stack.getAtt('Outputs.${key}')

Child:

utils.ts

export class OverriddenCfnOutput extends CfnOutput {
  public constructor(scope: Construct, id: string, props: CfnOutputProps) {
    super(scope, id, props);
    this.overrideLogicalId(id);
  }
}

stacks/childService.ts

export class ChildStack extends Stack {
  public constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const vpcId = new CfnParameter(this, 'vpcId', {
      default: 'N/A',
    });
    const vpcZones = new CfnParameter(this, 'vpcZones', {
      type: 'List<AWS::EC2::AvailabilityZone::Name>',
      default: 'N/A',
    });
    const vpcPublicSubnetIds = new CfnParameter(this, 'vpcPublicSubnetIds', {
      type: 'List<AWS::EC2::AvailabilityZone::Name>',
      default: 'N/A',
    });
    const vpcPrivateSubnetIds = new CfnParameter(this, 'vpcPrivateSubnetIds', {
      type: 'List<AWS::EC2::AvailabilityZone::Name>',
      default: 'N/A',
    });

    new ChildService(this, "Service", {
      vpcId: vpcId.valueAsString,
      vpcZones: vpcZones.valueAsList,
      vpcPublicSubnetIds: vpcPublicSubnetIds.valueAsList,
      vpcPrivateSubnetIds: vpcPrivateSubnetIds.valueAsList,
    });
  }
}

constructs/childService.ts

export class ChildService extends Construct {
  public constructor(
    scope: Construct,
    id: string,
    params: {
      vpcId: string;
      vpcZones: string[];
      vpcPublicSubnetIds: string[];
      vpcPrivateSubnetIds: string[];
    }
  ) {
    super(scope, id);
    const { vpcId, vpcZones, vpcPublicSubnetIds, vpcPrivateSubnetIds } = params;

    const vpc = Vpc.fromVpcAttributes(this, "Vpc", {
      vpcId,
      availabilityZones: vpcZones,
      publicSubnetIds: vpcPublicSubnetIds,
      privateSubnetIds: vpcPrivateSubnetIds,
    });

    new OverriddenCfnOutput(this, "url", { value: "urlOfAThing" });
  }
}

Parent (completely different repo):

utils.ts

export type ValuesOf<T extends any[]> = T[number];

/**
 * References the output of a child stack via their `key` (as opposed to their
 * `export`, which is globally accessible in a given CloudFormation env)
 */
export function getChildOutputs<
  T extends string,
  /**
   * NOTE: we assume the string comes through, even though it might not be
   * found, because we can't actually validate during synthesis at this time,
   * and the `key` we're looking for would _always_ be truthy because it would
   * be represented as a `token`, which is just a string
   */
  U = { [K in T]: string }
>(childStack: CfnStack, keys: T[]): U {
  return keys.reduce((acc: any, key) => {
    acc[key] = (childStack.getAtt(`Outputs.${key}`) as unknown) as string;
    return acc;
  }, {});
}

constructs/service.ts

...
const vpc = new Vpc(this, "Vpc");

// import child stack template and pass parameters
const childService = new CfnStack(this, "ChildService", {
  templateUrl: cfnTemplateBucket.urlForObject(
    "stage/project-name/cdk.out/stackName.template.json"
  ),
  parameters: {
    vpc: vpc.vpcId,
    vpcZones: toCommaDelimitedString(vpc.availabilityZones),
    vpcPublicSubnetIds: toCommaDelimitedString(
      vpc.publicSubnets.map(subnet => subnet.subnetId)
    ),
    vpcPrivateSubnetIds: toCommaDelimitedString(
      vpc.privateSubnets.map(subnet => subnet.subnetId)
    ),
  }
});
// get outputs from child service
const childServiceOutputs = getChildOutputs(childService, ["url"]);

// imported value of CFN Output, named "url" in the child stack
childServiceOutputs.url; // 'urlOfAThing'
...

[ivawzh]

Probably offtopic, but in the current doc

fromVpcAttributes(scope, id, attrs)
Import an exported VPC.

What does exported VPC mean? I thought it was referring to Cfn Outputs, but I can only get empty results back with Fn.importValue approach.

const vpc = Vpc.fromVpcAttributes(this, `ImportedVpc`, {
  vpcId: Fn.importValue('OutputVpcId'),
  availabilityZones: [hardcodedAzs]
})

vpc.publicSubnets.map(i=>i.subnetId) // => []
vpc.privateSubnets.map(i=>i.subnetId) // => []
vpc.isolatedSubnets.map(i=>i.subnetId) // => []

[kevin-lindsay-1]

@ivawzh an export is an output that gets placed into the global CFN scope. If you're using nested stacks, outputs are the way to go.

[ivawzh]

Do you mean exporting the VPC ID with something like this?

    new CfnOutput(this, 'OutputVpcId', {
      value: this.vpc.vpcId,
      exportName: 'OutputVpcId',
    })

But fromVpcAttributes require many more attributes. Or is there a way I could export the whole VPC object?

Yes, I am doing cross-cdk-repo VPC sharing.

[kevin-lindsay-1]

@ivawzh yeah, when I originally discovered the workaround I needed to output all of the values that are needed to accurately reference it. Unless the API has changed since I wrote it that is likely what you need to do.

[asanthan-amazon]

I am thinking why can't CDK populate the other attributes of VPC dynamically by quering respective VPC APIs using vpcId.