Pulumi: Explore self-describing config

[joeduffy]

At the moment, which config variables are required by a Pulumi program are completely opaque to the Pulumi runtime, CLI, and service. As a result, you need to run the program to figure out which config hasn't been set ... one config variable at a time. This yields a very poor experience, especially compared to competitive technologies. Ideally a program or library could advertise its config.

Here is one possible idea. There are many other avenues we could pursue, of course:

First, we introduce the notion of a "config module." Each Pulumi program can define one. There would be a default convention for each language host, so config.js in JavaScript and config.py in Python. The requirement for these modules is that they do not have side-effect, and especially that they do not allocate resources (something we can enforce); do very little other than read and prepare config.

This config module would contain statements very much like what we do today

import { Config } from "@pulumi/pulumi";
const config = new Config("program");
// region controls what AWS region to deploy into.
export let region: string = config.require("region");
// If createDomain is true, we will create a DNS entry.
export let createDomain: boolean = config.getBoolean("createDomain") || false;
// ...

except that each config variable may also include

• a description.
• a default (rather than ad-hoc code per the above).

(Note that, based on whether get or require is called, we already know if something is optional; and we can use the Boolean, etc. suffix to know what its expected type is.)

So, for instance:

import { Config } from "@pulumi/pulumi";
const config = new Config("program");
export let region: string = config.require("region", "the region to deploy into");
export let createDomain: boolean = config.getBoolean(
    "createDomain", false, "if true, create a DNS entry");
// ...

The CLI would know to execute this module if it wants to display config variables for the current program. This would give us the ability to display not only the currently set variables, but also those that need to be set. It would give us the ability to check _all_ variables up front, to ensure they are set, before even running the program. And it would also even give us an interactive way of setting config:

$ pulumi config set
Enter "region" [required; the region to deploy into]: us-west-2
Enter "createDomain" [optional, boolean; if true, create a DNS entry]: 

Note that figuring out how to execute the transitive closure of modules might be tricky, except that I will note that we now have logic to find the transitive closure as part of computing required plugins.

If we go down a path like this, we could even consider changing our config APIs. They are already a bit ... strange. For example, we could make an API more like optimist (https://github.com/substack/node-optimist), that used a more declarative style for expressing the arguments:

import { Config } from "@pulumi/pulumi";

export const config = new Config<{ region: string, createDomain?: boolean }>("program", {
    region: "the region to deploy into",
    createDomain: {
        default: false,
        description: "if true, create a DNS entry",
    },
});

[joeduffy]

Here's another idea.

What if we put the config metadata inside of the Pulumi.yaml file. This would not only give us the above properties, while also eliminating the need to manually specify the name as part of the bag.

For example, in Pulumi.yaml:

name: program
config:
    region:
        description: The region to deploy into.
    createDomain:
        type: boolean
        default: false
        description: If true, create a DNS entry.

and then inside your program, simply point us to the Pulumi.yaml file:

export const config = new Config<{ region: string, createDomain?: boolean }>("./Pulumi.yaml");

The type argument there is optional but gives you good typing inside of TypeScript. Obviously, there would be no such thing in JavaScript:

export const config = new Config("./Pulumi.yaml");

I also think some simple shortcuts could make this even easier.

First, if there's no file part in the path, assume a Pulumi.yaml file part:

export const config = new Config("./");

And, if you omit the file path altogether, just assume the Pulumi.yaml is in the current directory:

export const config = new Config();

[lukehoban]

I like this latest proposal. A bit of a shame to have to say everything twice in different places, but feels like a pay-for-play way to add this extra metadata.

There's also a nice symmetry with Pulumi.stack.yaml holding values for these config sections. The naming of this file didn't really make sense previously, but starts to make more sense if the config is specified in Pulumi.yaml.

Also - what's the scenario that requires specifying where to get the config? Why not just always use Pulumi.yaml for consistency (and so that the code isn't required at all to understand what is the applicable config)?

[joeduffy]

Great points. This feels like a strictly better place than we are.

The only point where I hesitate is that it definitely goes against our "it's just code" philosophy; but given that config is the one place where it _isn't_, in fact, just code, this feels right to me.

what's the scenario that requires specifying where to get the config

I definitely wasn't imagining supporting or suggesting a different file than Pulumi.yaml. I was mostly imagining the case where you had a module like this

/
    config/
        index.ts *config is defined here*
    index.ts
    Pulumi.yaml

In this case, you'd need to say new Config("../"). We do this in the Terraform providers because we end up spitting out multiple config modules. I suppose we could do something like search the current directory and, if not found, keep searching upwards until we find a Pulumi.yaml? That at least doesn't give you the impression you can pass an arbitrary YAML file here.

Note that this is currently M16, post release. If we like this general path, we may want to change this.

[ellismg]

I suppose we could do something like search the current directory and, if not found, keep searching upwards until we find a Pulumi.yaml

Just to confirm that I understand this correctly. I assume that each Pulumi package now brings its own Pulumi.yaml with it (where as previously it did not?) and that we'd use this same sort of technology in the providers themselves (e.g. @pulumi/aws node package as a Pulumi.yaml in the root, and it does similar things to get at configuration? In this world, I assume we have to do special work so the partial paths are relative to their correct locations within the node_modules folder itself?

[justinvp]

We want to align with the direction we're heading for configuring PaC policies.