Dbt: Set configs in schema.yml files

[drewbanin]

Describe the feature

Let's bring the schema.yml spec and node configs together! Right now, some configs can only be set for a node in the node's .sql file with config() or in the dbt_project.yml file. These configs include:

• materialized
• enabled
• tags
• database-specific configs (bind, sort, dist, etc)
• etc

These configs _can not_ be configured in the schema.yml file. At the same time, there are some configs that can _only_ be set in a schema.yml file:

• description
• columns (tags, descriptions, tests, etc)
• tags (for sources)

Instead, it would be ideal if these configs could be supplied in either place, with a sensible and well-defined precedence rules.

Example usage:

-- models/my_model.sql
/*
  Configure the description and columns for a model inline in a config.
  Note: I'd probably advise against doing this, but it should be possible
*/

{{ config(
  description="This is my model description",
  columns=[
    [
      {"name": "id", "description": "The primary key"}
    ]
) }}

select ...

# models/schema.yml
version: 2
models:
 - name: my_model
   config:
      tags: ["nightly"]
      materialized: view

Some configs, like tags or freshness in a source are set at the root level of the source. Today, this looks like:

version: 2
sources:
 - name: my_source
   tags: [my_tag]

We should continue to support this syntax, but it would also be good to allow a config level key which scopes these configurations, eg:

version: 2
sources:
 - name: my_source
   config:
      tags: [my_tag]

We should _not_ allow these configs to be configured at both the root-level and the config level for parsing simplicity.

Additional context

I fear that this might be a really big departure from how parsing works today. Some fields, like description are patched into model objects. Would this feature require us to drastically change how we parse nodes in dbt?

[drewbanin]

@beckjake really curious to hear your rough thinking on how tractable a change like this would be

[beckjake]

Off the top of my head, there's nothing about this that's hard in the sense of creating difficult data ordering dependencies or whatever. Since configurations are more unified now, allowing them inside a config block in the patch object should be more straightforward. Probably some extra work around config with patches and special-case handling existing fields like tags.

We will experience some awkwardness around the database/schema/alias fields. Maybe this is the time to move them (and tags?) finally off the node object and onto node.config, and just set them up as properties on the node that forward to the config? I bet that change would be very breaking and a huge mess! Currently those fields can get out of sync, which I think is even more awkward, and I'd have to look at the generate_*_name macros to see how those all pan out. I guess we'd have to defer those to after patch application, which doesn't seem too much worse.

So: lots of work, but doable.

[jtcohen6]

What do we think about adding this to 1.0?

[darrenhaken]

I would love this feature to be added.

I asked a question about meta being hierarchical in the DBT channel.
I'd like to be able to define metadata at a higher level (such as mart) and have it propagate.

An example would be owner like in your examples.

@drewbanin signposted this issue for me to comment out.

[DVAlexHiggs]

Not finding any information about 1.0.0 @jtcohen6
Is this the next release and you're skipping 0.18? Very exciting if so!

I'm hoping this feature will be in the next release, since the lack of it will be massively breaking for a lot of people, organisations and packages, dbtvault included.

Is there anything I can do to contribute? If so, where can I start?

[jtcohen6]

The next minor release (Marian Anderson) will be 0.18.0. It will include several issues currently tagged with the Marian Anderson milestone, and we will cut it from the current default branch (dev/marian-anderson).

Over the course of the next few minor versions, we're chipping away at a stable 1.0 version of dbt (aspirational target: EOY). As part of that effort, we're tagging and compiling a list of issues that we definitely want to include in our first major release.

• Milestone 1.0.0
• Open 1.0.0 issues

These issues require foundational changes to the current dbt codebase. Following @beckjake's comment above, reconciling node and resource-YML configs will require a significant lift in some potentially tricky spots. I'm appreciative of your interest in this change, and for offering to help; I just don't think we're in a place right now to help an external contributor work through something this knotty.