Dbt: Add a "full run" indicator to Jinja context

Created on 26 Mar 2020  路  4Comments  路  Source: fishtown-analytics/dbt

Describe the feature

Add a Jinja context that would indicate whether a dbt run was invoked with or without model selection arguments.

Who will this benefit?

In dbt projects with an important number of incremental models sourcing large event streams, where it would likely be a really bad idea to do a dbt run --full-refresh, this context would enable developers to put protective measures within the models to avoid full refresh runs on the whole project happening by accident or lack of care.

Example logic / context:
Assuming that the proposed context flags.FULL_RUN would return True if dbt was invoked without specifying models, a developer could add the following to his models:

{% if flags.FULL_REFRESH and flags.FULL_RUN %}
    {{ exceptions.raise_compiler_error("Whoa there cowboy, are you sure you want to do that?") }}
{% endif %}

More complex logic could be done as well to allow full runs _only_ if a specific variable is present or if executed within a specific environment, for example.

enhancement

Most helpful comment

@drewbanin That would be even better -- I didn't dare to be so bold in my request!

All 4 comments

Thanks for opening this issue, @MartinGuindon! I think rather than flags.FULL_RUN, we might be able to add a different flag that includes the provided CLI arguments. That might look like flags.args.models and flags.args.exclude, for instance.

My thinking is just that FULL_RUN isn't super well-defined at the moment, and a one-to-one mapping with the provided CLI selectors seems like the more straightforward implementation here. Do you buy that?

@drewbanin That would be even better -- I didn't dare to be so bold in my request!

@drewbanin this sounds very useful! I have a use case where I'd like to change schemas for _selected_ models on the fly, while not changing schemas for non-selected models. For example, I'd like to materialize a model I'm working on in a special "validation" schema but still make sure it gets its upstream data from the respective "production" schema. That way I could do a validation run of a model without having to full rebuild my development environment with production data.
So, being able to interrogate flags.args.models during a macro like get_custom_schema would be super handy.

@clausherther could you drop some thoughts into #2527? Your use-case sounds just like the one we have in mind for that feature! I think these two issues are definitely related :)

Was this page helpful?
0 / 5 - 0 ratings