Dbt: Create custom jinja filters

[drewbanin]

From Haney

Currently Jinja doesn't support (or at least I can't find support for) many set operations. So, I'm using adapter.get_columns_in_table but I want to "ignore" a list of columns. Jinja doesn't seem to support things like set(a) - set(b) but if there was a way to supply custom filter functions that were passed down to Jinja I could have handed it a custom function to do something like {{ set_a|not_in(set_b) }}

[josegonzalez]

+1 This would be pretty good. An example use case is loading a profile with a DSN as an environment variable, and then a filter that retrieves a specific part of that DSN. For example:

dbt:
  outputs:
    warehouse:
      type: "{{ dsn_env_var('REDSHIFT_URL', 'scheme') }}"
      threads: 4
      host: "{{ dsn_env_var('REDSHIFT_URL', 'host') }}"
      port: {{ dsn_env_var('REDSHIFT_URL', 'port') }}
      user: "{{ dsn_env_var('REDSHIFT_URL', 'user') }}"
      pass: "{{ dsn_env_var('REDSHIFT_URL', 'password') }}"
      dbname: "{{ dsn_env_var('REDSHIFT_URL', 'database') }}"
  target: warehouse

It looks like a context is injected here, but allowing customization of that base-context would allow us to inject custom filters.

[drewbanin]

It may be compelling to revisit this issue in the future when dbt has a more well-defined Python API. Until then, there isn't a good way to write Python code (to implement filters) in the context of a dbt project.