Dbt: get_relation method returns None for Redshift external tables

Created on 2 Apr 2020  路  2Comments  路  Source: fishtown-analytics/dbt

Describe the bug

get_relation method (https://docs.getdbt.com/docs/writing-code-in-dbt/jinja-context/adapter/#get_relation) returns None when provided with database, schema and table name of a Redshift external table (AWS Spectrum).

Steps To Reproduce

  1. Create an external table with AWS Spectrum: https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-tables.html
  2. Attempt to get_relation on that external table.

Expected behavior

Should return a regular relation as it does for native Redshift tables, e.g. an output like analytics.spectrum_schema.mytable instead of None

### Screenshots and log output
-

System information

Which database are you using dbt with?

  • [ ] postgres
  • [x] redshift
  • [ ] bigquery
  • [ ] snowflake
  • [ ] other (specify: - - - )

The output of dbt --version:

0.16.0

The operating system you're using:
MacOS Mojave

The output of python --version:
3.6.2

### Additional context
-

bug good first issue redshifpg

Most helpful comment

Thanks for the report @tomekzbrozek!

Looks like we use the Postgres code to query the information schema on redshift: https://github.com/fishtown-analytics/dbt/blob/dev/octavius-catto/plugins/redshift/dbt/include/redshift/macros/adapters.sql#L154

but redshift provides a totally different set of tables for fetching this info: https://docs.aws.amazon.com/redshift/latest/dg/r_SVV_EXTERNAL_TABLES.html

I think we'd need to copy the "base" pg implementation into the redshift plugin, then update the code to also fetch external tables too.

All 2 comments

Thanks for the report @tomekzbrozek!

Looks like we use the Postgres code to query the information schema on redshift: https://github.com/fishtown-analytics/dbt/blob/dev/octavius-catto/plugins/redshift/dbt/include/redshift/macros/adapters.sql#L154

but redshift provides a totally different set of tables for fetching this info: https://docs.aws.amazon.com/redshift/latest/dg/r_SVV_EXTERNAL_TABLES.html

I think we'd need to copy the "base" pg implementation into the redshift plugin, then update the code to also fetch external tables too.

thank you, yup I also used SVV_EXTERNAL_TABLES as a workaround to the issue reported here, pasting below to give a flavour.

({{ schema }} and {{ table }} vars are defined earlier in the body of my macro that serves a different purpose, I basically needed to know if the external table exists of not, this is why I came across this issue :) )

-- check_if_external_table_exists returns 1 if a given external table exists, 0 if doesn't
{% set check_if_external_table_exists %}
SELECT
  COUNT(*) AS if_exists_flag
FROM
  SVV_EXTERNAL_TABLES
WHERE
  schemaname || '.' || tablename = '{{ schema }}.{{ table }}'
{% endset %}
-- the loop below fetches results of check_if_external_table_exists:
-- 0 if table doesn't exist, 1 if table exists, 2 for other (erroneous?) cases
-- more context on the `execute` Jinja variable:
-- https://docs.getdbt.com/docs/writing-code-in-dbt/jinja-context/execute
{% if execute %}
  {% set if_exists_flag = run_query(check_if_external_table_exists).columns[0].values()[0] | int %}
{% else %}
  {% set if_exists_flag = 2 | int %}
{% endif %}
Was this page helpful?
0 / 5 - 0 ratings

Related issues

chrisburrell picture chrisburrell  路  3Comments

beckjake picture beckjake  路  3Comments

boxysean picture boxysean  路  3Comments

jtcohen6 picture jtcohen6  路  3Comments

smomen picture smomen  路  3Comments