Dbt: RecursionError: maximum recursion depth exceeded in comparison

[ArthurZ]

Issue

RecursionError: maximum recursion depth exceeded in comparison

Issue description

Running dbt run when my project has 3 files in the model folder works, but adding one more errors out with the above.

Results

Error above should not occur, must be able to complete dbt run

System information

The output of dbt --version:

installed version: 0.11.1
   latest version: 0.11.1

Up to date!

The operating system you're running on: MacOS 10.13.6

The python version you're using (probably the output of python --version)
Python 3.6.6 :: Anaconda custom (64-bit)

Steps to reproduce

In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example models, etc are all very helpful here.

Just keep adding new files with SQL into the models/afolder/

The output of the --debug command:

(MainThread): maximum recursion depth exceeded in comparison
2018-10-22 16:36:31,926 (MainThread): Traceback (most recent call last):
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/main.py", line 72, in main
    results, succeeded = handle_and_check(args)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/main.py", line 117, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/main.py", line 172, in run_from_args
    results = run_from_task(task, proj, parsed)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/main.py", line 180, in run_from_task
    result = task.run()
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/task/run.py", line 26, in run
    results = runner.run(query, ModelRunner)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/runner.py", line 244, in run
    return self.run_from_graph(Selector, Runner, query)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/runner.py", line 197, in run_from_graph
    manifest, linker = self.compile(self.project)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/runner.py", line 186, in compile
    return compiler.compile()
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/compilation.py", line 241, in compile
    manifest = dbt.loader.GraphLoader.load_all(self.project, all_projects)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/loader.py", line 21, in load_all
    nodes.update(loader.load_all(root_project, all_projects, macros))
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/loader.py", line 86, in load_all
    macros)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/loader.py", line 102, in load_project
    macros=macros)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/parser/base_sql.py", line 67, in load_and_parse
    macros)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/parser/base_sql.py", line 95, in parse_sql_nodes
    macros=macros)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/parser/base.py", line 125, in parse_node
    schema_override = config.config.get('schema')
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/model.py", line 82, in config
    self.in_model_config)
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/model.py", line 50, in _merge
    merged_config.copy(), config.copy()
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/utils.py", line 251, in deep_merge
    last = copy.deepcopy(lst.pop(len(lst)-1))
  File "/Users/azme/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/Users/azme/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/azme/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/azme/anaconda3/lib/python3.6/copy.py", line 281, in _reconstruct
    if hasattr(y, '__setstate__'):
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/clients/jinja.py", line 165, in __getattr__
    self.package_name = self.name
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/clients/jinja.py", line 165, in __getattr__
    self.package_name = self.name
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/clients/jinja.py", line 165, in __getattr__
    self.package_name = self.name
  [Previous line repeated 320 more times]
  File "/Users/azme/anaconda3/lib/python3.6/site-packages/dbt/clients/jinja.py", line 162, in __getattr__
    if name in ['unsafe_callable', 'alters_data']:
RecursionError: maximum recursion depth exceeded in comparison

[drewbanin]

Thanks for the report @ArthurZ! This one is definitely new to me. You indicated:

Running dbt run when my project has 3 files in the model folder works, but adding one more errors out with the above.

I have to imagine that this has more to do with the context of the models, and not the number of models itself! Are you able to share some example code that you're using? That might help us track this down

[raybuhr]

Whenever I got they error before it had nothing to do with recursion or stack traces in Python, but was always typos or unquoted variables. For example, if you build a dependent model with the ref function, make sure you quote the model name

E.g.

SELECT *
FROM {{ ref(my_cool_model) }}

Could blow up dbt and return an error like that. Where fixing it to:

SELECT *
FROM {{ ref('my_cool_model') }}

Would run as expected

[drewbanin]

I am going to pick my jaw up from off of the floor, and then I am going to submit a bugfix for this. How am I just hearing about this issue???

Thanks for the test case @raybuhr - i can reproduce that locally

[ArthurZ]

Hi all there,

Thank you for jumping on this issue so quick and sorry for the delay in replying.

To start off, @beckjake it is a good try, but no, all is quoited. And the SQL is a valid Redshift one I can actually execute.

So the offending code (mind it is modified for content, but not logic to protect the innocent) is:

SELECT systemid,
       DAY,
       COUNT(*) TEST_TOTAL
FROM
    (SELECT DISTINCT i.systemid,
                i.testid,
                l.created_at::date AT TIME ZONE 'UTC' AS DAY
     FROM coalesced_live_shards.test i
     JOIN coalesced_live_shards.test_allowed_gateway a ON (i.systemid = a.systemid
                                                              AND i.testid = a.testid)
     JOIN coalesced_live_shards.test_audit_log l ON (i.systemid = l.systemid
                                                        AND i.testid = l.testid)
     WHERE l.display_status = 'sent'
         AND i.active = 1
         AND a.gatewayid != -1
     UNION 
SELECT DISTINCT i.systemid,
                i.testid,
                l.created_at::date AT TIME ZONE 'UTC' AS DAY
     FROM coalesced_live_shards.test i
     JOIN coalesced_live_shards.test_audit_log l ON (i.systemid = l.systemid
                                                        AND i.testid = l.testid)
     WHERE l.display_status = 'sent'
         AND i.active = 1
         AND (i.systemid,
              i.testid) NOT IN
             (SELECT systemid,
                     testid
              FROM coalesced_live_shards.test_allowed_gateway)) a
GROUP BY systemid,
         DAY

[beckjake]

Hi @ArthurZ - I don't see any jinja in there at all, am I missing something? Is this the rendered form?

And I can compile/run the given SQL in dbt (obviously with "relation does not exist" errors since I don't have your schema)

[ArthurZ]

Hi @beckjake I did not use any Jinja. The underlined relations are actually complex views in my case. Would that be somehow of an issue?

[ArthurZ]

Investigating further reveals that adding a new file (even if it contains the same SQL that worked in an already existing file) generates this error.
My folder structure looks so:

[drewbanin]

Hey @ArthurZ - dbt just passes along your SQL to the warehouse, so I think this error _must_ come from either jinja in your model files, or possibly from configuration in dbt_project.yml or schema.yml.

It's super hard to say why this would happen without seeing some more code and/or logs. If you're in the dbt Slack, can you DM me? I'm @drew.banin. Or, feel free to post as much of your model code, dbt_project.yml, and schema.yml contents in this issue (scrubbing anything proprietary). I'd also be interested to see more of your logs/dbt.log file -- particularly the lines just above:

(MainThread): maximum recursion depth exceeded in comparison
2018-10-22 16:36:31,926 (MainThread): Traceback (most recent call last):

Thanks for helping us get this sorted!

[raybuhr]

The new files might not be causing the error then. Can you try removing those two new queries and running dbt again?

[ArthurZ]

Hey @drewbanin the log contents are posted above when I opened the ticket. There is nothing new on each run. Now I inspected both, the dbt_project.yml, and schema.yml files and can't see anything that would raise my suspicion. I will Slack you time permits. A big thank you!

[ArthurZ]

The new files might not be causing the error then. Can you try removing those two new queries and running dbt again?

Yes @raybuhr, if I delete just one of the newly added files dbt run works. Mind I did not even bother cleaning the schema.yml

[ArthurZ]

Heh, so it was not even the SQL part but the verbose=TRUE directive in the config that appears to be even redundant although worked in several files but not too many.
Big thanks to @drewbanin for setting me straight.

[drewbanin]

Just to be super clear for anyone else that finds this issue: The problem was that @ArthurZ specified True as TRUE, which _is not_ defined by Jinja. As a result, this code triggered the recursive exception that arises for undefined variables during parsing.

This was mostly fixed in #1085, though let's merge #1086 before closing this issue. cc @beckjake

[drewbanin]

1086 was merged -- closing this! Thanks for the report and the insight @ArthurZ and @raybuhr!