Dbt: `dbt docs serve` fails with `Not a directory: 'xdg-settings'` error

[peteratemarsys]

Describe the bug

I ran

• dbt docs generate (ran successfully)

+ dbt docs serve: fails with [Errno 20] Not a directory: 'xdg-settings' and the dbt docs does not open ni a new web browser tab

• if I comment out the line webbrowser.open_new_tab('http://127.0.0.1:{}'.format(port)) from /usr/local/Cellar/dbt/0.16.0_1/libexec/lib/python3.7/site-packages/dbt/task/serve.py
• and open http://localhost:8080/ manually then the docs website works as expected

Expected behavior

I expect a browser page to open up in my default browser with the docu page

Screenshots and log output

2020-04-03 19:42:31.269074 (MainThread): Running with dbt=0.16.0
2020-04-03 19:42:31.434592 (MainThread): running dbt with arguments Namespace(cls=<class 'dbt.task.serve.ServeTask'>, debug=False, log_cache_events=False, log_format='default', partial_parse=None, port=8080, profile=None, profiles_dir='/Users/plukacs/.dbt', project_dir=None, record_timing_info=None, rpc_method=None, single_threaded=False, strict=False, target=None, test_new_parser=False, use_cache=True, vars='{}', warn_error=False, which='serve', write_json=True)
2020-04-03 19:42:31.435831 (MainThread): Tracking: tracking
2020-04-03 19:42:31.442566 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105774750>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105cb3fd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10579cc10>]}
2020-04-03 19:42:31.445841 (MainThread): Serving docs at 0.0.0.0:8080
2020-04-03 19:42:31.446086 (MainThread): To access from your browser, navigate to:  http://localhost:8080
2020-04-03 19:42:31.446340 (MainThread): Press Ctrl+C to exit.


2020-04-03 19:42:31.449623 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105cc2d50>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105cc2dd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105cc2e90>]}
2020-04-03 19:42:31.450119 (MainThread): Flushing usage events
2020-04-03 19:42:31.961617 (MainThread): Encountered an error:
2020-04-03 19:42:31.961900 (MainThread): [Errno 20] Not a directory: 'xdg-settings'
2020-04-03 19:42:31.963837 (MainThread): Traceback (most recent call last):
  File "/usr/local/Cellar/dbt/0.16.0_1/libexec/lib/python3.7/site-packages/dbt/main.py", line 81, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/Cellar/dbt/0.16.0_1/libexec/lib/python3.7/site-packages/dbt/main.py", line 159, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/Cellar/dbt/0.16.0_1/libexec/lib/python3.7/site-packages/dbt/main.py", line 212, in run_from_args
    results = task.run()
  File "/usr/local/Cellar/dbt/0.16.0_1/libexec/lib/python3.7/site-packages/dbt/task/serve.py", line 36, in run
    webbrowser.open_new_tab('http://127.0.0.1:{}'.format(port))
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/webbrowser.py", line 86, in open_new_tab
    return open(url, 2)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/webbrowser.py", line 75, in open
    register_standard_browsers()
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/webbrowser.py", line 530, in register_standard_browsers
    raw_result = subprocess.check_output(cmd, stderr=subprocess.DEVNULL)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'xdg-settings'

2020-04-03 19:42:31.964146 (MainThread): unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 8080)>

System information

Which database are you using dbt with?

• bigquery

The output of dbt --version:

installed version: 0.16.0
   latest version: 0.16.0

The operating system you're using:
MacOS Catalina v10.15.4

The output of python --version:
Python 3.7.3

[beckjake]

I can't reproduce this locally, even in Python 3.7.3 exactly, but I have a few troubleshooting questions for you if you'd be so kind:

From a terminal window:

• What does echo $BROWSER say?
• What about echo $DISPLAY?
• What does which xdg-settings say?
  
  
  
  • If it reports a path, what happens when you type xdg-settings get default-web-browser?
  
  

There's a couple fixed bugs in 3.7 that seem like they might be related:

https://bugs.python.org/issue31014
https://bugs.python.org/issue35308

There's also this jupyter bug, which doesn't do much to illuminate the situation, at least for me - but it does indicate some sort of issue in Python: https://github.com/jupyter/notebook/issues/3746

[peteratemarsys]

• What does echo $BROWSER say?
  
  
  
  • nothing, returns empty
  
  
• What about echo $DISPLAY?
  
  
  
  • /private/tmp/com.apple.launchd.6byZvGIeM6/org.macosforge.xquartz:0
  
  
• What does which xdg-settings say?
  
  
  
  • xdg-settings not found
  
  

Yes, I've also found that issue and that led me to the idea of skipping the browser tab opener step (there is a command line option for that in jupyter). Some comments say that the issue appeared after upgrading to MacOS Catalina.

Thanks for looking into this!

[beckjake]

As a workaround, I suspect you can unset DISPLAY before you run dbt docs serve, or run it as DISPLAY="" dbt docs serve. Looking at how the webbrowser module is implemented, it goes through the xdg-settings route only if DISPLAY is set.

In 3.8 there appears to be slightly different behavior from subprocess.check_call(['xdg-settings', ...]) if such a binary doesn't exist, so upgrading your python installation may also fix it for you.

I'm on Catalina and can't seem to reproduce getting a NotADirectoryError from basically any execution of subprocess.check_call at all, but I do believe that Catalina is somehow responsible.

[peteratemarsys]

DISPLAY="" and dbt docs serve works, thanks!

I was unable to update my dbt python dependency from 3.7 to 3.8. I use brew to install dbt and that automatically links python 3.7 as its dependency. After an hour of googling I was not able to re-link it to brew's [email protected]. Any suggestions?

My ultimate solution now is to run dbt from Docker but I'm still curious how to do the above.

Thanks

[beckjake]

I think we would have to update the dbt homebrew formula to point at 3.8 in that case, so I'd stick with the DISPLAY fix.

[drewbanin]

I'm going to close this issue as it doesn't sound like a near-term code change is required here. Glad you got it sorted though @peteratemarsys!