Dbt: Invoking dbt as a module

[gouline]

Describe the feature

It would be nice to have a straightforward way of running dbt as a Python module.

To give some context, I use my own build tool https://github.com/gouline/molot that's written in Python and provides basic support for targets, dependencies and arguments. Something half way between Make and Gradle that helps with CI configuration. Things that can be done in Python (e.g. boto3, snowflake-connector-python) are done natively, everything else it just calls shell commands in a subprocess like a Makefile would.

So I'd like a way to invoke multiple dbt commands (e.g. run and docs generate) in one target by just importing dbt.main and calling a function multiple times. The way things are now, main() calls sys.exit(), which exists my outer build as well, and handle_and_check() requires raw command-line arguments passed manually as a list. Ideally, there should be a function of the form dbt.main.execute(command='run', profiles_dir=PROFILES_DIR, ...) that would execute a command and throw any exceptions for the caller to handle.

Describe alternatives you've considered

My current workaround is just invoking dbt command as a subprocess, but it seems like a backward way of doing it, considering that both are Python applications.

Additional context

Closest thing I could find on the issue tracker was https://github.com/fishtown-analytics/dbt/issues/1488, but that sounds more complex than what I'm proposing.

Who will this benefit?

This would benefit anyone who orchestrates their builds with Python scripts.

[drewbanin]

Hey @gouline - kind of funny we didn't already have an issue for this one :)

We've kicked around the idea of providing a public API for dbt in Python a couple of times now. I'm happy for us to add a public method like dbt.main.execute(command='run', profiles_dir=PROFILES_DIR, ...) which implements logic similar to handle_and_check.

I'd still like to provide a rich Python-based interface for running dbt projects in the future. I think that would entail model selection, configuration, execution, etc, etc. In this case though, I think a top-level method like execute (or similar) gets us moving in the right direction.

Thanks for raising this!

[Dandandan]

Hello I like to help out on this issue.
Any pointers on how this should be implemented?

Should it just pass on the args to the parser?