Prefect: DbtShellTask does not work without profiles_dir parameter

[fertek]

Hi,
profiles_dir parameter is marked as optional, but without that parameter DbtShellTask fails:
TypeError('expected str, bytes or os.PathLike object, not NoneType')

It happens because of this line:
https://github.com/PrefectHQ/prefect/blob/6ef1f36362a46762985dfcab27a3c4ea0ec8084a/src/prefect/tasks/dbt/dbt.py#L146

[Zaubeerer]

As @mhmcdonald, mentioned to me in this Slack thread:

you need to provide a profiles_dir to let dbt know where your profiles.yml file lives. Perhaps profiles.yml located in the same directory as where your flow files are located, but also very possible it's not.

This also indicates that it should not be an optional parameter, right?

[fertek]

dbt docs at https://docs.getdbt.com/dbt-cli/configure-your-profile/#using-a-custom-profile-directory says:

By default, dbt expects your profiles.yml file to be located in the ~/.dbt/ directory.

Therefore, I think that without profiles_dir parameter specified the ~/.dbt/ directory should be used.

[Zaubeerer]

And it should be ensured, that the ~/.dbt/ folder is created, if not existant.
I got several folder or file does not exist warnings...

[mhmcdonald]

@fertek @Zaubeerer It's been a while since I wrote this but I believe I left it as optional because you might have the location already set through the dbt envar DBT_PROFILES_DIR. If it wasn't possible to set the dir this way, I would have made it required.

I think defaulting the profiles_dir path to ~/.dbt/ is a great idea, just as long as the DBT_PROFILES_DIR envar gets priority in determining the path location. This would be in keeping with how dbt checks for the profiles.yml file.

[joshmeek]

Closed by #3462