Great_expectations: Multiple environment variables under a single key fail to substitute

Created on 20 Jun 2020  路  3Comments  路  Source: great-expectations/great_expectations

Describe the bug
Multiple environment variables under a single key fail to substitute

To Reproduce

  1. create a datasource with a connection string that pulls in multiple environment variables.
latest/features/datasource.html
datasources:
  staging:
    credentials:
      connection_string: "bigquery://${PROJECT}/${DATASET}"
    class_name: SqlAlchemyDatasource
    module_name: great_expectations.datasource
    data_asset_type:
      module_name: great_expectations.dataset
      class_name: SqlAlchemyDataset
 ```
2. Set an environment variable `export PROJECT=my_secret_project_1234` and another `export DATASET=foo`.
3. Run `great_expectations datasource list`
4. Note the incomplete substitution

1 Datasource found:

  • name: staging
    module_name: great_expectations.datasource
    class_name: SqlAlchemyDatasource
    credentials:
    connection_string: bigquery://my_secret_project_1234/${DATASET}
    data_asset_type:
    module_name: great_expectations.dataset
    class_name: SqlAlchemyDataset
    ```

    1. Run any feature that uses the datasource and note the backend specific error about failing to connect to a database.

Expected behavior
I expect all environment variable substitutions to be substituted.

Environment (please complete the following information):

  • first seen on ubuntu w/ GE 0.11.1
  • reproduced on macOS w/ GE 0.11.4
bug stale

Most helpful comment

@anthonyburdi and I discovered and reproduced this together.

All 3 comments

@anthonyburdi and I discovered and reproduced this together.

I'm currently working on a fix to this & a related bug and should have a PR soon. (branch)

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?\n\nThis issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Was this page helpful?
0 / 5 - 0 ratings