Great_expectations: Using a postgres expectation store fails in v0.13

[ryanaustincarlson]

Describe the bug
I can't get a postgres expectation store working in v0.13 -- I'm not sure if the format changed or if this is a bug, so I wanted to flag it here.

Existing docs for reference here.

To Reproduce
Steps to reproduce the behavior:

1. Spin up a postgres instance (e.g., with docker as below)

docker run \
    -e POSTGRES_USER=great \
    -e POSTGRES_PASSWORD=expectations \
    -e POSTGRES_DB=ge \
    -p "5432:5432" \
    postgres:13.1

1. Use the following snippet in your great_expectations.yml

stores:
  postgres_expectations_store:
    class_name: ExpectationsStore
    store_backend: 
      class_name: DatabaseStoreBackend
      credentials:
        drivername: postgresql
        host: localhost
        port: 5432
        username: great
        password: expectations
        database: ge
expectations_store_name: postgres_expectations_store

1. Try to list the stores

great_expectations store list

1. See an error similar to the one below

➜  great_expectations store list                                                                                                                                                                              [130 01:40:51P Wed 12/16/20]
Traceback (most recent call last):
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 84, in instantiate_class_from_config
    class_instance = class_(**config_with_defaults)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/database_store_backend.py", line 76, in __init__
    self.engine = self._build_engine(credentials=credentials, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/database_store_backend.py", line 164, in _build_engine
    engine = sa.create_engine(options, **create_engine_kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 164, in create
    raise TypeError(
TypeError: Invalid argument(s) 'manually_initialize_store_backend_id' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 84, in instantiate_class_from_config
    class_instance = class_(**config_with_defaults)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/expectations_store.py", line 133, in __init__
    super().__init__(
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/store.py", line 42, in __init__
    self._store_backend = instantiate_class_from_config(
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 86, in instantiate_class_from_config
    raise TypeError(
TypeError: Couldn't instantiate class : DatabaseStoreBackend with config :
    store_name      postgres_expectations_store
    credentials     {'host': 'localhost', 'port': 5432, 'username': 'great', 'password': 'expectations', 'database': 'ge'}
    manually_initialize_store_backend_id        91f04917-fe8e-49b7-bea1-db671ebfeb73
    table_name      ge_expectations_store
    key_columns     ['expectation_suite_name']

Invalid argument(s) 'manually_initialize_store_backend_id' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryan/.virtualenvs/ge-13/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/cli/cli.py", line 72, in main
    cli()
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/cli/store.py", line 23, in store_list
    context = toolkit.load_data_context_with_error_handling(directory)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/cli/toolkit.py", line 404, in load_data_context_with_error_handling
    context = DataContext(directory)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 3106, in __init__
    super().__init__(project_config, context_root_directory, runtime_environment)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 261, in usage_statistics_wrapped_method
    result = func(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 267, in __init__
    self._init_stores(self._project_config_with_variables_substituted.stores)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 354, in _init_stores
    self._build_store_from_config(store_name, store_config)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 324, in _build_store_from_config
    new_store = instantiate_class_from_config(
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 86, in instantiate_class_from_config
    raise TypeError(
TypeError: Couldn't instantiate class : ExpectationsStore with config :
    store_name      postgres_expectations_store
    store_backend       {'class_name': 'DatabaseStoreBackend', 'credentials': {'drivername': 'postgresql', 'host': 'localhost', 'port': 5432, 'username': 'great', 'password': 'expectations', 'database': 'ge'}, 'manually_initialize_store_backend_id': '91f04917-fe8e-49b7-bea1-db671ebfeb73', 'table_name': 'ge_expectations_store', 'key_columns': ['expectation_suite_name']}
    runtime_environment     {'root_directory': '/Users/ryan/local/scratch/great-expectations/ge13/great_expectations'}

Couldn't instantiate class : DatabaseStoreBackend with config :
    store_name      postgres_expectations_store
    credentials     {'host': 'localhost', 'port': 5432, 'username': 'great', 'password': 'expectations', 'database': 'ge'}
    manually_initialize_store_backend_id        91f04917-fe8e-49b7-bea1-db671ebfeb73
    table_name      ge_expectations_store
    key_columns     ['expectation_suite_name']

Invalid argument(s) 'manually_initialize_store_backend_id' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

Expected behavior
The config should be properly processed and will print out something like the output below

➜  great_expectations store list                                                                                                                                                                   [1 01:43:30P Wed 12/16/20]
1 Store found:

 - name: postgres_expectations_store
   class_name: ExpectationsStore
   store_backend:
     class_name: DatabaseStoreBackend
     credentials:
       database: ge
       drivername: postgresql
       host: localhost
       password: ******
       port: 5432
       username: great

Environment (please complete the following information):

• OS: macOS v10.15.4
• GE Version: v0.13.3

Additional context

• This configuration works just fine with v0.12.7
• A nearly identical configuration seems to work for _validation_ stores without issue (on v0.13.3) -- this issue seems specific to the ExpectationsStore class
• From the error message, it seems like we're propagating an unexpected parameter into the sqlalchemy engine. Not sure how to fix it, though! 😅

[eugmandel]

@ryanaustincarlson Thank you posting this issue. I will reproduce it and we will take care of it.

[ryanaustincarlson]

@eugmandel thanks! let me know if I can be helpful - if you have trouble reproducing, I'm happy to help debug what might be weird about my setup!

[eugmandel]

@ryanaustincarlson Reproduced - huge thanks for such a detailed a precise report! I think I know what happened there - we will work on a fix.

[anthonyburdi]

Hi @ryanaustincarlson thanks again for the clear and detailed issue report! The linked PR fixes the issue you addressed and adds a test case for it. If you would like to try it out to make sure it works in your environment, it has been merged into our develop branch and will be a part of the next patch release.