Great_expectations: Suggestions for making great_expectations.yml more flexible

[ian-whitestone]

I am trying to use a portion of the great expectations functionality. I just want to be able to store and run expectations, along with the dataset configuration for the SQLAlchemy datasource. At this point, I don't need/want docs, any validation operators, plugins, etc..., but the great_expectations.yml requires pretty much everything to be specified: a plugins directory, validation_operators, data_docs_sites, evaluation_parameter_store, and the list goes on.

I believe the default should be to raise warnings instead of errors, and there should be an option for advanced users to suppress these warnings if they are intentionally not providing these parameters/configurations.

In terms of extensibility, it would also be really nice to be able to specify additional configuration/fields in great_expectations.yml, and then be able to access them through the context object. Right now an error is raised if you have additional configs. (for context, I would like to specify some batches in the config, and add a convenience "run_all" method which picks up all these batches and their associated expectations, then runs them).

Would love to hear your thoughts!

[Aylr]

@ian-whitestone These are solid points all around. For the first portion of your idea, would you imagine these as defaults built in (so you don't have to specify all of them), or would you prefer functionality to be disabled if portions are missing?

For the extensibility portion would you mind sketching out a yml chunk for your run_all example? I'm intrigued.

[ian-whitestone]

@ian-whitestone These are solid points all around. For the first portion of your idea, would you imagine these as defaults built-in (so you don't have to specify all of them), or would you prefer functionality to be disabled if portions are missing?

I always prefer explicit over implicit, cause the latter usually causes more confusion as more stuff is happening under the hood. In my view, if you don't have one of them specified, and you try and use functionality that relies on that config, it should raise an exception.

For example, if I didn't have any validations_store specified and then I ran a validation operator that used the StoreValidationResultAction action, it would raise an error saying no validations_store has been specified.

I think this will help make great_expectations more lightweight, and allows users to build on the necessary components as needed.

For the extensibility portion would you mind sketching out a yml chunk for your run_all example? I'm intrigued.

I've actually realized I can handle this with a custom validation operator since you can pass in whatever kwargs you want to the class.

Example great_expectations.yml:

validation_operators:
  run_all_operator:
    class_name: ProcessBatchesValidationOperator
    module_name: custom_validation_operator
    batches:
      - datasource_name: 'domi'
        generator_name: 'apartments-generator'
        asset_name: 'listings'
        expectations:
          - 'listings.warning'
    action_list:
    - name: store_validation_result
      action:
        class_name: StoreValidationResultAction
    - name: send_slack_notification_on_validation_result
      action:
        class_name: SlackNotificationAction
        slack_webhook: ${SLACK_ALERTS_WEBHOOK}
        notify_on: all
        renderer:
          module_name: great_expectations.render.renderer.slack_renderer
          class_name: SlackRenderer

And associated python code:

import great_expectations as get
context = ge.data_context.DataContext()
results = context.run_validation_operator('run_all_operator', assets_to_validate=None)

Rather than relying on users manually loading/specifying their batches and passing into assets_to_validate, the custom validation operator I made, ProcessBatchesValidationOperator, will loop through the batches listed in batches, produce the equivalent assets_to_validate list, then run the same functionality as your ActionListValidationOperator.

[Aylr]

@ian-whitestone I completely agree with everything you are saying about explicitness, good errors, and light-weight-ness.

Also, I love the custom validation operator. Is that something you'd be willing to open source? 🤞 😄

[Aylr]

It occurs to me that some of the functionality you want here already exists in a newer feature on the command line: the great_expectations validation-operator run command in non-interactive mode: https://docs.greatexpectations.io/en/latest/command_line.html#non-interactive-good-for-production

;tldr you can write a tiny JSON file that specifies batches and suites and you can specify that config file and the CLI will do your bidding

[github-actions[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?\n\nThis issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.