Great_expectations: Optional dependencies

[ian-whitestone]

Great expectations currently comes with a lot of dependencies. For a simple production deployment (i.e. not using any of the dataset documentation features), you don't need altair, or all the other notebook/jupyter dependencies that get installed.

I would even argue that if you are just using SQLAlchemy datasets, you don't need pandas (unless you have expectations that involve pandas/numpy, like expect_column_kl_divergence_to_be_less_than).

Why is this a problem?

When incorporating great expectations into larger projects, it increases the chances of having dependency conflicts.

Maybe slightly less common, but in serverless deployments you need to minimize your deployment package size. numpy, pandas, scipy, notebook, altair, etc... are all very large packages which blow up your deployment package size and push you over the allowed limits.

[jcampbell]

Thank you @ian-whitestone : I agree this is a problem, and am open to a range of alternatives to address it.

Most directly: I'm currently building out plans for a refactor of expectations into independent classes that will operate as plugins, and so can control their dependencies more clearly. Once that is in place, it will be possible to be much more transparent about (and lean in) requesting dependencies.

I'm going to keep this issue open, and hope that we can track progress toward that here. I'm also open to smaller PRs along the way that adjust imports to improve modularity.

Lastly: there's a key issue here around messaging so that people aren't unexpectedly stuck partway through an operation. Open to strategy for how to address that risk.

[joeyfreund]

plans for a refactor of expectations into independent classes

@jcampbell, any updates on this?
If needed, I'm happy to help with the refactoring efforts.

[jcampbell]

@joeyfreund -- we've continued to have this as an objective but simply not gotten the time yet...so I think that having some help with the refactoring could be really useful. I'll send you a message in slack and see if we can set up a time to brainstorm moving forward!

[github-actions[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?\n\nThis issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.