Pandas-profiling: Potential integration with Great Expectations?

[abegong]

Hi, @sbrugman!

I’ve been a fan and admirer of pandas-profiling for a long time! I was happy when you started maintaining the project last year—excited to have a good excuse to introduce myself.

I’m one of the core contributors to Great Expectations, an open source library for data testing and documentation. I don’t know if Great Expectations has been on your radar, but we have a very active community and are gaining adoption quickly. We currently provide some light data profiling as part of the package, and our community has been pushing us to do more.

We’re wondering if it makes sense to do tighter integration with pandas-profiling. Specifically, we’re interested in the possibility that pandas-profiling could generate test suites in the Great Expectations format, in addition to HTML and JSON. It would make both packages more valuable, and avoid re-inventing and maintaining the same profiling logic.

Originally, we didn’t think that such an integration made sense, because pandas-profiling rendered results directly into HTML. Reverse-engineering that HTML would be just as much work as (and far less stable than) profiling the data from scratch.

Then last week, a friend mentioned that pandas-profiling now has a to_json method, and you’ve started to add typing, etc. to pandas-profiling’s intermediate logic. This seems really promising.

Please let me know if you’re up for a call sometime in the next couple of weeks. If you’re interested in collaborating, we could build something really cool and useful together. Even if you’re not in a place where collaborating makes sense, I’d love to say hi and get to know each other.

You can reach me at [email protected]. Thanks!

• Abe

PS: Apologies for not using the normal feature template. This was really meant to be an email, but I couldn't find a working address for you.

[sbrugman]

Thanks for reaching out. I've sent you a message.
Let's keep this issue open so that we can share the outcome.

[tsanikgr]

This would be super cool! Keen to be kept in the loop!

[sbrugman]

We're actually working on this behind the scenes. More details coming.

[sbrugman]

Work in progress 🏗️ Currently working on using the visions typeset to automatically generate expectations (see visions' PyData talk for more info).

[sbrugman]

Details on the integration here!