Great_expectations: Installing development dependencies is extremely slow when using new pip 20.3's new resolver

[verhey]

Describe the bug
The changes made to dependency resolution in pip 20.3 appears to have made installing great_expectations' development dependencies with pip install -r requirements-dev.txt glacially slow.

To Reproduce
Steps to reproduce the behavior:

1. Check out great_expectations locally (at the time of this bug's creation the most recent commit was 73ef686f59f34893c5c5a295482cdfa360c5b905)
2. Initialize a fresh virtualenv and activate it virtualenv venv -> source venv/bin/activate
3. Check your pip version, and update to anything beyond 20.3 if you're out of date. pip version -> pip install --upgrade pip
4. Attempt to install dev dependencies (and time its execution) with time pip install -r requirements-dev.txt
5. Wait a while, go to lunch, etc.

Expected behavior
Development dependencies install in a reasonable (few minutes or so) timeframe

Actual behavior

• pip install takes a long time and stdout is filled with lots of the following messages:

INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtrackingv
INFO: pip is looking at multiple versions of {PACKAGE_NAME} to determine which version is compatible with other requirements. This could take a while.

• pip downloads some dependencies multiple times and then appears to select one after some time

INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
  Downloading scipy-1.3.2-cp38-cp38-macosx_10_9_x86_64.whl (28.1 MB)
     |████████████████████████████████| 28.1 MB 1.0 MB/s 
  Downloading scipy-1.3.1.tar.gz (23.6 MB)
     |████████████████████████████████| 23.6 MB 85 kB/s 
  Downloading scipy-1.2.1.tar.gz (23.1 MB)
     |████████████████████████████████| 23.1 MB 3.7 MB/s 
  Downloading scipy-1.2.0.tar.gz (23.3 MB)
     |████████████████████████████████| 23.3 MB 2.8 MB/s 
  Downloading scipy-1.1.0.tar.gz (15.6 MB)
     |████████████████████████████████| 15.6 MB 2.2 MB/s 
  Downloading scipy-1.0.1.tar.gz (15.5 MB)
     |████████████████████████████████| 15.5 MB 1.7 MB/s 
  Downloading scipy-1.0.0.tar.gz (15.2 MB)
     |████████████████████████████████| 15.2 MB 3.5 MB/s 
  Downloading scipy-0.19.1.tar.gz (14.1 MB)
     |████████████████████████████████| 14.1 MB 2.2 MB/s 
  Downloading scipy-0.19.0.zip (15.3 MB)
     |████████████████████████████████| 15.3 MB 2.9 MB/s 

Environment (please complete the following information):

• OS: OSX 10.15.7
• Python: 3.8.2
• pip: 20.3.3
• GE Version: develop at commit 73ef686f59f34893c5c5a295482cdfa360c5b905

Additional context:

• Good news, when using the new resolver there doesn't appear to be any warnings about duplicate black versions as reported in #2028. Using the --use-deprecated=legacy-resolver flag or older pip version will throw the black error unless it's removed from one of the requirements files
• time pip install -i https://pypi.org/simple --use-deprecated=legacy-resolver -r requirements-dev.txt took 115 seconds locally, albeit with some cached packaged and some errors.
• Slack discussion

Suggested workarounds:

• Use an older pip version or --use-deprecated=legacy-resolver until this is resolved on pip's end. GE isn't the only project to have this problem with the new resolver it appears

[colmsnowplow]

Experienced the same issue, with it taking several hours.

Thanks for the workaround - I'll just add - in order to use --use-deprecated=legacy-resolver to workaround this issue, I followed these steps:

pip install black==19.10b0

Comment out black==19.10b0 in requirements-dev-test.txt

pip install -r requirements-dev --use-deprecated=legacy-resolver

YMMV.

[eugmandel]

I believe there is no fix that we should apply.

[verhey]

@eugmandel The only fix I could think of would be a docs PR to the setting up your dev environment docs, since they don't currently mention the issue. If you'd like, I'd be happy to make that PR, otherwise I'm good to close this.

Original intent was to make sure the team was aware of the problem - what you'd have to do to fix it is likely pretty extreme when the old resolver works fine. I'd guess the modifications necessary would end up as a breaking change for at least _someone's_ project.

One thing to note is pip originally planned to remove the --use-deprecated=legacy-resolver flag in January 2021 with pip 21.0. I'm not sure if they're still going forward with that, but it may be worth keeping in the rear view mirror.

You can use the deprecated (old) resolver, using the flag --use-deprecated=legacy-resolver, until we remove it in the pip 21.0 release in January 2021.

[spbail]

Hi all! This should be solved by #2256 which introduced a constraints-dev.txt (see updated set up instructions: https://docs.greatexpectations.io/en/latest/contributing/setting_up_your_dev_environment.html#contributing-setting-up-your-dev-environment)

Feel free to comment/reopen if that didn't solve it.