Pants: Improve the performance of running Pytest with Pants

[stuhood]

[stuhood]

After some exploration of the current overheads in pytest running, @jsirois's efforts have focused on implementing a very promising --venv mode for PEX. Wrapping this ticket up will involve incorporating that feature into Pants, and using it for pytest.

[stuhood]

@jsirois : What are the next steps for this one?

[jsirois]

Using the --venv mode after upgrading to Pex 2.1.25. I'm doing the upgrade today and will start on either using --venv for Pants python tool PEXes or else on using --pex-repository for subset resolves.

[jsirois]

This is a much more vague issue than I'd like, but after the --venv improvement for pytest startup latency probably comes solving the fresh .pyc compile problem outlined in #11339. After that I run out of attractive ideas at the rule layer.

[stuhood]

Agreed: I'm happy to have this ticket scoped down to just using the --venv mode, and to schedule pyc compilation independently.

[g-cassie]

Just bumping on this. Any chance it will make it into 2.3.x?

[Eric-Arellano]

Hey Gordon!

Yes, that's the plan. John has been working on changes to Pex to run in --venv mode. Check out https://github.com/pantsbuild/pex/issues/1115#issuecomment-740867989 for John's benchmarking.

The Pex changes are landed, and John has been working to wire it up to Pants.

[Eric-Arellano]

FYI https://github.com/pantsbuild/pants/issues/11339 is the other major lever we'll have to reduce Pytest latency. Not as much work has gone into that one yet.

--

Then, the other major performance work we've been doing is https://github.com/pantsbuild/pants/issues/11105. That Pex work has also landed, and we now only need to wire it up. It's intended to work around the case where one change to constraints.txt results in every single test being invalidated; now, only the actually used deps will invalidate.

--

The better cache management tools have not been started, but are tracked by https://github.com/pantsbuild/pants/issues/11167 and on the public roadmap: https://groups.google.com/g/pants-devel/c/F8Saug3BrFw/m/ZtWDP4YmDwAJ

[jsirois]

I broke out the --venv optimization to #11531. Iff further optimizations make sense, those will get broken out too.

[Eric-Arellano]

This is now implemented! The main remaining improvement is to avoid .pyc recompiles: https://github.com/pantsbuild/pants/issues/11339. But let's track that in the dedicated issue.