Mypy: Restrictions on MYPYPATH

[orenbenkiki]

Given that PYTHONPATH contains .../pythonX.Y/site-packages
And a package that contains some private 3rd party stubs under .../pythonX.Y/site-packages/foo/some_stubs

When setting MYPYPATH to .../pythonX.Y/site-packages/foo/some_stubs

Then mypy gives the error .../pythonX.Y/site-packages is in the MYPYPATH. Please remove it..

This is wrong because:

1. MYPYPATH does not include ../pythonX.Y/site-packages. It does include .../pythonX.Y/site-packages/foo/some_stubs but that is a very different directory.

2. Due to this error mypy refuses to use the stubs listed in MYPYPATH in this case.

This probably isn't intentional? The relevant PEP explicitly allows for MYPYPATH and does not seem to mention any restriction on the location of stubs added via MYPYPATH.

It is possible to work around the problem by copying (or linking) the stubs to a directory which is not under ../pythonX.Y/site-packages, say into ~/.foo-stubs and adding this path to MYPYPATH.

[msullivan]

I think this is intentional.

Does it work to add a py.typed to the stubs package?

[orenbenkiki]

Hmmm, I'll try tomorrow.

What would be the motivation of forbidding this?

[ethanhs]

The semantics of MYPYPATH are entirely up to the type checker using it. In addition, we do not want to add anything in site packages to the MYPYPATH, as that can be problematic. In general it is better to add a py.typed file and not use MYPYPATH for installed packages.

[gvanrossum]

The reason for this check is that almost any time someone tries to include site-packages in $MYPYPATH it causes problems -- it just is almost never what you want. Hence the explicit check (easy to find in the mypy source by grepping for "Please remove it"). I don't think the use case of stubs distributed in a subdirectory of an installed package is important enough to make an exception, and your workaround sounds better -- though using a separate stubs package as described in PEP 561 would probably be better.

[orenbenkiki]

The semantics of MYPYPATH are entirely up to the type checker using it.

Makes sense.

In addition, we do not want to add anything in site packages to the MYPYPATH, as that can be problematic.

And:

The reason for this check is that almost any time someone tries to include site-packages in $MYPYPATH it causes problems -- it just is almost never what you want.

Fair enough. I would then suggest the error message be improved to "MYPYPATH contains the path: ... which is under the site packages path: ..." - this would make it clear what the intention is. I found the current error message to be confusing as it claimed MYPYPATH contained the site packages path itself, which it does not.

In general it is better to add a py.typed file and not use MYPYPATH for installed packages.

I don't see how adding py.typed helps here.

My scenario is that I have one package providing common utilities and a few other packages that use this utilities package. One of the services the utilities package offers is some very partial numpy & pandas stubs which are tailored to a specific problem domain. This is "good enough" until proper stubs are published (holding fingers).

your workaround sounds better -- though using a separate stubs package as described in PEP 561 would probably be better.

My workaround is to create a symbolic link from the sub-directory in the site packages to a local .my-stubs directory and adding that into MYPYPATH. This is automated by the utilities package. This works but I was uncomfortable about going against the "spirit of the law" here.

I do not want to take my very partial, very domain-specific stubs and claim the numpy-stubs and pandas-stubs package names. That would signal an intent to evolving the packages towards becoming the end-all-be-all stub packages for numpy and pandas and that isn't something I can commit to.

[woolinsilver-kainos]

I agree with @orenbenkiki .

The error message is not helpful, I had to Google for 15 mins to find this rationale.

[ethanhs]

@woolinsilver-kainos agreed the error message could be improved. Would you like to submit a PR to improve it?

[jraygauthier]

This is occurring to me and it quite bad as this was my generic workaround for #5701 when using nix/nixos (see other issue for details). Now, I'm stuck adding each problematic package to MYPYPATH to avoid the "error: Cannot find module named 'blah'".

[jraygauthier]

Would it be possible to expose a flag / configuration to allow this, at least until a proper solution is found for #5701 ?

[msullivan]

Does the symlink workaround work for you?

I'm not wholly opposed to adding a flag I guess.

[orenbenkiki]

The symbolic link works for me in my specific setup, but it is a pretty fragile solution and a bit of a hack.

Are you suggesting a command line flag that would be allowed to add an entry to the search path, even if this entry is under the site packages directory? That would solve the problem, but it seems if you are going that way, you might as well lift the restriction for normal entries in MYPYPATH in the 1st place...

All that aside, the error message is still too cryptic IMVO.

[jraygauthier]

@msullivan :

Does the symlink workaround work for you?

It might do. Will have to try. However, I do not like much the stateful approach as I will be left with some dangling / stable / potentially harmful symlink in my working directory.

Seems to me like the original approach (@orenbenkiki took) of adding individual packages from site-package (and not the whole thing) should be allowed by default without requiring hacks of the sort.

@orenbenkiki :

Are you suggesting a command line flag that would be allowed to add an entry to the search path, even if this entry is under the site packages directory?

This was my suggestion indeed.

but it seems if you are going that way, you might as well lift the restriction for normal entries in MYPYPATH in the 1st place...

I do not agree with this point as the point of the restriction is to warn the user about a potential error on his part. Now, the flag tells: I understand this, however, I know what I'm doing. You explicitly opt-in for the free pass.

[jraygauthier]

@orenbenkiki : By the way, not sure why you closed the ticket as it seems there was no proper resolution?

[orenbenkiki]

@jraygauthier : You are right. Re-opening.

There are actually two sub-issues:

• Just fixing the error message to not be confusing.

• Actually providing the functionality.

I'm not sure how explicitly adding to MYPYPATH a directory that exists inside a package in site-packages is a potential error. What is the scenario where this is an actual error?

Note in my case, it isn't the root directory of a package, it is a sub-directory within some package.

That said, if a command line flag is deemed as more explicit (this being Python and explicitness being a value), sure, that would solve the second sub-issue.

[sjosegarcia]

I am a little lost on whats the issue at hand, but I do not have a MYPYPATH defined anywhere and I get the same error.

[ethanhs]

@sjosegarcia where are you running mypy from? if it is in site packages then you will get this error

[sjosegarcia]

I am running it within my virtualenv @ethanhs .

[shackra]

same here, running it from virtualenvwrapper