Pylint: [Python 3.9] Value 'Optional' is unsubscriptable (unsubscriptable-object) (also Union)

[hjwp]

Steps to reproduce

Install Python 3.9, then:

# minimal_repro.py
from typing import Optional, Union
foo: Optional[int] = 3
bar: Union[int, str] = 4

Current behavior

Pylint reports

minimal_repro.py:3:5: E1136: Value 'Optional' is unsubscriptable (unsubscriptable-object)
minimal_repro.py:4:5: E1136: Value 'Union' is unsubscriptable (unsubscriptable-object)

Expected behavior

This isn't an error in Python 3.8 so I don't _think_ it should be an error in 3.9?

pylint --version output

pylint 2.6.0
astroid 2.4.2

(I tried pip install pylint astroid --pre -U)

[dkmiller]

Also encountering this with the Union type in Python 3.9: https://github.com/dkmiller/pyconfigurableml/pull/16/checks?check_run_id=1217453549

[hjwp]

have updated the issue to cover Union as well.

interestingly, it doesn't seem to be a problem for some other types that use the subscript syntax, eg List:

baz: List[int] = [1, 2, 3]   # this is fine.

[ReagentX]

This is definitely a false positive, the docs haven't changed: https://docs.python.org/3/library/typing.html#typing.Optional

[JulienPalard]

Self-contained reproducer, which is not Python 3.9 dependent (but the typing issue is 3.9 dependent):

class deco:
    def __init__(self, f):
        self.f = f

    def __getitem__(self, item):
        return item


@deco
def Optional():
    pass


print(Optional[int])

Probably related to https://github.com/PyCQA/pylint/issues/2578:

pylint doesn't currently correctly understand decorators

Looks like some code give up checks if the function is decorated, IIRC:

if isinstance(owner, astroid.FunctionDef) and owner.decorators:
    return False

it would be a simple quick-fix if we have no good way of studying the decorator from now.

As my reproducer is not 3.9 specific I'm digging about why the initial reproducer is failing on 3.9:

• The inferred function on line 1722 of typecheck.py is Optional in 3.9
• The inferred function on line 1722 of typecheck.py is Union in 3.8

Optional in 3.8 is declared as:

Optional = _SpecialForm('Optional', doc=...)

while in 3.9 it's declared as:

@_SpecialForm
def Optional(self, parameters):
    ...

As pylint does not resolve decorator, the inference is still OK in 3.8 (no decorator) but fails in 3.9 (decorated).

Looks like it could be fixed by using something like:

if inferred.decorators:
    first_decorator = helpers.safe_infer(inferred.decorators.nodes[0])
    if isinstance(first_decorator, astroid.ClassDef):
        inferred = first_decorator.instantiate_class()
    else:
        return  # TODO: Handle decorators that are not classes

in visit_subscript in typecheck.py.

[hippo91]

Thanks to all for your reports. I'm not sure the issue illustrated by @JulienPalard is the same as the original one.
We have to work on supporting python 3.9 but for now it is not yet fully supported.

[leycec]

Also note that PEP 585 (first supported by Python 3.9.0, of course) significantly expanded the set of subscriptable stdlib types to:

• tuple
• list
• dict
• set
• frozenset
• type
• collections.deque
• collections.defaultdict
• collections.OrderedDict
• collections.Counter
• collections.ChainMap
• collections.abc.Awaitable
• collections.abc.Coroutine
• collections.abc.AsyncIterable
• collections.abc.AsyncIterator
• collections.abc.AsyncGenerator
• collections.abc.Iterable
• collections.abc.Iterator
• collections.abc.Generator
• collections.abc.Reversible
• collections.abc.Container
• collections.abc.Collection
• collections.abc.Callable
• collections.abc.Set
• collections.abc.MutableSet
• collections.abc.Mapping
• collections.abc.MutableMapping
• collections.abc.Sequence
• collections.abc.MutableSequence
• collections.abc.ByteString
• collections.abc.MappingView
• collections.abc.KeysView
• collections.abc.ItemsView
• collections.abc.ValuesView
• contextlib.AbstractContextManager
• contextlib.AbstractAsyncContextManager
• re.Pattern
• re.Match

Additionally, note that PEP 560 (first supported by Python 3.7.0) formalized support for the __class_getitem__() class dunder method, which theoretically enables any class to permit subscription. Between these two PEPs, it's not necessarily clear that pylint can continue to emit reliable unsubscriptable-object warnings – not for types, anyway.

What is clear is that all of the external projects disabled pylint due entirely to this single issue. That's bad. The optimal solution might be for pylint to either drop unsubscriptable-object or assume all types to be safely subscriptable under Python >= 3.9.

[JulienPalard]

Tried https://github.com/PyCQA/pylint/pull/3890 on @leycec list:

test.py

import re
import collections
import contextlib

print(tuple[0])
print(list[0])
print(dict[0])
print(set[0])
print(frozenset[0])
print(type[0])
print(collections.deque[0])
print(collections.defaultdict[0])
print(collections.OrderedDict[0])
print(collections.Counter[0])
print(collections.ChainMap[0])
print(collections.abc.Awaitable[0])
print(collections.abc.Coroutine[0])
print(collections.abc.AsyncIterable[0])
print(collections.abc.AsyncIterator[0])
print(collections.abc.AsyncGenerator[0])
print(collections.abc.Iterable[0])
print(collections.abc.Iterator[0])
print(collections.abc.Generator[0])
print(collections.abc.Reversible[0])
print(collections.abc.Container[0])
print(collections.abc.Collection[0])
print(collections.abc.Callable[0])
print(collections.abc.Set[0])
print(collections.abc.MutableSet[0])
print(collections.abc.Mapping[0])
print(collections.abc.MutableMapping[0])
print(collections.abc.Sequence[0])
print(collections.abc.MutableSequence[0])
print(collections.abc.ByteString[0])
print(collections.abc.MappingView[0])
print(collections.abc.KeysView[0])
print(collections.abc.ItemsView[0])
print(collections.abc.ValuesView[0])
print(contextlib.AbstractContextManager[0])
print(contextlib.AbstractAsyncContextManager[0])
print(re.Pattern[0])
print(re.Match[0])

It only gives two false positives to be inspected:

test.py:10:6: E1136: Value 'type' is unsubscriptable (unsubscriptable-object)
test.py:11:6: E1136: Value 'collections.deque' is unsubscriptable (unsubscriptable-object)

[JulienPalard]

type is made subscriptable by Objects/abstract.c (https://github.com/python/cpython/commit/48b069a003ba6c684a9ba78493fbbec5e89f10b8), to allow, for example, type[int]. I think we should fix this in another PR to now slow down https://github.com/PyCQA/pylint/pull/3890/.

[JulienPalard]

collections.deque is made subscriptable by implementing __class_getitem__, which should also be implemented in a distinct PR I think.

[NeilGirdhar]

Awesome job, Julien! Is there an issue I can follow to track progress on subscripting type and deque?

[JulienPalard]

@NeilGirdhar opened them ^

[GaetanLepage]

Hello !
Has the issue been solved ?

[JulienPalard]

Yes it's been fixed by https://github.com/PyCQA/pylint/pull/3890, but it has not been released yet as there's other things to fix for full 3.9 compatibility.

[JulienPalard]

The button to ask for more frequent releases is the pink one on top of this page :-)

[axzfkdpmrzrheue]

JulienPalard commented 9 hours ago
The button to ask for more frequent releases is the pink one on top of this page

@JulienPalard So now you want people to pay up just to release changes that have already been merged? That's extortion! Shame on you for saying that as an official member of https://github.com/python ; it's a major conflict of interest. I advise getting a job like the rest of us, so you don't have to beg.

How about people voluntarily sponsor for a job well done instead... Currently it's not done at all, so it doesn't deserve sponsorship.

Also, there is no way that I would ever personally sponsor any unethical organization such as PyCQA that bans people when it disagrees with them. Identify and clear the ban list first.

[JulienPalard]

Sad I have to clarify: I was not begging, I'm not part of PyCQA nor pylint team so I would get no money from donations to this project, this can't be extortion.

My message was a "wink of eye", my goal was to put a smile on everyones face.

(Anyway donations are hepful.)

[aleksihakli]

Just ran into this and popped in to say that it's easy to just run the QA matrix with Python 3.8 with typical QA tools installed anyway, refer to the commit above.

[hippo91]

@JulienPalard do not worry, i think that 99% of people understood you were joking.
@axzfkdpmrzrheue, just to close this sterile discussion, i'm a member of pylint's development team. For your information i'v got also a full time job and a family to take care of. The only income i get from my work for pylint is some thank you and sympathetic smileys. And it is cool as it is.
I, and all other member of the team, do not work on pylint for money. So next time, take it easy. The release will come soon. We do our best for that but our free time is limited.
The best way to accelerate the release process is to contribute. :smile:

[mcat56]

This is still an issue?

arg: Optional[int] = None

Optional: _SpecialForm
Value 'Optional' is unsubscriptablepylint(unsubscriptable-object)

[cdce8p]

@mcat56 This should be fixed and I unfortunately can't reproduce it. Can you give us some more details about your setup?

pylint --version

[metal3d]

I've got the same problem...

pipenv run pylint --version
pylint 2.6.0
astroid 2.4.2
Python 3.9.1 (default, Jan 20 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]

[Tenzer]

@metal3d Pylint 2.7.1 is the latest version and I believe the problem you see should have been fixed in version 2.7.0, so try to update Pylint.

[metal3d]

Something weird...

I'm using pipenv to install it, it sticks on 2.6.0.

I removed from pipenv to let use my system version, wich is the same and the problem disapeared:

pylint --version
pylint 2.6.0
astroid 2.4.2
Python 3.9.1 (default, Jan 20 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]

I suspect pipenv problem wich get my "system" package with bad environment mapping.

Creating a new environment:

pipenv run pylint --version
pylint 2.7.1
astroid 2.5
Python 3.9.1 (default, Jan 20 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]

So, I removed pylint from my pipenv environment (because the update seems to fail ), and reinstalled... the problem disappeared...

So... sorry for the inconvenience :)

[solesensei]

python -m pip install pylint -U

[metal3d]

python -m pip install pylint -U

I'm using pipenv. And the upgrade failed. I know how works pip 🙂