Mypy: SelfType or another way to spell "type of self" (or, How to define a copy() function)

[gvanrossum]

A colleague and I were wondering how to define a copy() method in a base class so that when called on an instance of a subclass it is known that it returns an instance of that subclass. We found the following solution:

T = TypeVar('T')
class Copyable:
    def copy(self: T) -> T:
        return self.__class__()  # type: ignore
class X(Copyable):
    def foo(self):
        pass
x = X()
x.foo()
xx = x.copy()
xx.foo()  # OK

Note that the # type: ignore is required for two reasons: you can't call self.__class__(), _and_ it doesn't like the return type. Try e.g. return self and you get

x.py:5: error: Incompatible return value type: expected T`-1, got x.Copyable

It works in --py2 mode too:

    def copy(self):
        # type: (T) -> T
        return self.__class__()  # type: ignore

UPDATE: This doesn't actually work. :-( See next comment.

[JukkaL]

Hmm if this works it's not by design. I actually get errors when type checking the example:

t.py:11: error: Need type annotation for variable   (on line xx = x.copy())
t.py:12: error: None has no attribute "foo"

To make this work properly we could provide a new kind of type: a "self" type, which is basically the concrete type of self (in a derived class it would refer to the derived class, even if the definition is inherited). It's been used in a bunch of languages, though I'm not sure if any mainstream language has it. To make it work well we'd also need to support calling self.__class__.

We could have something like this:

from typing import SelfType

class Copyable:
    def copy(self) -> SelfType:
        return self.__class__()  # TODO still need to be able to call self.__class__
class X(Copyable):
    def foo(self):
        pass
x = X()
x.foo()
xx = x.copy()
xx.foo()  # OK

SelfType would probably only be usable in a covariant position such as in a return type. self and self.__class__(...) would have the type SelfType.

[gvanrossum]

Oh, I could have sworn that it worked but it was late. :-( So yes, I guess this is two feature requests: SelfType and calling self.__class__.

[gvanrossum]

(It seems to work when the class is generic, but who knows what other effects that has.)

[JukkaL]

This was closed accidentally?

[gvanrossum]

Could we make this work without new syntax? The following might have potential:

T = TypeVar('T', bound='Copyable')  # 'Copyable' is a forward ref
class Copyable:
    def copy(self: T) -> T: ...

Mypy would have to special-case the snot out of this, and I don't know if bound='ForwardRef' currently works, but it would avoid a new magic variable.

We could use something similar based on Type[T] (see https://github.com/python/typing/issues/107) for a factory class method:

# Same as above, then
    @classmethod
    def factory(cls: Type[T]) -> T: ...

[refi64]

Mypy would have to special-case the snot out of this

Eeeewww...

But, more seriously, _why_ exactly does this need to be special-cased? Other than the forward reference, this seems to generally be what I would think would be normal. Is it because the type passed to bound is the type using the type variable?

[gvanrossum]

Mostly because it's self. And because AFAIK it doesn't work now. :-)

[refi64]

Ah, ok. So the "special-casing" is allowing self to be a type variable with bound set to the current type?

[gvanrossum]

The bound may not even be necessary. The key behavior we'd want that's not implemented right now is that if you have a subclass:

class C(Copyable):
    pass
x = C().copy()
y = C.factory()

we want the type of x and y to be C.

[JukkaL]

This seems to me like it could work. This would save us from having to define a special "self type", and we wouldn't need to update the implementation of typing. A pretty neat idea.

A remaining open issue is enforcing a compatible __init__ signature in subclasses.

Here is a more detailed example of how this could work internally:

T = TypeVar('T', bound=Copyable)
class Copyable:
    def copy(self: T) -> T:
        t = self.__class__   # Type[T]
        return t()   # T -> good

We'd probably have to special case at least these things:

• Implement Type[T] (which is a prerequisite for this).
• Allow an annotation like this on self and enforce that the bound is compatible. Propagate the annotated type of self through the body of a method.
• Substitute a value for the type variable when looking up a method of an instance or class that uses the trick. C().copy should have type like () -> C instead of () -> T.
• Make self.__class__ produce the right type (probably don't need to special case self here). Maybe also support type(self)?
• Update overriding compatibility checks to know about this. Any override of copy() would also have to use a type variable for self.
• Disallow using the type variable in an argument type as they are contravariant, or at least disallow most uses (maybe some could actually be supported).
• It seems that using this T in a non-covariant position, such as in List[T], of the return type should be rejected.

In phase 2, figure out __init__ signature enforcement. We can leave this out in v1, but there is a risk that this will be tricky to retrofit. It would be nice to have a plausible design for phase 2 when we start working on the first phase.

[rwbarton]

As discussed in https://github.com/python/typing/issues/107, with Type[C] this could also make sense for class methods

class C:
    @classmethod
    def make_some(cls: Type[T], n: int) -> List[T]: ...

If d has type Type[D], and D is a subtype of C, then d.make_some(n) would have type List[D].

[gvanrossum]

Yes, but it would need a type variable.

[rwbarton]

Oops, yes! Corrected.

[drakedevel]

I recently ran into the factory use-case @gvanrossum suggested in a real application, trying to write something like this:

T = TypeVar('T', bound='Msg')

class Msg:
    ...
    @classmethod
    def parse(cls: Type[T], in_f: Any) -> T:
        params = {}
        for name, type_ in cls.fields:
            params[name] = in_f.read(type_)
        return cls(params)

class MsgA(Msg): ...

msg = MsgA.parse(...)

As expected, this doesn't work. Possible workarounds are to move parse to the top level, and call it with parse(MsgA, ...), or even just to replace @classmethod with @staticmethod and call it with Msg.parse(MsgA, ...). No changes to the type signatures or function bodies are needed. In my opinion that's a good argument for this syntax to work in the @classmethod context!

[gvanrossum]

Yeah, also see https://github.com/python/typing/issues/254 which comes to
the same conclusion -- it should work, but it's a bug in mypy.

[gvanrossum]

@elazarg Do you want to give this a try? We'll also need some new words for PEP 484 on the issue (but no new syntax, technically). The idea is as follows:

T = TypeVar('T')
class C:
    @classmethod
    def new(cls: Type[T]) -> T: <make one>
    def copy(self: T) -> T: <make a copy>
class D(C): pass
d = D.new().copy()  # has type D

There are probably some additional implied constraints on the type of cls and self here, they are still consrained by Type[C] and C even if T does not spell out that constraint.

[gvanrossum]

Oh, forgot that the PEP text is ready to go: https://github.com/python/peps/pull/89

Having a working implementation in mypy would help here.

[elazarg]

I will try.

[gvanrossum]

Thank you! It shouldn't be too hard, mostly you should probably just
disable setting the type of 'cls' or 'self' to something derived from the
current class when it's specified explicitly; the existing machinery should
take over from there.

[elazarg]

It also requires instantiating the typevars in the signature at member access, and checking for strictly-covariant overriding. We'll see what's more.

[elazarg]

It will be nice to have this declaration in the typing module:

Self = TypeVar('Self', covariant=True) 

Where the bound is documented to be that of the current class (lexically).

---

I have a feeling that the suggested syntax, although nice and seemingly intuitive, is somewhat misleading. It looks like it is standard type variable, but it must be _exactly_ the type of self, so it must be strictly covariant and cannot be used (bare) as the type of other parameters.

This leaves place for mistakes that can be avoided or have better diagnostics by avoiding the declaration of the type of self. It can also be parametrized:

from typing import Self, TypeVar

T = TypeVar('T')

class MyList(Generic(T)):
    @classmethod
    def new(cls) -> Self: <make one>
    def copy(self) -> Self: <make a copy>
    def twice(self) -> Tuple[Self, Self]: <make two copies>

    def copy_as_mapped(self, mapper: Callable[[T], Bla]) -> Self[Bla]: <make a mapped copy>
    def copy2(self, another: Self) -> Self: ... # Error: using bare Self in parameters is unsafe

It is also somewhat more greppable, although self: \w is OK.

[elazarg]

(It may also ease the implementation, but that's not really important)

[gvanrossum]

The problem with adding a new typevar (or anything) to typing.py is that
we'd have to wait for the next typing.py release, and it only makes sense
to release that ~concurrently with Python 3.5.x releases.

[gvanrossum]

We have this now.

PEP change: https://github.com/python/peps/commit/ada7d3566e26edf5381d1339b61e48a82c51c566

mypy changes: ba85545813885b4177605f2aff998aea671d79fd and several followups (e.g. 8a5f4634a72245b2ea63f4a2a68886c5a7629ae5, a90841f47d5d35282134f7ba7208648d8271bde0)

[kyprifog]

For anyone else that stumbles on this and is looking for the SelfType application, this helped me:

https://github.com/python/mypy/issues/3661