Attrs: asdict() and the way back with “private” attributes

[sscherfke]

This is actually not unexpected, but I hoped it would work:

>>> import attr
>>> @attr.s
... class A:
...     x = attr.ib()
...     _y = attr.ib()
... 
>>> a = A(1, 2)
>>> attr.asdict(a)
{'_y': 2, 'x': 1}
>>> A(**_)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() got an unexpected keyword argument '_y'

Maybe a function attr.fromdict(cls, dct) would solve this? It would remove the leading _ from all dict keys and then create the instance.

Since you exlicitly talk about serialization to JSON in the docs, there should IMHO also be an easy way for deserialization.

[hynek]

So preamble: I _do_ not think, that attrs should solve every problem ever. Its main focus is to reduce common boilerplate.

That said, it seems like a attr.fromdict(A, a) may have it’s place for symmetry alone?

[sscherfke]

Yes. An alternative could be to add an additonal kwarg (like remove_leading__ ;-)) to asdict() so that its return value can be passed as kwargs to the class’ __init__().

Actually, I would favor the kwarg for asdict() over a fromdict() function.

[hynek]

I’m on the fence here a bit. If you serialize stuff to JSON, you usually would want to exclude private data (which you can with the correct filter; maybe I should add one in fact). And since the result is just a dict, you can rename the keys if you don't like them.

Another alternative would be to allow filter callables to return the dict name (to make it more general than just stripping _).

[sscherfke]

I have some “private“ attributes and manually added properties to make them read-only. That's why they should be included when I convert to a dict:

@attr.s
class A:
    x = attr.ib()
    _y = attr.ib()

    @property
    def y(self):
        return self._y

[glyph]

The main thing that _I_ want fromdict to do is to be able to re-structure-ify semistructured JSON data. For example, if I have (pardon the formatting, getting the hang of doing stuff like this with ipython):

import attr
from attr.validators import instance_of as an

@attr.s()
class B(object):
    z = attr.ib()
@attr.s()
class A(object):
    x = attr.ib(validator=an(B))
    y = attr.ib()

b = B("test")
a = A(b, 7)

a

A(x=B(z='test'), y=7)

attr.asdict(a)

{'x': {'z': 'test'}, 'y': 7}

asdict is correctly _de_-structuring B; my attribute definition has enough information in it to _re_-constitute it (it's an instance of a B, which is an attrs class), so I would like it to _re_-structure it on the way back in.

[hynek]

The problem is: how would you determine the type? You'd have to add some typing information into the dicts.

[glyph]

@hynek: fromdict is descending a tree. In the data, it has {'z': 'test'}. But, since it is looking at the 'x' key in the top-level dictionary, it's _also_ looking at A.x, which tells it that this attribute should be interpreted as a B, because of the instance_of validator.

[glyph]

It's not possible in the _general_ case of course, Python is Python and you don't know anything about anything for sure, but if you want deserialization it seems like instance_of validators everywhere is a small price to pay.

[hynek]

So you'd need list_of/tuple_of/set_of validators? #39

[glyph]

Yeah, I think so. Although this could be implemented with just instance_of for starters and then support added for each of those...

[hynek]

I have to admit that I'm highly unenthusiastic about adding his to attrs proper b/c it works only under very specific conditions. What if I want to use a different kind of validator? This seems too narrow for a generic library. I suppose the best way forward here is to implement a shallow version for attrs and build something recursive on top of it. Maybe even using th meta data fields that have been proposed.

[glyph]

The problem with doing this in an external / application-specific plugin is that I won't be able to deserialize straightforward value objects that came from other libraries without monkeypatching them.

[hynek]

I feel like this discussion shouldn’t be “how can we build a deserialization system with what we have that works only under very narrow conditions that limit your overall attrs usage” but: “how would a good system look like and what does attrs need to get there?” (one thing that comes to mind is to introduce a complimentary type attribute).

I’m sorry, I know this is party pooping but I really don’t want to merge half-baked solutions. So far attrs is remarkably regret-free and I would like to keep it that way. :)

[sscherfke]

I have meanwhile solved my problem in a similar way that you described in your recent blog post, except that I do not use the single dispatch (which would be a good idea): https://bitbucket.org/sscherfke/aiomas/src/tip/src/aiomas/codecs.py

Apart from the dict with the attributes I also store the type information so that I can later reconstruct it properly. It only work for exact types (not sub classes).

[glyph]

I feel like this discussion shouldn’t be “how can we build a deserialization system with what we have that works only under very narrow conditions that limit your overall attrs usage” but: “how would a good system look like and what does attrs need to get there?” (one thing that comes to mind is to introduce a complimentary type attribute).

I wasn't proposing the use of instance_of just because it happens to be there already, but rather because I felt that it was already a proper semantic expression of the necessary constraints to provide serialization and deserialization.

For the more obscure cases where instance_of (and the implied additional list_of; I think set_of is too fancy for JSON) won't work, the appropriate thing would not be type but rather from_dict_hook (or somesuch); a generic callable that takes some jelly^W "low-level JSON-style data structures" and returns some high-level Python-style data structures. For absolute completeness, we may need a callable to go the other direction as well, for when you have a field that is de-serialized into a non-attrs object. (Although why you would have nonattrs objects in your program is beyond my ability to comprehend, of course.)

I’m sorry, I know this is party pooping but I really don’t want to merge half-baked solutions. So far attrs is remarkably regret-free and I would like to keep it that way. :)

Not at all! I am not submitting any PRs yet because I also feel it is extremely important to keep attrs extremely compact and semantically coherent.

[glyph]

I have meanwhile solved my problem in a similar way that you described in your recent blog post, [...]

The problem with this approach is that it assumes there is only one possible representation for each possible type, _and it is valid_ to find any possible type in any possible location in the object graph. The reason I prefer an attrs-based – and in particular, a validator= based – approach to this problem is that it allows me to conveniently define a _schema_ ahead of time, so I am not accidentally serializing to JSON in an inappropriate way.

[sscherfke]

I know it has its limitations but it worked well enough to do the job. If @hynek comes up with something better, I’d happily adopt it. :)

[hynek]

I wasn't proposing the use of instance_of just because it happens to be there already, but rather because I felt that it was already a proper semantic expression of the necessary constraints to provide serialization and deserialization.

Sure but since we don’t have an official way to compose validators, this is a no-go to me.

Hence my thought of proposing an official type field that could be used for more than just validation. Else someone needs to make Cyli happy in #13 and traverse it.

For the more obscure cases where instance_of (and the implied additional list_of; I think set_of is too fancy for JSON)

Well, this is still fromdict, not fromjson. This JSON-centric thinking already bit me once in #69.
won't work, the appropriate thing would not be type but rather from_dict_hook (or somesuch); a generic callable that takes some jelly https://twistedmatrix.com/documents/16.3.2/api/twisted.spread.jelly.html^W "low-level JSON-style data structures" and returns some high-level Python-style data structures. For absolute completeness, we may need a callable to go the other direction as well, for when you have a field that is de-serialized into a non-attrs object. (Although why you would have nonattrs objects in your program is beyond my ability to comprehend, of course.)

The thought alone makes me sad!

[glyph]

Sure but since we don’t have an official way to compose validators, this is a no-go to me.

I don't get it. If we want to write some compositional validators, we could easily add some code to traverse them in fromdict. Right now we don't have any validators we would want to compose.

In the case of a compositional decorator, either (A) only one of them can affect serialization (instance_of for example), or (B) it's an error to try to serialize, if there's a conflict. This seems like something fine to consider when adding the first compositional validator.

[hynek]

Sure but since we don’t have an official way to compose validators, this is a no-go to me.

I don't get it.

What I’m talking about is if someone wants x to be an int and smaller than 24.

So you write your validator:

def fancy_validator(inst, attr, value):
    attr.validators.instance_of(int)(inst, attr, value)
    if value >= 24:
        raise ValueError("too big")

(untested :))

You have no chance to deduce the type of the attribute just because the user needed a stricter validator…

Hence my tendency to allow a special-cased type attribute.

[glyph]

Hence my tendency to allow a special-cased type attribute.

Implicitly, we'd have to expand the validator interface to do this. So your validator would look something like this:

@attr.s
class FancyValidator(object):
    _composed = attr.ib()
    def __call__(self, inst, attr, value):
        self._composed(inst, attr, value)
        if value >= 24:
            raise ValueError("too big")
    def get_deserializer(self):
        return self._composed.get_deserializer()
fancy_validator = FancyValidator(attr.validators.instance_of(int))

[glyph]

get_deserializer is a total straw man there, by the way; I am still not sure exactly what that interface should be. But it seemed clear - obvious almost? - that if we wanted composition we'd have to have some kind of expanded validator interface anyway that lets us access the structure underlying the different basic validators.

If this doesn't seem like a clear direction to you I'd be happy to think harder about it on #13 and post some more realistic examples.

[hynek]

I have come to the conclusion that this is too heavy for attrs proper; at least for now.

I would recommend that y’all experiment with ideas externally and if some design emerges nice and practical we can talk again about merging it into attrs.

I’ll rather miss a feature than add something I’m not happy with and so far nothing has been presented that would make me go “oh yeah”.

Maybe everyone interested in this topic may want to help out at https://github.com/Tinche/cattrs?

[hynek]

(I’m keeping it open so people don’t open new ones and to not lose the discussion)

[glyph]

I have come to the conclusion that this is too heavy for attrs proper; at least for now.

Absolutely :)

[glyph]

I would recommend that y’all experiment with ideas externally and if some design emerges nice and practical we can talk again about merging it into attrs.

OK. If I'm going to do this, the main thing _I_ want is a public API for extracting which validator is used, and a public API for extracting the type (callable?) from the validator. Is that possible already?

[hynek]

Validators should be an attribute on Attribute and you can get the type: https://github.com/hynek/attrs/blob/master/src/attr/validators.py#L12 if your skunkworks go somewhere, we can promote it to an official public API. It’s safe to use meanwhile. :)

[glyph]

Oh good, so it's exposed publicly as type(instance_of(object)) :)

[Tinche]

I've considered using this approach for cattrs (get the type from the instance_of validator) but I think using metadata is the right call for this.

The approach I'm using currently (our newest game, a month out from production) is (omitting @attr.s and import for brevity):

def ensure_cls(cls):
    """If the attribute is an instance of cls, pass, else try constructing."""
    def converter(val):
        if isinstance(val, cls):
            return val
        else:
            return cls(**val)
    return converter

class A:
    z = attr.ib()

class B:
    x = attr.ib(convert=ensure_cls(A))

However I don't think this is the way to go. I think the way to go is using an external converting component (think we all agree on this) and using Python 3 type annotations in the type metadata. Something I've been playing with:

class A:
    z = attr.ib(metadata={'type': int})  # or have a nice wrapper, 'typed(int)'

class B:
    x = typed(A)
    y = typed(List[Mapping[int, int]])

>>> cattrs.load({'x': {'z': 1}, 'y': [{'1': 1}]}, B)  # Note the dict has strings for keys, as is usual when loading from JSON
B(x=A(z=1), y=[{1: 1}])

I have the loading of primitives down (http://cattrs.readthedocs.io/en/latest/loading.html) but have not done any work on attrs classes until we land metadata.

[glyph]

Huh. Maybe I could backport cattrs to py2 to start playing around with it. typed() is kind of what I want.

[glyph]

@Tinche Would the B in your example there just be @attr.s or do I need @cattr.s or something there?

[Tinche]

@glyph Just attr.s. I've done work on loading primitives and generics but none on loading attrs classes (waiting for metadata), so that part bitrotted and doesn't work currently. :)

[hynek]

So there’s seems to be a whole ecosystem for serialization/deserialization and I feel like attrs putting something into core would do no good – maybe even hamper innovation so I’m closing this.

To make them more discoverable, I’ve added a link to the readme in f6ab6c3.

[shabbyrobe]

This is just an experience report, feel free to disregard it if there's no appetite to look at this further.

I ran into this issue with dataclasses, which led me to look into attrs, where I find the situation is the same. While scouring the Python docs, I was somewhat astonished to find that there was an asdict but no inverse, and had to double-and-triple check to make sure I wasn't just missing something.

It feels to me that supporting one but not the other is the least desirable option, and that attrs (and dataclasses too for that matter) should provide neither or both.

[hynek]

The problem is that deserialization is _a lot_ more complex than serialization; even though it's not apparent on first sight. There's a reason why there's multiple packages that do that: there's a lot of complexities that can be solved in multiple ways. We just don't have the bandwidth to support one of them.

[shabbyrobe]

Thanks for the reply @hynek, that makes sense. I think the right thing then would be to remove asdict(), but I completely understand if there's no desire to do that for practical reasons.