Marshmallow: skip_missing is gone, how to skip None elements ?

[touilleMan]

I saw that v2.0.0b removed the skip_missing option, claiming it skips the missing entries by default.
However in my case (using mongoengine's Document) a field is never missing: it returns None which is, strictly speaking, _something_...

I can see from the commit (https://github.com/marshmallow-code/marshmallow/commit/a3908d36967ac30764ceb66c52c99357425fbdec) that previous behavior of skipping empty values (like None or empty list/tuple) has been obliterated. Is there now a new way to do this ?

I'm thinking for now to overload the _serialize method of each field to check for the empty value and return the missing singleton in such a case... but this sound really cumbersome to me, there must be a better way !

[ghost]

Same problem with nested Schema representing lazy loaded SqlAlchemy entities during serialization.

[mcquay239]

I beleive this is a relevant test:

assert not Marshaller()({"v": None}, {"v": fields.Integer()})

[VladimirPal]

@sloria Tell us please how can we skip None elements?
I've created this PR https://github.com/marshmallow-code/marshmallow/pull/254 and it works for me, it will skip None elements, but it breaks a lot of tests, looks like i don't understand something

[VladimirPal]

I think we need to have skip_if argument:

    name = fields.Str(skip_if=None)

@sloria If it's ok i can create PR

[sloria]

I'm not sure that a user would want to configure skippable values at the individual field level; it might make more sense to configure at the field _class_ level.

Something like:

class SkippableString(fields.Str):
    skip_values = [None, '']

# Or, if you prefer
fields.Str.skip_values = [None, '']

Thoughts?

[VladimirPal]

What if i need all my fields with None value have been ignored in schema? I need to create some Meta class?
Here is my case:
In frontend i have a backbone model and it fetch data from backend, backend returns this None values and when i save model, i get errors - "Field may not be null", because i sent the data that i got from backend, i think it's a weird behaviour

[touilleMan]

I solve this trouble in my library (marshmallow intregration with mongoengine) by creating a mixin class which overload the _serialize method

class SkipEmptyClass(object):
    def __init__(self, *args, **kwargs):
        skip_empty = kwargs.pop('skip_empty', True)
        super(SkipEmptyClass, self).__init__(*args, **kwargs)
        self.skip_empty = skip_empty

    def _serialize(self, value, attr, obj):
        value = super(SkipEmptyClass, self)._serialize(value, attr, obj)
        if (self.skip_empty and
                (value is None or isinstance(value, (list, tuple)) and not value)):
            return missing
        return value

# Overload your marshmallow field class here
class String(SkipEmptyClass, fields.String):  # watch out inheritance order with MRO
    pass

see: https://github.com/touilleMan/marshmallow-mongoengine/blob/master/marshmallow_mongoengine/fields.py#L7

[VladimirPal]

@touilleMan Thanks, it will help me.
But i think it's really ultimate issue and we need to have more easy way.

[sloria]

Sorry for the delayed response on this.

How about using a post-dump method to remove any null values? Something like:

from marshmallow import Schema, fields, post_dump

class BaseSchema(Schema):
    SKIP_VALUES = set([None])

    @post_dump
    def remove_skip_values(self, data):
        return {
            key: value for key, value in data.items()
            if value not in self.SKIP_VALUES
        }


class MySchema(BaseSchema):
    foo = fields.Field()
    bar = fields.Field()


sch = MySchema()
sch.dump({'foo': 42, 'bar': None}).data  # {'foo': 42}

[Bachmann1234]

Howdy! Thanks @sloria for this. Though you will run into problems if you have dicts or anything not hashable. Luckily I just needed to filter Nones out so I could change it to

if value is not None

[makmanalp]

A related but tangential trouble is the default behavior for missing inputs - I'm not sure that missing fields do get skipped by default, specifically during dumps() I get AttributeError: "foo" is not a valid field for <obj> and such if a field is missing.

I think before this hits 2.0 final, it might be a decent idea to enumerate all the possibilities - serialization, deserialization, missing field, null field, field with type specified, field without type specified (in fields Meta). I volunteer myself for the task unless anyone objects.

Also, hi @Bachmann1234! :)

[Bachmann1234]

howdy!

[sloria]

@Bachmann1234

Though you will run into problems if you have dicts or anything not hashable.

True; in these cases, you can just make SKIP_VALUES a list.

@makmanalp

I get AttributeError: "foo" is not a valid field for and such if a field is missing.

Would you mind posting code that produces this error?

I think before this hits 2.0 final, it might be a decent idea to enumerate all the possibilities

Are you planning on updating the docs? Making code changes?

I volunteer myself for the task unless anyone objects.

Thank you!

[makmanalp]

@sloria now I'm failing to reproduce the issue with 2.0.0b4 and 5 with a minimal example - probably means this was an edge case. I'm going to see if I can find the problem I'd hit and narrow it down from there.

[makmanalp]

@sloria got it:

import marshmallow as ma


class Foo(ma.Schema):
    field = ma.fields.Integer()

    # Uncommenting the following line fixes the issue
    # generated_field = ma.fields.Integer()

    @ma.pre_dump
    def hook(s, data):
        data["generated_field"] = 7

    class Meta:
        # Removing generated_field from here drops it from the output
        fields = ("field", "generated_field")

s = Foo()
print(s.dumps({"field": 5}))

Error is:

Traceback (most recent call last):
  File "test.py", line 19, in <module>
    print(s.dumps({"field": 5}))
  File "/usr/local/lib/python2.7/site-packages/marshmallow/schema.py", line 549, in dumps
    deserialized, errors = self.dump(obj, many=many, update_fields=update_fields)
  File "/usr/local/lib/python2.7/site-packages/marshmallow/schema.py", line 514, in dump
    self._update_fields(obj, many=many)
  File "/usr/local/lib/python2.7/site-packages/marshmallow/schema.py", line 688, in _update_fields
    ret = self.__filter_fields(field_names, obj, many=many)
  File "/usr/local/lib/python2.7/site-packages/marshmallow/schema.py", line 747, in __filter_fields
    '"{0}" is not a valid field for {1}.'.format(key, obj))
KeyError: u'"generated_field" is not a valid field for {\'field\': 5}.'

On second thought perhaps this is behaving as intended? - maybe I can just do a post_dump instead. Although that'd lose the benefits of field validation.

More details about the use case - I was trying to refactor the internals of a system without breaking API contracts. I use marshmallow as a way to keep the contract stable. All data flows through schemas before going out. In the API and corresponding data models, there were 3-4 different schemas with fields like country_id, department_id, municipality_id but since all the other fields were matching, I could consolidate a ton of models into one single model with a location_id. Then at the marshmallow layer I retrofit everything to the old API contract by converting location_id back into country_id, department_id or whatever is required.

[sloria]

@makmanalp That appears to be a bug. A pre_load method should be able to modify the object before the implicit field creation happens.

Should be an easy fix; will work on it shortly.

[sloria]

@makmanalp Should be fixed now.

@touilleMan I'm going to hold off on adding back skip_missing. See https://github.com/marshmallow-code/marshmallow/issues/229#issuecomment-134387999 for a possible solution.

[makmanalp]

@sloria thanks!

[lafrech]

class BaseSchema(Schema):
    SKIP_VALUES = set([None])

    @post_dump
    def remove_skip_values(self, data):
        return {
            key: value for key, value in data.items()
            if value not in self.SKIP_VALUES
        }

This also removes None values when the field has allow_none=True. To avoid this, we'd need to iterate over the Schema fields. But then it's not that easy because the name of the fields may not match (due to attribute/dump_to/prefix...).

I didn't try it but I think it could be done clean and easy in Marshaller.serialize() by adding a skip_none parameter:

https://github.com/marshmallow-code/marshmallow/blob/dev/marshmallow/marshalling.py#

if value is missing or (skip_none and value is None and getattr(field_obj, 'allow_none', False)):
    continue

OK, reading https://github.com/marshmallow-code/marshmallow/commit/a3908d36967ac30764ceb66c52c99357425fbdec, I realize this looks like the skip_missing you removed...

When serializing, marshmallow ignores missing attributes (attributes that raise an AttributeError). I've been happy with this when using @touilleMan's umongo (MongoDB ODM) because when a value is missing, object.attribute raises AttributeError. Now, I'm trying to work with SQLAlchemy and missing fields are expressed as None so my API spits a lot of null values. I'm a total SQLAlchemy beginner so I may be missing something. But I had the same issue when dealing with simple objects. Looks like umongo's handy attribute management is the exception rather than the rule and the solution should be on Marshmallow's side.