Marshmallow: Turn off automatic skipping of unknown fields

[ryanlowe0]

When loading/deserializing, unknown field names are skipped. It would be nice to be able to raise an error instead, particularly when calling .validate(). Currently this doesnt seem possible, even with a custom validator, because the field filtering appears to already have occurred by the time the validator gets a hold of it, e.g.

class MySchema:
    foo = field.String()

@MySchema.validator
def check_unknown_fields(schema, data):
    for k in data:
        if k not in schema.fields:
            raise ValidationError("Unknown field name: '{}'".format(k))

errors = MySchema().validate({'bar': 'abc'})

doesnt work because data will be an empty {} by the time check_unknown_fields() is called.

[sloria]

@ryanlowe0 Thanks for the suggestion.

One simple change would be to pass the raw input dict to schema-level validators

@MySchema.validator
def check_unknown_fields(schema, data, raw_data):
    for k in raw_data:
        if k not in schema.fields:
            raise ValidationError("Unknown field name: '{}'".format(k))

If this were implemented, would this meet your use case?

[ryanlowe0]

Yes indeed, that is pretty much just what I was thinking!

[sloria]

Great. Then I think we can move forward with this. Pull requests welcome!

[ryanlowe0]

@sloria I started to implement adding the proposed raw_data argument, however I have a concern before putting it into a PR.

Sending a second raw_data arg rather than changing the behavior of data arg sounds great from a backwards compatibility standpoint, but only if it were optional. Unfortunately, this is proving tricky due to the way validators are collected and stored before being called. I thought maybe I could use inspect.getargspec to detect whether the additional arg was present, but apparently it doesnt work properly with partials in Python 2.6/7 (perhaps only in 3.4).

There may yet some way to get around this, but it may get ugly. An alternative idea would be to introduce some optional meta attribute or something that would cause raw data to be sent to validators _instead_ of filtered data.

In short, which of these sounds best?

1. Figure out some way to make raw_data be an optional third arg to user-defined validators
2. Keep the (schema, data) arg signature but add a meta-level flag to toggle between raw/filtered data
3. Always require the third raw_data arg
4. Stop filtering data altogether

Numbers 3 and 4 are simplest but break backwards compatibility..

[sloria]

@ryanlowe0 I like the idea of making raw_data optional (option 1) for backwards compat, at least for the time being; we may decide to make the argument required in a future release.

You could modify utils.get_func_args to handle partials. Here's one way:

#...
if isinstance(func, functools.partial):
    argspec = inspect.getargspec(func.func)
return argspec.args

It should also handle callable objects (i.e. inspect obj.__call__ when appropriate).

[philtay]

+1 for this feature.

If it's going to be implemented, we should also take in account dump_only fields. A dump_only field should be treated as "unknown" during deserialization.

[sloria]

Just a heads up: the way to pass raw data to a schema validator is changed in 2.0.0b3 (implicit passing has been removed).

See the relevant section in the docs: https://marshmallow.readthedocs.org/en/dev/extending.html#validating-original-input-data

[norbertpy]

Would have been great if setting strict=True would actually raise an error on unknown fields.

[lafrech]

strict might be misnamed but it is just about whether raising an Exception or not, not being more strict in the validation.

Whether to trigger a validation error on unknown fields is independent and is discussed in https://github.com/marshmallow-code/marshmallow/issues/524. You may comment there to advocate for an integration of the check_unknown_fields validation method into the core.

[sloria]

This is now possible in 3.0.0b12.

https://marshmallow.readthedocs.io/en/3.0/quickstart.html#handling-unknown-fields