Marshmallow: _deserialize() got an unexpected keyword argument 'partial'

[palango]

I recently updated to 3.0.0rc8 (see https://github.com/raiden-network/raiden/pull/4334), and now run into a stacktrace in one test that I cannot make sense of.

i'm posting this here in the hope that someone can give me a hint on how to proceed in finding the problem.

The stacktrace is the following:

raiden/tests/integration/long_running/test_settlement.py::test_batch_unlock[matrix-False-2] Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/Users/paul/Projects/brainbot/raiden/raiden/tests/integration/long_running/test_settlement.py", line 344, in run_test_batch_unlock
    state_change_identifier=HIGH_STATECHANGE_ULID,
  File "/Users/paul/Projects/brainbot/raiden/raiden/storage/restore.py", line 42, in channel_state_until_state_change
    state_change_identifier=state_change_identifier,
  File "/Users/paul/Projects/brainbot/raiden/raiden/storage/wal.py", line 25, in restore_to_state_change
    state_change_identifier=state_change_identifier
  File "/Users/paul/Projects/brainbot/raiden/raiden/storage/sqlite.py", line 788, in get_snapshot_before_state_change
    row.identifier, row.state_change_identifier, self.serializer.deserialize(row.data)
  File "/Users/paul/Projects/brainbot/raiden/raiden/storage/serialization/serializer.py", line 68, in deserialize
    data = DictSerializer.deserialize(json.loads(data))
  File "/Users/paul/Projects/brainbot/raiden/raiden/storage/serialization/serializer.py", line 56, in deserialize
    return schema.load(deepcopy(data))
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 732, in load
    data, many=many, partial=partial, unknown=unknown, postprocess=True
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 851, in _do_load
    index_errors=self.opts.index_errors,
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 686, in _deserialize
    index=index,
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 438, in _call_and_store
    value = getter_func(data)
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 679, in <lambda>
    val, field_name, data, **d_kwargs
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/fields.py", line 300, in deserialize
    output = self._deserialize(value, attr, data, **kwargs)
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/fields.py", line 1373, in _deserialize
    deser_val = self.value_field.deserialize(val, **kwargs)
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/fields.py", line 300, in deserialize
    output = self._deserialize(value, attr, data, **kwargs)
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/fields.py", line 609, in _deserialize
    result.append(self.inner.deserialize(each, **kwargs))
  File "/Users/paul/Projects/brainbot/raiden/venv/lib/python3.7/site-packages/marshmallow/fields.py", line 300, in deserialize
    output = self._deserialize(value, attr, data, **kwargs)
TypeError: _deserialize() got an unexpected keyword argument 'partial'
2019-07-05T13:47:51Z <Greenlet "Greenlet-6" at 0x1096ef6a8: run_test_batch_unlock(raiden_network=[<App 0xe3B942e7821A0970EA79185A738BfBd3BC382AAb>,, token_addresses=[b"/\xde\x9d\xc4\xdedR\x87\xe5\xce\xeb:\x88\xe7a\x, secret_registry_address=b'\x96\xd1>\x04\x07tI\x9bF~1\xc7H\x94\xf8\xe2Ql\x9, blockchain_type='geth', deposit=200)> failed with TypeError

The value at that point is

{'message_identifier': '707648407656946282', 'recipient': '0x87A749D9b9c0c91AC009AeeBd74313D1a736A24C', 'canonical_identifier': {'token_network_address': '0x31F486f03875aA82781F3d1207294Cf7F4B0399F', 'chain_identifier': '337', 'channel_identifier': '1'}, 'transfer': {'target': '0x87A749D9b9c0c91AC009AeeBd74313D1a736A24C', 'token': '0x2fdE9dC4De645287e5CEeb3A88e7610ccE06db27', 'initiator': '0xe3B942e7821A0970EA79185A738BfBd3BC382AAb', 'balance_proof': {'locksroot': '0x31859db4b92d706b10dc04a7da091f56c59d0bd31271e8af86cd70c48f9f7f15', 'transferred_amount': 0, 'canonical_identifier': {'token_network_address': '0x31F486f03875aA82781F3d1207294Cf7F4B0399F', 'chain_identifier': '337', 'channel_identifier': '1'}, 'balance_hash': '0x099f54eb555c61ac9d218d789f1314d0f49ce18bf1d918c102abbd343a077324', 'nonce': '1', 'locked_amount': 10}, 'route_states': [{'route': ['0x87A749D9b9c0c91AC009AeeBd74313D1a736A24C'], 'forward_channel_id': '1'}], 'lock': {'amount': '10', 'secrethash': '0xca6730d3e5cb46a8dbed803cb3de6a3357a00ee27f0871c8919ff0eb39fff727', 'expiration': '90'}, 'payment_identifier': '1'}, '_type': 'raiden.transfer.mediated_transfer.events.SendLockedTransfer'}

and kwargs is {'partial': False}.

Some more background:
We have a deeply nested tree of state objects. These state objects are dataclasses, and we use marshmallow-dataclasses to automatically create schemas for them. There are also some Field definitions, they can be seen in the PR linked above.

I'd be thankful for for any hint or idea :)

[lafrech]

_deserialize should accept **kwargs. Which Field is it? I saw you added **kwargs to your custom fields in https://github.com/raiden-network/raiden/pull/4334/files.

[palango]

Which Field is it?

Is there an easy way to find out? I didn't yet figure out which field is the problem from the stacktrace.

But I just checked all custom fields again, and I added **kwargs to all of them.

[lafrech]

I suppose one way would be to cover the call to _deserialize in deserialize with a try/except to catch the TypeError and print the field...

[palango]

Thanks for the tip @lafrech , this turns out to be a problem in the https://github.com/Bachmann1234/marshmallow-polyfield project.

[lafrech]

Yep. It has just been reported: https://github.com/Bachmann1234/marshmallow-polyfield/issues/27.

[charlax]

Got bitten by this, should this be added to https://marshmallow.readthedocs.io/en/latest/upgrading.html?

[lafrech]

Yeah, possibly.

This is what was done for MA2: https://marshmallow.readthedocs.io/en/latest/upgrading.html?#custom-fields

Anyone wants to send a PR?

[charlax]

Sure.