Msprime: Simulating from a starting tree sequence strips metadata codecs

[molpopgen]

This affects 0.7.4 in combo with tskit 0.3.2. I discovered this when I could no longer recover metadata after "recapitating" tree sequences.

Everything is pip-installed on Linux.

Minimal example (I think):

import msprime
import tskit

print(msprime.__version__)
print(msprime.tskit.__version__)
print(tskit.__version__)

MutationMetadata = tskit.metadata.MetadataSchema(
    {
        "codec": "struct",
        "type": "object",
        "name": "Mutation metadata",
        "properties": {
            "s": {"type": "number", "binaryFormat": "d"},
            "h": {"type": "number", "binaryFormat": "d"},
        },
        "additionalProperties": False,
    }
)

tc = tskit.TableCollection(1.0)

tc.mutations.metadata_schema = MutationMetadata
tc.populations.add_row()

for _ in range(10):
    tc.nodes.add_row(time=0.0, population=0, flags=tskit.NODE_IS_SAMPLE)

ts = tc.tree_sequence()

print("Schema ok:")
print(ts.tables.mutations.metadata_schema.__dict__)

ts2 = msprime.simulate(from_ts=ts)

print("Schema ok:")
print(ts.tables.mutations.metadata_schema.__dict__)

print("Schema gone:")
print(ts2.tables.mutations.metadata_schema.__dict__)

Output on my machine:

0.7.4
0.3.2
0.3.2
Schema ok:
{'_schema': OrderedDict([('additionalProperties', False), ('codec', 'struct'), ('name', 'Mutation metadata'), ('properties', OrderedDict([('h', OrderedDict([('binaryFormat', 'd'), ('type', 'number')])), ('s', OrderedDict([('binaryFormat', 'd'), ('type', 'number')]))])), ('required', ['s', 'h']), ('type', 'object')]), '_string': '{"additionalProperties":false,"codec":"struct","name":"Mutation metadata","properties":{"h":{"binaryFormat":"d","type":"number"},"s":{"binaryFormat":"d","type":"number"}},"required":["s","h"],"type":"object"}', '_validate_row': <bound method create.<locals>.Validator.validate of <jsonschema.validators.create.<locals>.Validator object at 0x7f4da07db580>>, 'encode_row': <function StructCodec.make_object_encode.<locals>.<lambda> at 0x7f4da07c49d0>, 'decode_row': <function StructCodec.__init__.<locals>.<lambda> at 0x7f4da07c4ee0>}
Schema ok:
{'_schema': OrderedDict([('additionalProperties', False), ('codec', 'struct'), ('name', 'Mutation metadata'), ('properties', OrderedDict([('h', OrderedDict([('binaryFormat', 'd'), ('type', 'number')])), ('s', OrderedDict([('binaryFormat', 'd'), ('type', 'number')]))])), ('required', ['s', 'h']), ('type', 'object')]), '_string': '{"additionalProperties":false,"codec":"struct","name":"Mutation metadata","properties":{"h":{"binaryFormat":"d","type":"number"},"s":{"binaryFormat":"d","type":"number"}},"required":["s","h"],"type":"object"}', '_validate_row': <bound method create.<locals>.Validator.validate of <jsonschema.validators.create.<locals>.Validator object at 0x7f4da07ed070>>, 'encode_row': <function StructCodec.make_object_encode.<locals>.<lambda> at 0x7f4da07df280>, 'decode_row': <function StructCodec.__init__.<locals>.<lambda> at 0x7f4da07df700>}
Schema gone:
{'_schema': None, '_string': '', '_validate_row': <function validate_bytes at 0x7f4da0c06550>, 'encode_row': <bound method NOOPCodec.encode of <tskit.metadata.NOOPCodec object at 0x7f4da07dbee0>>, 'decode_row': <bound method NOOPCodec.decode of <tskit.metadata.NOOPCodec object at 0x7f4da07dbe50>>}

[petrelharp]

Hm, good point! We should keep around the metadata schemas. I'm less sure whether we should keep the top-level metadata. I think "keep unless it is overwritten" is a good default, though? And, more convenient for pyslim?

[jeromekelleher]

The problem is 0.7.4 is using an older version of tskit that doesn't know anything about metadata schemas, and so the extra columns are being lost. We could ship another bugfix release before 1.0, where we backport on the changes required for a newer tskit C API version.

Is it worth it though? 1.0 will be along "soon".

[benjeffery]

Yep, we haven't had an msprime release since the metadata, so this isn't really a bug, more of a missing feature! One idea would be to do a 1.0 pre-release rather than a backport?

[molpopgen]

Hm, good point! We should keep around the metadata schemas. I'm less sure whether we should keep the top-level metadata. I think "keep unless it is overwritten" is a good default, though? And, more convenient for pyslim?

I hit this when writing an example for my student using fwdpy11. I recapitated and then couldn't get my selection coefficients.

I'm not using any top level metadata, largely as I haven't figured it out. Just provenance and table metadata, all of which need to make it through.

[molpopgen]

So when msprime is reporting using tskit 0.3.2, that doesn't mean the metadata from that version are used? I checked that table dumping and copying preserve the schema, leading me to conclude that msprime does something else here.

[jeromekelleher]

Yeah, this is confusing. The msprime Python API is using the reported version of tskit, but the underlying C code that it's actually compiled against can be different. There's two copies of the tskit C code involved here, and msprime is currently using quite an old version.

It probably wouldn't be that much effort to backport the new tskit C API stuff onto 0.7.4, it'll just be a case of updating the submodule repo and fixing a few C API breakages (I'd imagine).

[molpopgen]

Okay, that makes sense. Yeah, that is confusing!
I'm not too worried about making an 0.7.5 or whatever. A 1.x pre-release/alpha would do.

I'll update my example for my lab to precapitate rather than recapitate, thus dodging this entirely.

[jeromekelleher]

OK, will close this then. I'll try and get an alpha pre-release out ASAP - it should be mostly good enough, I think.