Msprime: Add metadata section to provenance

[jeromekelleher]

The metadata section of the provenance provides a space to define the meaning of metadata for nodes, individuals etc.

We define a space to describe the user-bits of the flags columns in nodes and individuals. Formally, the top bits of the flags field is considered metadata, since tskit doesn't interpret them.

[bhaller]

Here's what I have just implemented in SLiM:

{
    "environment": {
        "os": {
            "machine": "x86_64",
            "node": "d93-172.uoregon.edu",
            "release": "17.6.0",
            "system": "Darwin",
            "version": "Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64"
        }
    },
    "metadata": {
        "individuals": {
            "flags": {
                "16": "SLIM_TSK_INDIVIDUAL_ALIVE: the individual was alive at the time the file was written",
                "17": "SLIM_TSK_INDIVIDUAL_REMEMBERED: the individual was requested by the user to be remembered",
                "18": "SLIM_TSK_INDIVIDUAL_FIRST_GEN: the individual was in the first generation of a new population"
            }
        }
    },
    "parameters": {
        "command": [],
        "model": "initialize() {\n\tinitializeTreeSeq();\n\tinitializeMutationRate(1e-7);\n\tinitializeMutationType(\"m1\", 0.5, \"f\", 0.0);\n\tinitializeGenomicElementType(\"g1\", m1, 1.0);\n\tinitializeGenomicElement(g1, 0, 99999);\n\tinitializeRecombinationRate(1e-8);\n}\n1 {\n\tsim.addSubpop(\"p1\", 500);\n}\n2000 late() { sim.treeSeqOutput(\"~/Desktop/junk.trees\"); }\n",
        "model_type": "WF",
        "seed": 1781039230571
    },
    "schema_version": "1.0.0",
    "slim": {
        "file_version": "0.2",
        "generation": 2000,
        "remembered_node_count": 0
    },
    "software": {
        "name": "SLiM",
        "version": "3.0"
    }
}

We are only going to output the "flags" portion in this release of SLiM, I think, allowing the design of the remainder of the metadata section to be determined later. For flags, we're using integers 16-31 as proposed; they are strings, not integers, in JSON because they are keys in a dictionary, as I understand it. For the values, we're putting a symbolic name followed by a colon-space and then a definition, but I don't think that format ought to be legislated in the schema, as this is really just documentation. @petrelharp @jeromekelleher any revisions?

[jeromekelleher]

Looks good @bhaller. It might be useful to have a little structure in there for the flags though, maybe:

"metadata": {
        "individuals": {
            "flags": {
                "16": {
                     "name": "INDIVIDUAL_ALIVE",
                     "description": "The individual was alive at the time the file was written",
                 },
                "17": {
                       "name": "INDIVIDUAL_REMEMBERED",
                       "description": "The individual was requested by the user to be remembered",
                 }
                 // etc
             }
        }
    },

I agree it's just documentation, but having this minimal structure would allow the flags to be displayed in more nicely (I'm imagining opening a 'properties' menu about a tree sequence here in a GUI. A nicely formatted description of the flags would make it easier for users for parse).

I think name should probably be mandatory and description optional?

[bhaller]

Yep, I like that, done:

{
    "environment": {
        "os": {
            "machine": "x86_64",
            "node": "d93-172.uoregon.edu",
            "release": "17.6.0",
            "system": "Darwin",
            "version": "Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64"
        }
    },
    "metadata": {
        "individuals": {
            "flags": {
                "16": {
                    "description": "the individual was alive at the time the file was written",
                    "name": "SLIM_TSK_INDIVIDUAL_ALIVE"
                },
                "17": {
                    "description": "the individual was requested by the user to be remembered",
                    "name": "SLIM_TSK_INDIVIDUAL_REMEMBERED"
                },
                "18": {
                    "description": "the individual was in the first generation of a new population",
                    "name": "SLIM_TSK_INDIVIDUAL_FIRST_GEN"
                }
            }
        }
    },
    "parameters": {
        "command": [],
        "model": "initialize() {\n\tinitializeTreeSeq();\n\tinitializeMutationRate(1e-7);\n\tinitializeMutationType(\"m1\", 0.5, \"f\", 0.0);\n\tinitializeGenomicElementType(\"g1\", m1, 1.0);\n\tinitializeGenomicElement(g1, 0, 99999);\n\tinitializeRecombinationRate(1e-8);\n}\n1 {\n\tsim.addSubpop(\"p1\", 500);\n}\n2000 late() { sim.treeSeqOutput(\"~/Desktop/junk.trees\"); }\n",
        "model_type": "WF",
        "seed": 1783301962445
    },
    "schema_version": "1.0.0",
    "slim": {
        "file_version": "0.2",
        "generation": 2000,
        "remembered_node_count": 0
    },
    "software": {
        "name": "SLiM",
        "version": "3.0"
    }
}

Thanks for the comment, definitely better to have the structure there. (BTW, I'm writing name first and then description, but the JSON library I'm using is reordering them alphabetically; apparently JSON is order-agnostic so it shouldn't matter anyway.)

[jeromekelleher]

Hmm, I'm having second thoughts about this one. I 100% agree that we should document the meaning of these flags, but I'm not sure that the provenance is the place to do it. One can make the argument that provenance is to describe how the tree sequence was produced not how it should be interpreted and so this information shouldn't be in here. This makes sense to me, but it's not really sufficient to convince me that this current (sensible and practical) proposal isn't the long term solution.

The clincher for me is that a tree sequence doesn't have one provenance record, it can have many. The idea is that each record in the provenance table records the operations on some existing tree sequences that produced the current tree sequence. Suppose we had the following:

script = """
initialize() {
 initializeTreeSeq();
 initializeMutationRate(0);
 initializeMutationType("m1", 0.5, "f", 0.0);
 initializeGenomicElementType("g1", m1, 1.0);
 initializeGenomicElement(g1, 0, 1e8-1);
 initializeRecombinationRate(1e-8);
}
1 {
sim.addSubpop("p1", 500);
}
5000 late() {
sim.treeSeqOutput(); // pyslim sees this and replaces it with a tempfile argument
}
"""
ts = pyslim.simulate(script) # It'd be super-awesome if we could do this...
ts = msprime.mutate(ts, 1e-8) # Add some mutations
ts = ts.simplify(samples=range(10)) # Take a subset
ts.dump("stuff.trees")

When we read back in stuff.trees, it has three provenance records. The first says it was produced by SLiM. The second says it took this tree sequence and ran msprime.mutate on it with some parameters, and the third says it took a subset of the topology for a particular set of samples. In principle, given this list of provenances, we should be able to produce precisely the same chain of operations in order to construct the final tree sequence. (This isn't actually true right now because I'm not recording enough of the parameters in msprime. You can see what the list of operations were to produce a tree sequence though.)

Anyway, the point here is, where should the information be recorded which is required to interpret the flags and metadata? In principle, any step along the chain could change the metadata/flags so we should look to the last record to get this information. But then, we'd end up duplicating this schema information in each provenance record, which seems wasteful and error prone (would processing steps always take care to copy the metadata section from the previous provenance record if they don't alter the metadata definitions?).

I think what we really want is a top-level field in the tree sequence that somehow encodes the metadata flags and schemas. Then, if a processing step doesn't change the metadata definitions, it just leaves these alone. Crucially, this doesn't take any specific action from library users, it just happens by default. The program producing the initial tree sequence (e.g. SLiM) documents how the flags and metadata should be interpreted and subsequent processing steps don't touch them unless they're explicitly doing funky things like transcoding metadata from packed binary to JSON (or something) or rewriting flags (suppose some program also used bit 16 for something and wanted to shuffle SLiMs flags bits along). This seems much more robust an elegant to me.

I haven't thought this through properly, but suppose for the sake of argument that the individual table had two properties, user_flags and metadata_schema. For user_flags we might have

{   
    "16": {
        "description": "the individual was alive at the time the file was written",
        "name": "SLIM_TSK_INDIVIDUAL_ALIVE"
    },  
    "17": {
        "description": "the individual was requested by the user to be remembered",
        "name": "SLIM_TSK_INDIVIDUAL_REMEMBERED"
    },  
    "18": {
        "description": "the individual was in the first generation of a new population",
        "name": "SLIM_TSK_INDIVIDUAL_FIRST_GEN"
} 

and for metadata_schema we might have

{
    "type": "struct",
    "format": "<ii",
    "fields": {
          "0": { 
                "name": "birth_time",
                "description": "The time this individual was born in generations"
           },
           "1": {
                "name": "another property",
                "description": "I can't think of something else you'd store, but you see where I'm going"
           }
    }
}

This is just an example where we're using the syntax from Python's struct module to encode two packed little-endian integers and their descriptions. The cool thing about this is that we could use it to automatically decode the metadata. We might also support type="json" and have a JSON schema in there to tell the user how to interpret the metadata.

Hmmm, I've gone a long-way off-topic here! OK, so my concrete proposal for now is that we don't include this metadata field in the provenance schema for version 0.6.1, and instead think about attaching user_flags and metadata_schema fields to the Node and Individual tables for 0.6.2 (this can be done in a backward compatible way). SLiM can keep the metadata section in the provenance if you prefer, since it's not going to do any harm.

[jeromekelleher]

OK, in the interest of shipping 0.6.1 I'm going to go ahead without the metadata field in the provenance schema. It's very easy to add it in 0.6.2 if we decide we want it, since it's a non-breaking change. I would never be a mandatory field, so I don't see any harm in having the initial schema without it.

[bhaller]

Good morning. OK, I can see your reasoning. It doesn't necessarily seem bad to have a record of the history of the metadata/flags that have been used by past stages of processing, and it might help to provide some clue regarding the present state of things. But it certainly isn't designed to provide an accurate snapshot of the present state of the file, which is the more useful functionality to provide, I agree. Doing that right would doubtless be rather tricky, in general, but your ideas toward that seem reasonable. So, I guess SLiM's provenance will stay the way it is for now, and we'll see where this goes. Make sense @petrelharp?

[jeromekelleher]

Sounds good to me @bhaller. In the meantime, v0.6.1 is shipped and available from pypi and conda.

[petrelharp]

Your proposal is sensible. And, it seems like we are recapitulating VCF headers.. =) But, I also think that the provenance scheme is totally Good Enough For Now.

Where should the information be recorded which is required to interpret the flags and metadata? In principle, any step along the chain could change the metadata/flags so we should look to the last record to get this information. But then, we'd end up duplicating this schema information in each provenance record, which seems wasteful and error prone (would processing steps always take care to copy the metadata section from the previous provenance record if they don't alter the metadata definitions?).

I think that at minimum, we just need any program that sets new flags to say so. But right now, both SLiM and pyslim are sticking the flag definitions in provenance, because they might set the flags 9if they aren't there already); and it at least says what they are assuming. So, another program can scan through provenances, checking for any conflicts. Your proposal would be a lot cleaner, but dealing with this now seems like premature standardization.

[petrelharp]

Also,

ts = pyslim.simulate(script) # It'd be super-awesome if we could do this...
yes! Should be easy, at least on unix-ey platforms.

[bhaller]

Sure, that'd be a nice addition. Should take optional command-line args too, and a way to collect the stdout and stderr into strings. I reckon you could just throw the script into a file in /tmp and run it from there. And SLiM only runs on unix-ey platforms anyway, so you can safely assume that.

[jeromekelleher]

OK, going to close this issue so. We can pick it up again when thinking about metadata schemas and stuff.