Opentrons: RFC: Versioning and Labware Def Specification

[IanLondon]

overview

This is my attempt to compile all our ideas about labware definition identification and versioning in one place. If we're all on board on this, the next step will be to break these things up into tickets, and after that we can figure out what part of the labware v2 project each ticket is required for.

GOAL: make labware access sane and reliable as we move to support custom labware across our stack. Settle on a set of requirements so we can ship the labware v2 project soon, without compromising our ability to build off of this later.

1. Definitively separate the namespaces between standard and custom labware stored on the robot.

2. Always reference labware by a compound id of loadName, version, and namespace. Drop otId.

Proposed UX when users upload labware to their robot:

If the namespace is "opentrons", they will be stopped and given an error -- users are not allowed to upload labware defs to the "opentrons" namespace.

Otherwise, for non-"opentrons" labware, if the loadName + version combo matches an existing labware, they will be asked to confirm they want to replace the def on their robot. Calibration data will be cleared if they replace a def.

A newly-uploaded labware def will have no calibration data. So it will always need to be calibrated. Calibration data will not be passed along to the next "version" even if the loadName matches an already-calibrated labware.

Changes

v2 labware schema changes

• Drop otId. Move loadName key to the top level of the defs.

• Add a namespace key to the top level. For all our shared-data/definition2 defs, this will be set to "opentrons". (Custom labware will use custom_beta until we ship the labware cloud. The "beta" is b/c labware in this namespace may have their calibrations wiped once the labware cloud is up.)

• Add a version key to the top level of the defs. This will be an integer.

(Also may be a good time to add schemaVersion: 2 so that we can reliably tell which schema a def is using, same as what we do for JSON protocol files #3450 )

shared-data changes

Make the updates outlined above to the labware v2 JSON schema, to all schema defs in definitions2/, and to labware v2 flow types.

Add data integrity unit tests to shared-data which ensure that within our standard defs (shared-data/definition2/), every loadName + version combo is unique and every namespace is "opentrons".

Calibration data

Store calibration data not by otId, but by loadName, version, & namespace together.

APIv2 changes

1. In APIv2, the labware load function will be changed to make the namespace and version required (for now, see note below)

# namespaces are separate
customplate = labware.load(‘generic_96_plate’, slot='1', namespace='standard', version=1)
defaultplate  =  labware.load(‘generic_96_plate’, slot='2', namespace='custom_beta', version=1)

~Seth outlined a feature where the version field can default to "latest" and robot server could push messages to Run App notifying users that the version has changed since their last run and allowing them to opt-in to a new calibration + def or otherwise instructing them to pin the version in their protocol and re-run it.~ Behavior fleshed out in #3474

1. The robot currently stores definitions in a single namespace. It will need to store them in a namespaced manner so the files don't overwrite each other.

Backwards compatibility with APIv1 labware.load

Currently, the APIv1 labware.load is shimmed to first read through its v1 definitions, then if nothing is found with the given name it will look through its v2 definitions.

When we add the version key to v2 labware defs, and allow the loadName to be common for different versions, we will need the shim to deal with this.

??? Perhaps the shim can always use version 1? Or the latest? And it could implicitly search all namespaces with precedence rules ("opentrons" > "custom" and others) to approximate the existing one-common-namespace behavior.

PD changes

Currently, PD can store custom labware defs in Redux, but uses otId and expects that to be unique. Instead, it will need to use its own internal uuid to key its definitions. This uuid will only be used in the scope of saving and reloading a protocol in PD application metadata, and will not be accessed outside of PD. When users reference an opentrons definition to add to the deck (click add labware, select a labware from labware modal), they'll add the latest one implicitly. When users load a protocol, PD will put the definitions that are in the protocol into its Redux set of definitions, so loading and saving will not change the protocol.

Since v2 labware defs now will include all 3 identifier keys (namespace, loadName, version), PD doesn't need to do anything to help robot server manage definitions.

Later, we can give users a notification when they load a protocol with outdated labware definitions, giving them the option to update the definitions to the latest version if they want.

Run App

For v1 labware using labware.load, no changes need to be made. Run App just gets v1 labware names from the server and uses its internally-bundled shared-data/definitions/ to get a def used to render the definition. It only uses the def for render. If it gets a name that doesn't exist in its baked-in set of defs, it shows the "Custom Labware" placeholder.

v2 labware is about to be implemented. We can follow that same pattern for now with the "Custom Labware" placeholder, and later consider having the API provide the def for rendering so that custom labware can be rendered. Taking the short route for now, it's possible that a protocol has custom labware def with a loadName that also exists in shared-data/definitions2. In that edge case, Run App would show the official labware, instead of the custom one.

NOTE: before we make the ticket, we will investigate if the long route is significantly longer. Maybe we can make an RPC change so v1 labware still returns the v1 name string, and v2 labware returns the whole def so that Run App never needs to look in shared-data/definitions2 itself, and will have no problem correctly rendering custom v2 labware.

Labware Creator

~For the short-term, we can hard-code "version": 1 for all custom labware. Later if we want we can easily make this a new field in Labware Creator and have the user set it.~

Since JSON protocols will use calibrations for a labware namespace + loadName + version, Labware Creator should bump the version of a labware def when definitions are loaded, modified, and saved. Changes to metadata fields alone don't need to bump the version.

This versioning is important to avoid a problem with mismatched calibrations. If a user calibrates loadName: myCoolLabware, version: 1 on their robot, then modifies the def in Labware Creator to fix the height and keeps it as version 1, then attaches the new def to a JSON protocol and runs it, we cannot comfortably detect a change because the namespace, loadName, and version is the same. So the old calibration would be used, possibly causing a crash.

(This isn't a problem when users explicitly upload labware defs to the robot because that action it will replace custom labware with the same loadName and version and clear the calibration data. If JSON protocols were to do the same, users would have to calibrate every single run, which is bad news bears. We want to avoid matching definitions by doing a deep compare or hash, because of floating point issues and other ugliness.)

Namespacing: we can hard-code the "custom_beta" namespace and not allow users to set their own in Labware Creator. Later when we move to make labware more shareable, we can add the facility of specifying namespaces, maybe tied to their user account at that point.

We can keep the proposed warning/error dissuading users from using an existing "standard" loadName for simpler UX. The changes outlined in this RFC would allow us to drop that warning if we later wanted to, but we can keep it in for as long as we want.

Looking forward: will namespaces, versions, and loadnames be long-lived?

The split between custom and standard namespaces is required for now because we don't have a central authority that can assign unique names and manage accessing labware by names. Versioning is required because users cannot opt-out of updates to standard labware when they update their robot server version.

However, namespaces will still be very useful when different users are creating their own resources and we're creating officially-sanctioned resources. Perhaps we'll and allow user accounts to publish definitions in their own unique namespace. Versioning will still continue to be very useful both to ensure reproducibility/sharability for protocols, and to allow opt-in "would you like to update def X for this protocol?" flexibility.

"Loadnames" also seem like a long-lived concept in Python protocols, because it's inconvenient and error-prone to copy and paste unreadable UUIDs into your protocol.

I expect that we'll continue to use loadnames, namespacing, and versioning as our approach to labware access evolves.

[sfoster1]

I don't think we should hardcode version 1 for custom labware; it seems better to

• Have a local cache in labware designer mapping load names to version names
• If a loadname is in the cache, save with (cached version + 1) - and add more complexity as needed for detecting whether a change was really significant to the definition, etc

The more I think about it the more I think the permissive lookup is necessary before apiv2 is released. Adding labware to a protocol is a core feature of the api that users will do multiple times a protocol, and adding complexity here both hurts the ux and encourages copy-pasting boilerplate.

[b-cooper]

1. +1 for adding the schemaVersion to the Labware V2 schema.

2. +1 for Seth's point of permissive lookups being an MVP feature. I would even include a permissive default parameter for the load function that defaults to the opentrons namespace.

3. There are a number of places to look for JSON geometry data or schema info in shared-data at the moment. It would be worth it to clarify where things are located with a standard structure.

I think this shift would be a perfect time to reorganize the directories in shared-data.

Right now we have:

/definitions, /definitions2, /robot-data & /labware-json-schema, /protocol-json-schema, /robot-data/schemas/*

I think we could benefit from organizing things by model. For instance:

• '/labware' shared-data pertaining to the labware data model
  
  
  
  • './schemas/' // all labware schemas, named by version
  
  
  • './fixtures' // all labware tests fixtures (nested by labware schema version)
    
    
    
    
    
    • './v1/'
    
    
    
    • './v2/'
    
    
    
    • '/definitions' // (nested by labware schema version)
    
    
    
    • './v1/'
    
    
    
    • './v2/'
    
    
    
  
  
• '/protocol' shared-data pertaining to the json protocol data model
  
  
  
  • './schemas/' // all protocol schemas, named by version
  
  
  • './fixtures' // all protocol tests fixtures (nested by schema version)
    
    
    
    
    
    • './v1/'
    
    
    
    • './v2/'
    
    
    
    • './v3/*'
    
    
    
  
  
• '/deck' shared-data pertaining to the deck data model
  
  
  
  • './schemas' // all deck schemas, named by version
  
  
  • './fixtures' // all deck tests fixtures (nested by schema version)
    
    
    
    
    
    • './v1/'
    
    
    
    • './definitions' // all deck defintions (nested by schema version)
    
    
    
    • './v1/'
    
    
    
  
  
• './module' // shared-data pertaining to the module data model
  
  
  
  • './schemas' // all module schemas, named by version
  
  
  • './fixtures' // all module tests fixtures (nested by schema version)
    
    
    
    
    
    • './v1/'
    
    
    
    • './definitions' // all module definitions [previously called specs] (nested by schema version)
    
    
    
    • './v1/'
    
    
    
  
  
• './pipette' // shared-data pertaining to the pipette data model
  
  
  
  • './schemas'
    
    
    
    
    
    • './name/' // all pipette name schemas, named by version
    
    
    
    • './model/' // all pipette model schemas, named by version
    
    
    
  
  
  • './fixtures' // all module tests fixtures (nested by schema version)
    
    
    
    
    
    • './name/' // all pipette name fixtures, named by version
      
      
      
      
      
      
      
      • './v1/'
      
      
      
      
    
    
    
    • './model/' // all pipette model fixtures, named by version
      
      
      
      
      
      
      
      • './v1/'
      
      
      
      
    
    
    
    • './definitions'
    
    
    
    • './name/' // all pipette name definitions, named by version
      
      
      
      
      
      
      
      • './v1/'
      
      
      
      
    
    
    
    • './model/' // all pipette model definitions, named by version
      
      
      
      
      
      
      
      • './v1/'
      
      
      
      
    
    
    
  
  

[umbhau]

Have not given this quite enough thought, but want to raise a quick question:

As I understand it, the main reason we would 'version' labware definitions is b/c the existing definition has an error (discrepancy from physical object), and we want to be able to correct it without forcing a breaking change on users who are already up and running with the existing definition.

Is it strictly necessary to add versioning to accomplish this (and therefore the set of user-facing changes to support versioning)? Or would it, e.g., be sufficient to know that the new labware is different and give users the choice to update?

[umbhau]

Second question: Is the namespace addition necessary if we enforce unique names across both standard and custom labware?

[IanLondon]

@umbhau

Is it strictly necessary to add versioning to accomplish [the ability for users to opt-out of updates to standard labware definitions] (and therefore the set of user-facing changes to support versioning)? Or would it, e.g., be sufficient to know that the new labware is different and give users the choice to update?

There are alternative mechanisms we could use, but they would be stateful and IMO introduce more complexities for the user. For example, if a user has a working protocol with good calibration for labware X, and they opt out of an update we shipped for labware X, they probably ought to have some UI in Run App to opt back into it later? When they opt out, their local robot database will diverge from the "official" definitions. If they need to factory-reset their robot for some reason, or buy a second robot, or exchange their robot for a new one, they will have a hard time getting their protocols to work like they did before.

I don't think that the versions would be a burden on the user. Most of the time, we should be able to ship an update, notify them that they need to re-calibrate because of a def was updated, and then continue on. If they were relying on a hack that needs the old buggy version, they can specify the version in their protocol and it will continue to work as it did before. But in a majority of cases they aren't relying on a hack on a buggy def, so they don't need to think about "version". They only need to recalibrate certain labware when Run App tells them an update happened.

Is the namespace addition necessary if we enforce unique names across both standard and custom labware?

Without namespaces, if a user makes biorad_96_plate and Opentrons later releases a standard labware def with the same name, then when the user updates their robot they will run into a conflict. Since we are using a standard convention for constructing a loadName, a collision is likely to happen at some time. This will mess up their protocol runs and might be hard to diagnose and fix.

We need namespaces because we can't stake out names in advance before we release them.

[pantslakz]

Heyas, all this sounds great and I believe versioning is something that will resonate well with users. Thank you for the very thorough and thoughtful explanation @IanLondon .

A couple thoughts/questions:

• Would we surface version number to users in the labware creator? I think this could be useful, especially when they are in the create/test/import/export process.
• In the future, I could imagine a scenario where a user may want to start with an Opentrons labware (as a template), and edit it to make a custom version. In this instance, could the loadname remain the same, but the namespace changes to "custom_beta", and version at 1?
• The term "directory" might be easier for users to understand over "namespace"

[IanLondon]

Would we surface version number to users in the labware creator? I think this could be useful, especially when they are in the create/test/import/export process.

We could if we wanted to. Why would a user want to use versioning for their custom defs? It might be easier to keep it at 1 for as long as local files on your computer are the way that users deal with labware (that is, until we do the cloud thing). Versions are more useful for opentrons standard labware, because users can't easily opt-out of updates we push without version. Maybe I'm missing a use case though?

In the future, I could imagine a scenario where a user may want to start with an Opentrons labware (as a template), and edit it to make a custom version. In this instance, could the loadname remain the same, but the namespace changes to "custom_beta", and version at 1?

That could work. They'd have to use the namespace arg in labware.load('load_name_here', slot='1', namespace='custom_beta') if they did that. But they could also change the loadName in Labware Creator, potentially.

The term "directory" might be easier for users to understand over "namespace"

Ah true, "namespace" isn't very meaningful for non-programmers! "Directory" is more intuitive, but since it's not literally tied to a folder in a file system, that might be confusing? Maybe "organization" or "scope" or "group" or something? I foresee us eventually having this "namespace" thing be tied to user accounts if/when we move to make labware defs sharable on a cloud platform, so "organization" seems like a forward-looking thing.

No users should have to think about this namespace/directory/organization word until we do a cloud thing, all they have to know is that some labware is official "opentrons" that comes in with software updates and the rest is custom labware that they upload, so hopefully this word won't surface to users if/until the cloud thing.