Opentrons: RFC - Split API into multiple projects

[amitlissack]

Overview

The API project contains the http server. It also includes code required to run protocols, talk to smoothie board, read various robot configs, take pictures, and more.

All the code is packaged and distributed as a library as well as launched as an application.

As we move to FastAPI as a new framework and work to redesign some of our API this is a good time to do some refactoring.

Proposal for changes

Divide API into multiple projects:

1: Opentrons library providing interfaces to Robot.
2: API application which uses Opentrons library to talk to robot
3: Simulator application

Provide improved abstractions in Opentrons library

Make it easier to interact with the robot.

Create wrappers of all hardware resources

nmcli, camera, etc.

I believe this is akin to what @iansolano was thinking as Services

This seems to have been started in hardware_control/modules. Let's keep going and apply this to everything.

Create simulations and/or mocks of all hardware resources

nmcli, camera, etc.

Hide Feature Flags and ARCHITECTURE

API bootstrap or jupyter notebook code should call factory methods to create the appropriate object instances based on running environment.

For example:
Every http handler in server/endpoints/control.py checks a feature flag to determine what kind of object is held in request's com.opentrons.hardware value. This can be abstracted using a fleshed out HardwareAPILike interface.

Simplify configurations

Eliminate command line arguments from API and Hardware Control setup.

[sfoster1]

Looks good in general, but a few interesting things:

1. To me the hardware control library is very focused on the motion hardware and modules the robot uses to execute protocols. For things like the nmcli wrapper or the camera, I'd prefer to see them separated from hardware control and be part of either
   
   
   
   • the application server, or
   
   
   • the update server, with its role changed to a "system server" that handles all the non-protocol-interaction related things: updates, networking, cameras, sounds, etc
   
   
2. I'm not entirely sure that a separate simulator is the best way to go; I like having that tightly coupled with the actual hardware handling, and we have in the back of our head changes like pushing the simulator/real hardware distinction even lower by making a motor controller simulator that interacts with the rest of the system through gcode the same way. I think we should keep the "protocol execution and calibration" stack unified at least for now, comprising
   
   
   
   • the python protocol API
   
   
   • json protocol API execution
   
   
   • (if we ever separate it) labware and geometry handling and motion planning
   
   
   • the hardware_control module and friends
   
   
   • the legacy robot api
   
   
   • the hooks for saving calibration and executing calibration actions (these actually are pretty minimal - things like making sure we can control the robot outside the context of a protocol)
   
   
   • the command line based frontends for simulation and execution
   
   

[amitlissack]

Regarding your first point, I think that any code that would be imported and used by multiple applications, scripts or via a jupyter notebook should not be in update-server or api. I could see that updates and networking would not fall into this category, but lights, sound, camera, etc could be useful from some jupyter notebook.

👍 on point 2.

[sfoster1]

Thinking about it more, I agree with you about the multimedia support stuff. We should also probably consider integrating it into both the json and python protocol APIs.

[iansolano]

Sorry I'm a little late to the party. I'm in agreement with the general premise here, curious to see how we prioritize implementation (i.e. what parts to peel off first, mocks for HW might be nice since we _could_ start by improving testing).

Abstracting HardwareAPILike interfaces, simulations and/or mocks, and separating things like nmcli wrapper make a lot of sense to me.

With regards to wrappers of all hardware resources and services, this is somewhat similar in concept, the idea being isolate the business logic and provide a clean interface for engineers to interact with. Services objects in my particular wording applied to the the associated business logic around a particular resource (api endpoint) and abstracting the work around what and how to display json in api calls as well as how those api calls interact with our system (if that makes sense). Hopefully, in the nearish future I'll make PR example of this with the new fastapi implementation.

Excited about where this is heading 🎉. Thanks for getting the ball rolling @amitlissack

[amitlissack]

First Steps

Split up code

Create a new sub project in mono repo

It can me named py-opentrons or something else more imaginative.
This would contain everything in api project minus opentrons.server and opentrons.app packages. Almost everything in current api is applicable to building a package vs an application.

Refactor api sub-project

This is the api server application. For the time being both the aiohttp (opentrons.server) and fastapi (opentrons.app) packages will be in this project.

Directory Structure

| README.md
| Makefile
| ...
└- tests
|  | __init__.py
|  └- app
└- app
   | __init__.py
   | main.py
   └- aiohttp ( to be deprecated )
   └- fastapi ( move contents one level up when aiohttp is deprecated )

Makefile for API

• api will not be packaged for pypi.
• It will be an application installed and run as a service (/usr/local/opentrons/api)
• target to generate openapi.json from fastapi app

Split Configuration

API configuration for the moment is just a handful of items and they should be moved from command line args and into their own yaml file.

[amitlissack]

Second Steps

Further divide API and library code

The request handlers in opentrons.server.endpoints contain a good amount of code that should be moved to the opentrons library.

Examples

• the move function in control.py should be moved out to the shared library. Interacting with the motion lock is too much intimate knowledge of the hardware for a request handler.
• take_picture should call a method in the library rather than invoking ffmpeg.

Move business logic to FastAPI

This ticket #4963

[amitlissack]

We all agree that this should happen. Created an epic for the project #5009