Prefect: Cloudpickle error when base_image isn't specified

[nanseay]

Description

If you remove the base_image kwarg from storage and attempt to deploy a flow, a Cloudpickle error is triggered
Traceback (most recent call last):
File "/root/.prefect/healthcheck.py", line 12, in
import cloudpickle
ModuleNotFoundError: No module named 'cloudpickle'

Expected Behavior

No errors and flow deploys as expected (note: using kwarg base_image="prefecthq/prefect:0.7.0-3.7" allowed me to deploy my flow successfully)

Reproduction

Remove base_image kwarg from storage and deploy flow

Environment

Running Core 0.7.0

[joshmeek]

@cicdw @wagoodman Do you think this could be due to cached layers? I haven't encountered this yet and I often don't provide a base image.

[cicdw]

@nanseay could you include what version of Python you are running?

[nanseay]

Yep, Python 3.7.3

[joshmeek]

We looked at this offline and it seems like it's an issue with assumptions on the Docker storage. If you provide that your base_image is python:3.7 and you specify a prefect_version of 0.7.0 it won't install that version of prefect (no matter what you provide). This is due to the assumption that if a base image is provided the docker storage does not perform the extra prefect installation.

https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/docker.py#L94

[wagoodman]

After more discussion, we think the best way forward is to always attempt to install the prefect package unless it is already installed. For example, regardless of the base_image or prefect_version there is an expectation that prefect will be installed, thus we should always attempt to install it. However, we should check for existing installations (via pip show prefect) and do not reinstall if it already exists.

Additionally, it would be good to check if the user specified a prefect_version that the final installed version matches what the user provided. This will be a safety measure for when a user brings a specific image that already has prefect installed, and additionally provides a prefect_version (which mismatches the version from the base image)... we want the flow deploy to fail since there is an unexpected version in use.