Prefect: cannot turn off checkpointing with prefect server

[quartox]

Description

prefect server start does not respect turning off checkpointing.

Expected Behavior

Files should not be written to ~/.prefect/results if checkpointing is turned off.

Have tried:

@prefect.task(checkpoint=False)

in config.toml:

[flows]
checkpointing = false

Reproduction

Run prefect server start with a task using @prefect.task(checkpoint=False) and see task write result to ~/.prefect/result.

Environment

{
  "config_overrides": {
    "flows": {
      "checkpointing": true
    },
    "server": {
      "database": {
        "host": true
      },
      "endpoint": true,
      "graphql": {
        "host": true
      },
      "hasura": {
        "host": true
      },
      "host": true,
      "host_port": true,
      "port": true,
      "ui": {
        "endpoint": true,
        "graphql_url": true,
        "host": true,
        "host_port": true,
        "port": true
      }
    }
  },
  "env_vars": [
    "PREFECT__FLOWS__CHECKPOINTING"
  ],
  "system_information": {
    "platform": "Linux-4.15.0-91-generic-x86_64-with-debian-buster-sid",
    "prefect_version": "0.10.4",
    "python_version": "3.7.7"
  }
}

[cicdw]

Hi @quartox - just to confirm, you re-registered your task with checkpoint=False explicitly set correct? The reason I want to triple check is that this line should prevent checkpointing under any scenario as long as checkpoint=False is set on the task. Of course if this check isn't being respected then this is definitely a bug!

[quartox]

Can confirm, re-registered the task with checkpoint=False, new version found in UI and it continues to write files.

[joshmeek]

Looked into this and it is happening on this call here https://github.com/PrefectHQ/prefect/blob/6d141372cf89064d24bbafad582a9db26be0cbd5/src/prefect/engine/cloud/task_runner.py#L114

where the prepare_state_for_cloud does not check whether or not checkpointing is disabled
https://github.com/PrefectHQ/prefect/blob/6d141372cf89064d24bbafad582a9db26be0cbd5/src/prefect/engine/cloud/utilities.py#L4-L23

@cicdw Should we account for this behavior now or in the new Results wiring PR where I _think_ some of this behavior is changing?

From debugging session:

> /Users/josh/Desktop/code/prefect/src/prefect/engine/cloud/task_runner.py(114)call_runner_target_handlers()
-> cloud_state = prepare_state_for_cloud(new_state)
(Pdb) 
[2020-05-01 16:47:02] DEBUG - prefect.LocalResultHandler | Starting to upload result to /Users/josh/.prefect/results/prefect-result-2020-05-01t16-47-02-314073-00-00...
[2020-05-01 16:47:02] DEBUG - prefect.LocalResultHandler | Finished uploading result to /Users/josh/.prefect/results/prefect-result-2020-05-01t16-47-02-314073-00-00...

[cicdw]

@joshmeek yea for sure, actually that entire function was removed in the new PR so I think that will close this issue once it's released.

[lauralorenz]

Sounds like 0.11.0 should be going out next week, so we will revisit whether this is still a bug then since based on the above conversation the offending line should be removed from that release.