Prefect: Start a Local Agent without Default Storage Labels

Created on 1 Sep 2020  路  5Comments  路  Source: PrefectHQ/prefect

Current behavior


Starting a LocalAgent will add storage labels by default, which can be confusing for new users and cluttered for users with many agents. Typically, starting a LocalAgent will produce the following by default:

[2020-09-01 15:34:50,612] INFO - agent | Starting LocalAgent with labels ['my-machine.local', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage']

Proposed behavior


Starting a Local Agent should automatically exclude the following labels:
'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage'

These labels should have some option to turn off by default or be removed by default.

Implemented here: https://github.com/PrefectHQ/prefect/blob/5edb9b39bb53e2cb11d37832f3f3f56bb1a3bc6a/src/prefect/agent/local/agent.py#L97

Example


prefect agent start -t "TOKEN"
Produces:

[2020-09-01 17:56:00,216] INFO - agent | Starting LocalAgent with labels ['my-machine.local', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage']
[2020-09-01 17:56:00,216] INFO - agent | Agent documentation can be found at https://docs.prefect.io/orchestration/
[2020-09-01 17:56:00,216] INFO - agent | Agent connecting to the Prefect API at https://api.prefect.io
[2020-09-01 17:56:00,532] INFO - agent | Waiting for flow runs...

But can clutter the UI:
0

Potential Results:
Agent management will be decluttered for users with many Agents.
Quality of life change - specifying execution environments via flow affinity is a bit easier to navigate.

Please let me know if there is a current solution or reasoning I'm not aware of!

enhancement

Most helpful comment

Or maybe add some information about best practice regarding multiple agents on different hosts working on different runs of the same flow or something like that...

All 5 comments

https://prefect-community.slack.com/archives/C014Z8DPDSR/p1600439888019800?thread_ts=1600359958.136900&cid=C014Z8DPDSR

This seems to be a common problem. Should we consider adding some specific message for flows registered with the Local Environment and/or Local agents?

This Flow was registered with the LocalExecutor and has been given the following labels: "dylans-macbook-pro", "github-storage". You'll need an agent with matching labels to connect to Prefect Cloud for it to run
For example

Or maybe add some information about best practice regarding multiple agents on different hosts working on different runs of the same flow or something like that...

Tangentially related to #2888 I think we can start to relax the hard enforcement of default labels

I perfectly agree with @kmoonwright. Prefect shouldn't force me to use labels and these should be disabled by default (such as how GitLab Runners system is working for example).

There is already a flag (Default=True) to put the Hostname Label. hostname_label.
I'm foreseeing something very similar for those storage labels!

Was this page helpful?
0 / 5 - 0 ratings

Related issues

orcaman picture orcaman  路  3Comments

jlowin picture jlowin  路  3Comments

emcake picture emcake  路  3Comments

ponggung picture ponggung  路  3Comments

jlowin picture jlowin  路  4Comments