Prefect: How can I do flow.register() and flow.run_agent() to local prefect-server in docker environment?

[ponggung]

Current behavior

I use docker-compose to build my project, and I try to run prefect-server dashboard locally.

after register and run...

if __name__ == "__main__":
    flow.register(project_name="MapReduce")
    flow.run_agent()

It could be successful if I run my code outside docker, but if I run in docker,
it shows

  File "/opt/conda/lib/python3.7/site-packages/prefect/core/flow.py", line 1366, in register
    no_url=no_url,

Proposed behavior

Please describe your proposed change to the current behavior

I set network connection between my-network and prefect-serve in docker-compose.yml

networks:
  my-network:
  prefect-server:
    external: true

I can send requests to cli_apollo_1:4200 in container ,
but still not work to register my script to local prefect-server.

Example

Please give an example of how the enhancement would be useful

Is there any way to specify host and port like this?

    flow.register(host="cli_apollo_1", port=4200, project_name="MapReduce")
    flow.run_agent(host="cli_apollo_1",port=4200)

[joshmeek]

@ponggung Could you post the full stack trace from the error?

[ponggung]

@ponggung Could you post the full stack trace from the error?

Sure

from prefect import Flow, task
import prefect

# ------------------------------------
# define some tasks
@task
def numbers_task():
    return [1, 2, 3]

@task
def map_task(x):
    return x + 1

@task
def reduce_task(x):
    return sum(x)

# ------------------------------------
# build a flow
with Flow("MapReduce") as flow:
    numbers = numbers_task()
    first_map = map_task.map(numbers)
    second_map = map_task.map(first_map)
    reduction = reduce_task(second_map)


if __name__ == "__main__":
    flow.register(project_name="MapReduce")
    flow.run_agent()

(base) root@73b40e9346b8:/app/tutoraial/prefect-tutorial# python local_mapreduce.py 
Traceback (most recent call last):
  File "local_mapreduce.py", line 33, in <module>
    flow.register(project_name="MapReduce")
  File "/opt/conda/lib/python3.7/site-packages/prefect/core/flow.py", line 1370, in register
    version_group_id=version_group_id,
  File "/opt/conda/lib/python3.7/site-packages/prefect/client/client.py", line 596, in register
    project = self.graphql(query_project).data.project  # type: ignore
  File "/opt/conda/lib/python3.7/site-packages/prefect/client/client.py", line 215, in graphql
    token=token,
  File "/opt/conda/lib/python3.7/site-packages/prefect/client/client.py", line 178, in post
    return response.json()
  File "/opt/conda/lib/python3.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/opt/conda/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/opt/conda/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/conda/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[oleksandr]

@ponggung I've managed to do that via putting the following ~/.prefect/backend.toml (for Mac OS, different host name for Linux):

backend = "server"

[server]
host = "http://docker.for.mac.host.internal"
port = "4200"
host_port = "4200"
endpoint = "${server.host}:${server.port}"

and that registered my task at server successfully.

[joshmeek]

Yeah @oleksandr! Ultimately this can be solved by setting the server host to point back to your host machine if you are running inside a Docker container. This is due to the fact that you have the prefect server running on your host machine and your flow registration, happening inside another container, needs to reach back out to get to the server.

e.g.

export PREFECT__SERVER__HOST="http://host.docker.internal"