Prefect: Create a beginner-friendly walkthrough of setting up a Dask Cluster for Prefect

Created on 27 Feb 2020  路  8Comments  路  Source: PrefectHQ/prefect

Items we should cover:

  • how to choose number of workers
  • what dependencies should workers have
  • how to choose node types / sizes
  • how to choose nprocs / nthreads
  • how to choose resource requests / limits for workers (if on k8s)
  • how to spin them up
  • what can happen to disrupt your flow (e.g., worker evictions)
  • possibly mention networking from workers to schedulers (security / exposing scheduler as a service / etc.)

cc: @joeschmid / @jsignell

docs
Source
picture cicdw
👍2

Most helpful comment

Ok just a couple updates as a list here:

  • [x] there is a PR to the dask blog Julia and I worked on (https://github.com/dask/dask-blog/pull/63)
  • [x] I will give a tl;dr of configuration concerns in a tooltip and link to the dask blog on the Dask Deployment tutorial in the prefect docs once the above is merged (https://github.com/PrefectHQ/prefect/pull/3060/files)
  • [ ] there is a google doc for a separate blog that is more prefect-focused about connecting to your dask deployment that is still on me
picture lauralorenz on 16 Jul 2020
👍2

All 8 comments

What form would this documentation take? If it were a blog post rather than straight docs it might be more generally useful. This is also the kind of thing that we'd be happy to collaborate on.

jsignell picture jsignell on 3 Mar 2020

Hey @jsignell - sorry for the delayed response! I think either blog or documentation could work well, and I don't think they're mutually exclusive

cicdw picture cicdw on 6 Mar 2020

Agree! Let me know if you want to team up :)

jsignell picture jsignell on 11 Mar 2020
👍1

Hey @jsignell I'm interested in this too and have some bandwidth to team up! It also so happens that I am a beginner Dask gal in the first place so on the one hand, sorry I will be a n00b but on the other hand, I think I will be a great person to write with since I am the type of person to be writing to. I think this might be easier to brainstorm in a google doc or similar in the initial phase, so if you agree, if you don't mind requesting access to this google doc with whatever email you want to hang out on, we can start to work asynchronously there or organize to sync up to chat about it. Let me know if you wanted to team up a different way. Thanks :)

lauralorenz picture lauralorenz on 11 Mar 2020

Sweet! I just requested access :)

jsignell picture jsignell on 12 Mar 2020

I'd be interested in helping out on this as well, if y'all need a hand.

jcrist picture jcrist on 13 Apr 2020
👍2

@jcrist I'll add you to the invite for our next sync :)

jsignell picture jsignell on 13 Apr 2020
❤1

Ok just a couple updates as a list here:

  • [x] there is a PR to the dask blog Julia and I worked on (https://github.com/dask/dask-blog/pull/63)
  • [x] I will give a tl;dr of configuration concerns in a tooltip and link to the dask blog on the Dask Deployment tutorial in the prefect docs once the above is merged (https://github.com/PrefectHQ/prefect/pull/3060/files)
  • [ ] there is a google doc for a separate blog that is more prefect-focused about connecting to your dask deployment that is still on me
lauralorenz picture lauralorenz on 16 Jul 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rej-jsa picture rej-jsa  路  4Comments
joshmeek picture joshmeek  路  4Comments
mark-w-325 picture mark-w-325  路  3Comments
orcaman picture orcaman  路  3Comments
Trymzet picture Trymzet  路  4Comments