Argo: Named parallelism limits

Created on 11 Mar 2019  路  2Comments  路  Source: argoproj/argo

FEATURE REQUEST:
I propose to have named parallelism, and by that I mean the ability to define several parallelism limits, and let different templates define which parallelism limit it consumes.

Why
We have several different tasks scattered around in the workflow which consume some limit resource, e.g. a database with limited capacity. We want maximum parallelism in general, but have a global limit of e.g. 10 to the database so it does crash. The different tasks does not fit after each other in a steps list.

Concrete proposals
I have 3 different proposals, in increased complexity

Proposal 1
Each task consumes a single parallelism-limit:
```...
spec:
parallelism-limits:

  • name: influx-limit
    value: 10
  • name: sql-limit
    value: 4
    templates:
  • name: some-thingy-which-needs-influx
    parallelism: influx-limit # 10 of these can run at the same time
    ...
*Proposal 2* 
Each task can consume 1 of several limits:
```...
spec:
  parallelism-limits:
  - name: influx-limit
    value: 10
  - name: sql-limit
    value: 4
  templates:
  - name: some-thingy-which-needs-both-influx-and-sql
    parallelism:
    - influx-limit 
      sql-limit # So at most 4 of this task can run at the same time
...

Proposal 3
Each task can consume x of several limits:
```...
spec:
parallelism-limits:

  • name: influx-limit
    value: 10
  • name: sql-limit
    value: 4
    templates:
  • name: some-thingy-which-needs-both-influx-and-sql-and-a-lot-of-influx
    parallelism:

    • influx-limit: 3 # at most 3 of these can run at the same time

    • sql-limit:1

      ...

      ```

duplicate enhancement

Most helpful comment

I'm not sure if this is relevant, but I'd be quite interested in a global limit that works across workflows if that isn't already possible. That is, submitting two workflows, and having them essentially _share a semaphore_ (for lack of better phrasing).

All 2 comments

I'm not sure if this is relevant, but I'd be quite interested in a global limit that works across workflows if that isn't already possible. That is, submitting two workflows, and having them essentially _share a semaphore_ (for lack of better phrasing).

This sounds like #2550 . Closing and duplicate.

Was this page helpful?
0 / 5 - 0 ratings