Screwdriver: [Feature Request] Disable build cache with specific job

Created on 17 Sep 2019 · 13Comments · Source: screwdriver-cd/screwdriver

What happened:
Currently, build cache scopes are event, pipeline and job.
I'd be nice if download build cache step can be skipped in the specific job for build speed up.

What you expected to happen:
Users can disable build cache in the specific job.

feature

Source

kumada626

👍1

All 13 comments

There are some solutions to fix this issue.

Solutions1

This idea is to add new object disable or enable in cache.

example1

cache:
  event:
    - $SD_SOURCE_DIR/node_modules
  pipeline:
    - ~/.npm
    - test.txt
  job:
    test: [/tmp/test-dir, /tmp/test.txt]
    publish: [publish.txt]
  disable:
    - job1
    - job2

All jobs other than job1 and job2 caches file at this case.

example2

cache:
  event:
    - $SD_SOURCE_DIR/node_modules
  pipeline:
    - ~/.npm
    - test.txt
  job:
    test: [/tmp/test-dir, /tmp/test.txt]
    publish: [publish.txt]
  enable:
    - job3

Only job3 caches files at this case.

Solutions2

This idea enables cache level indication.

cache:
  event:
    - $SD_SOURCE_DIR/node_modules
  pipeline:
    - ~/.npm
    - test.txt
  job:
    test: [/tmp/test-dir, /tmp/test.txt]
    publish: [publish.txt]
  disable:
    event:
      - job2
  enable:
    pipeline:
      - job1
      - job3

And we have to handle it as an error; event should not to be put both disable and enable.

disable:
  event:
    - job2
enable:
  event:
    - job1

Solution2.1

This idea does not set both enable and disable together.

cache:
  event:
    - $SD_SOURCE_DIR/node_modules
  pipeline:
    - ~/.npm
    - test.txt
  job:
    test: [/tmp/test-dir, /tmp/test.txt]
    publish: [publish.txt]
  enable:
    pipeline:
      - job1
      - job3

Solution3

example:

cache:
  event:
    - filename: $SD_SOURCE_DIR/node_modules
      enable:
        - job1
        - job3
  pipeline:
    - filename: ~/.npm
      enable:
        - job2
    - filename: test.txt
      disable:
        - job3
  job:
    test: [/tmp/test-dir, /tmp/test.txt]
    publish: [publish.txt]

This one is more flexible like:

enable/disable setting for each files
enambe/disable coexistence in a cache level

This solution breaks present specification, so users have to migrate their cache settings.

Solution4

Specifying true/false for each jobs.

cache:
  event:
    - $SD_SOURCE_DIR/node_modules

jobs:
  main:
    cache: false
    image: node:12
    steps:
      - ls $SD_SOURCE_DIR/

yoshwata on 16 Apr 2020

What do you prefer to? Or any ideas? I think solution1 is sufficient.

yoshwata on 16 Apr 2020

I think it would be easier if I could control it in the job definition part.

cache:
  event:
    - $SD_SOURCE_DIR/node_modules

jobs:
  main:
    cache: false
    image: node:12
    steps:
      - ls $SD_SOURCE_DIR/

Even if users can control each cache level indication finely, I think that the impact on speed will be small. This is to speed up the build, so I don't think users need detailed control.

wahapo on 16 Apr 2020

Why solution2 and solution3 can specify diable and enable?
I think solution2 and solution3 able to specify only disable or enable is good idea.

yuichi10 on 16 Apr 2020

I think the solution by @wahapo looks better than the others.
It is simple to write rules and users can understand it easily.

MysticDoll on 16 Apr 2020

I prefer to @wahapo 's suggestion of adding disable flag to job's setting. It looks a bit cumbersome to set it up in the cache part.

kumada626 on 16 Apr 2020

@wahapo Added as solution4
@yuichi10 Added as solution2.1

yoshwata on 16 Apr 2020

How many people do you think cache on a per-file basis?
If a lot of people are going to use it, I think it needs to be cached on a per-file basis.

If there are a few use cases that user needs to write a detail cache setting, solution4 looks good.

yuichi10 on 16 Apr 2020

I also recommend @wahapo 's idea.
If you really want to control the cache in detail at all levels of job, event, and pipeline, I think it would be cleaner to make a YAML just for controlling the cache.