Screwdriver: Conditionally retry a step on failure

[jithine]

What happened:

Sometimes a step can fail because of external dependencies. There is no option to retry the command under different circumstances without restarting a new build or code changes.

What you expected to happen:

Provide an option to conditionally retry failed steps.

1. Screwdriver workflow to provide a step config to specify that step must be retried.
2. Retry should support setting different environment variables.
3. Optionally provide a condition which should determine whether retry should happen or not.
4. For Screwdriver provided setup & teardown steps. cluster admins should be able to define the retry condition.
   
   
   
   1. User's can optionally specific retry condition without ability to override command
   
   

For example

steps:
 sd-setup-scm:
    command: git clone foo bar....
    retry: # object below or just `true`
      condition: $GIT_SHALLOW_CLONE == true # optional
      maxRetry: 3 # optional, default 1
      interval: 3 # optional, default 0 (second)
      environment: # optional
         GIT_SHALLOW_CLONE: false

How to reproduce it:

N/A

[jithine]

This is also related to https://github.com/screwdriver-cd/screwdriver/issues/1208 When making model changes we should keep both features in mind

[catto]

I really want this feature 👍 How about adding some useful keys and changing indentation?

steps:
 sd-setup-scm:
    command: git clone foo bar....
    retry: # object below or just `true`
      condition: $GIT_SHALLOW_CLONE == true # optional
      maxRetry: 3 # optional, default 1
      interval: 3 # optional, default 0 (second)
      environment: # optional
         GIT_SHALLOW_CLONE: false

[jithine]

Adding retry options under retry object makes sense.

[tkyi]

Another ability a user asked for related to this issue was optionally being able to specify restarting from a previous job.

[rm-you]

Also -- it would be ideal if condition could be a regex matcher or something for the log output. For example, scanning the output for .*dial tcp: i/o timeout.* (and being able to set that restart config GLOBALLY in our template) would resolve more than 50% of our spurious failures.