Screwdriver: Shared Steps

[stjohnjohnson]

As a user, I want to be able to share code for building software, so that my builds are less prone to drift.

Currently, users share code by cloning git repositories or installing packages in their build. However, due to the ability to select your own containers to run it, both of these options are limited to either only share shell scripts or specific to one language. We need an option to effectively share language independent commands.

Screwdriver should support an "omnibus-like" package manager natively and offer a cached installation/download of a global Screwdriver SDK state.

Rules:

• Package manager is available in every build
• Global SDK state is restored into the package manager installation/download location
• Package manager can use the global SDK state to reduce installation/download time
• Method to update global SDK state
• Method to refer to installed packages via semver

---

Example

We could use Habitat.sh and then just restore a globally stored /hab directory. Then users (and bookends) do things like:

Not require git to be installed

$ hab pkg install core/git
$ hab pkg exec core/git git clone

Bundle launcher

$ hab pkg install screwdriver/launcher
$ hab pkg exec screwdriver/launcher logservice
$ hab pkg exec screwdriver/launcher launcher

Provide a collection of commands

$ hab pkg install nodejs/6
$ hab pkg exec nodejs/6 npm install
$ hab pkg exec nodejs/6 eslint

Downside

Habitat currently only allows you to refer to packages via:

• name
• name + exact version
• name + exact version + exact buildnumber

So it doesn't allow users to do things like ^2.3.0 (version greater or equal to 2.3.0 but smaller than 3.0.0).

---

Future plans

It would be good to be able to cache what users install as well, to improve their execution time. Something like the following:

• Job started for the first time
  
  
  
  • If main job already completed, load last-successful-state from main job
  
  
  • If main job not completed, restore global state
  
  
• Job started for the non-first time
  
  
  
  • Load last-successful-state for that job
  
  
• Job finished (non-PR)
  
  
  
  • Store state as last-successful-state for that job
  
  

[bdangit]

Package versions in Habitat are not limited to that. There is another one which could you help with control "major" or "major.minor" releases.

• {origin}/{pkg_name}{major}
• {origin}/{pkg_name}{major}{minor}

examples:

• screwdriver/launcher2 for 2.x.x
• screwdriver/launcher25 for 2.5.x

What you should understand is that these are the package names and that we shouldn't lock down to the exact_version and the buildnumber. You will want to continuously track the head. ref: https://github.com/habitat-sh/core-plans/blob/master/CONTRIBUTING.md#package-name-conventions

So in the examples above, you would see there's a plan.sh for launcher2 that would track all 2.x.x changes. And then you can have another plan.sh for launcher3 that would track all 3.x.x. If you need more controlled changes and releases to your customers, then you can go down to the minor level and have many plans for each minor revision like launcher20, launcher21, etc.

[petey]

There must be some token that separates semver version parts, or you wouldn't be able to tell the difference between screwdriver/launcher21 being version 21.x.x or 2.1.x. Habitat specifically has origin/name/version/release. https://www.habitat.sh/docs/concepts-packages/

[stjohnjohnson]

We could look into giving a PR to Habitat to support at least the basic Semver concepts like:

• 2: latest 2.x.x
• 2.4: latest 2.4.x

[bdangit]

https://github.com/habitat-sh/habitat/issues/1459

You have freedom over the pkg_name. So if you want to have a delimiter of _ or 0 or whatever that is sane, then you can make it look like 2_1. It almost is not relevant, because when you install packages, you have to specify the exact pkg_name anyway.

[catto]

We have started thinking about the design.

[s-yoshika]

Design proposal:
We use habitat to realize the shared step feature,
because

• Shareable package
• Many usefull packages available from public depo
• Store application and it's dependencies into /hab
• Easy to use(just fetch a binary file)

In user's screwdriver.yaml:

jobs:
  main:
    steps:
      - mvn-test: hab pkg install core/maven && hab pkg exec core/maven mvn test

Habitat can almost resolve the shared step feature, so it seems that there is no many development for this feature.
The need things is below:

• Build container must be installed hab command
  
  
  
  • install in launcher's Dockerfile
  
  
  • or
  
  
  • create a new bookend which installs hab
  
  
• Builds should be able to share hab packages to avoid extra packages install
  
  
  
  • docker and k8s executors can mount /hab directory in the host to containers with read-write permissions
  
  
• It might be better to have a wrapper command for hab
  
  
  
  • Basically, Executing command by hab have to include 2 commands(pkg install, pkg exec), the wrapper command allows for user to write just 1 command
  
  
  • example: sd_step exec $name [$version] $command…
  
  

Please let me know if any points are missing.

[d2lam]

Thanks for the update @s-yoshika
I think it makes more sense to include hab in the launcher. Making a bookend to install hab doesn't make sense to me because there will be different installation methods depending on the build container's image.

Do we want to handle semver in this pass? For example, allow user to specify ^1.2.0. Or we can leave it to the second pass and have a MVP for now.

[petey]

I think we've kinda earmarked habitat as our tech of choice for shared steps since the beginning, I think the pros outweigh the cons for it. We should tackle the semver issue in a later pass.

The launcher is currently using an alpine linux base image for its docker build. This allows the image to be extremely tiny, but comes with caveats:

• It compiles its code with musl, as opposed to glibc, which often causes issues with binaries.
• Alpine has a limited set of packages available for it, and habitat is not one of them.

I have misgivings about mounting anything into containers read/write, especially if this is shared between all builds on the cluster. Let's figure out sharing hab packages later.

A sugar command that does hab pkg install $pkg && hab pkg exec $pkg $command makes a lot of sense, but only if hab does a no-op if it has already installed that package.

[d2lam]

@petey Looks like it's smart enough to not reinstall if the package already exists:

[1][default:/src:0]# hab install core/redis
» Installing core/redis
↓ Downloading core/redis/3.2.4/20170215222111
    569.30 KB / 569.30 KB \ [=========================================================================================================================] 100.00 % 3.87 MB/s  
→ Using core/glibc/2.22/20160612063629
→ Using core/linux-headers/4.3/20160612063537
✓ Installed core/redis/3.2.4/20170215222111
★ Install of core/redis/3.2.4/20170215222111 complete with 1 new packages installed.

Install again:

[2][default:/src:0]# hab install core/redis
» Installing core/redis
→ Using core/redis/3.2.4/20170215222111
★ Install of core/redis/3.2.4/20170215222111 complete with 0 new packages installed.

[bdangit]

yes, habitat is "idempotent" in that respect. I believe there's still a local cache that hab will look in before reaching out to a depot.

With regards to installation of hab, there are many ways you can install the hab binary:

1. you can use this cookbook: https://supermarket.chef.io/cookbooks/habitat
2. you can use this install.sh script: https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh

If you don't want to mess with installing hab, you can always export docker images or tarballs where it has your shared steps and the hab binary included.

[s-yoshika]

@d2lam @petey @bdangit Thanks for your reviews and advices!

semever:
Okay, we want to plan semver support in second pass.

hab install in launcher container:
I think that we have not to compile hab in launcher container. We can just include hab single binary file into laucher docker image as @bdangit says.

re-install for packages:
@d2lam, @bdangit thanks for follow, I also confirmed that hab don't re-install same packages.

I have misgivings about mounting anything into containers read/write, especially if this is shared between all builds on the cluster. Let's figure out sharing hab packages later.

Okay, we think sharing the packages in later pass.
If Sharing volume with read/write mode between all builds isn't good, it may be better that we mount volumes per job not all builds.

I breakdown the plan into some passes.

First pass:

• Change launcher's Dockerfile
  
  
  
  • install hab and it's wrapper command in Dockerfile
  
  
  • install path: /opt/sd/bin/
  
  
• Create a hab wrapper command(sd_step(tentative))
  
  
  
  • accept the folloing arguments:
    
    
    
    
    
    • $package_name
    
    
    
    • $package_version
    
    
    
    • $commands…
    
    
    
  
  
  • the command just execute following commands
    
    
    
    
    
    • hab pkg install …
    
    
    
    • hab pkg exec …
    
    
    
  
  
  • only handle the $version as exact version at this pass
  
  
• Sharing packages
  
  
  
  • no share at this pass, use just local volume in containers
  
  

Second pass:

• hab wrapper command
  
  
  
  • can handle semver for hab package version
  
  
• Sharing packages
  
  
  
  • docker and k8s executors mount hab dir to containers for sharing packages
  
  

[bdangit]

Maybe I'm missing some context but why are you mounting a shared volume just to share a pretty "static" binary? If you are deploying on K8s, any node that runs the container would have the Docker image you start would load up immediately by checking local cache before pulling from a Docker hub.

[s-yoshika]

Sorry for confusing you.
Yes, binary for hab command should be included into docker image.
But, I think that we have to mount shared volume for /hab directory to sharing downloaded hab packages (e.g. core/git) and their dependencies between builds to improve their execution time.

[bdangit]

I see. Thanks @s-yoshika. This might bite you or may help you. One thing that may happen is two Hab processes (or containers) downloading the same dependency at the same time. Another scenario is a dependency is paritially downloaded and the other container may think that it's ready which depends on well Hab implemented package integrity check. All of these are good to do in the second pass. :)

Btw, here's a curlbash command that can be used to install hab

  $ curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | sudo bash

[kumada626]

We have started creating shared-step, and decided to create the sd_step (hab wrapper command) as a go binary. So we'll need a new repo for the sd_step go binary. Could you create an empty repo for the sd_step go binary? @d2lam

[d2lam]

@kumada626 Sorry I didn't see this until now: https://github.com/screwdriver-cd/sd-step

[d2lam]

I named it sd-step but if you have a better name, let me know.

[kumada626]

@d2lam Thanks!

[s-yoshika]

We are starting the second pass (support for semver).


It is design bellow.

Background:

Now, sd-step command accepts only the exact version number for a habitat package but we want to specify the version number more flexibly with semver (like ^1.0.0).

How:

The sd-step just wraps habitat but habitat itself doesn’t support the flexible version such as semver.
so we need to implement the version parse feature for semver in the sd-step command.

Currently, the sd-step command has pkg-version option which specifies the habitat package version to install.
So we extends the option to parse semver formated version too.

The command usage is like this:

$ sd-step exec core/git git clone https://github.com/screwdriver-cd/sd-step.git --pkg-version ^2.7.4

Problem:

In habitat, the format of the package version seems not to be particularly restrictive.
It can be designed by the author(s) of the application or service.

https://www.habitat.sh/docs/concepts-packages/

So we can’t necessarily apply the semver mechanism to any habitat’s package versions.

So we suppose that the package version has semver like version($majorver.$minorver.$patchver) if the pkg-version option in semver format is passed.

Even if a user specifies the pkg-version option with semver format, but specified package have any unknown format of version(not semver like version), then the sd-step command ignore the pkg-version option (latest version will be used).

[catto]

I think the restriction that sd-step will support only semver-like version would be okay for now.
When a package doesn't have a semver-like version, sd-step should print a warning message for using the latest version or exit with an error. I suggest the latter one for safety.

[s-yoshika]

@catto Thanks! for you review.

When a package doesn't have a semver-like version, sd-step should print a warning message for using the latest version or exit with an error. I suggest the latter one for safety.

Good idea. I think that Exiting with an error is prefer than just ignoring pkgVersion option.

[catto]

semver feature has been implemented and published: screwdriver-cd/sd-step#3
launcher now includes it in v4.0.20 or later

TODO:

• [x] use shared step in sd-setup-scm step
• [x] add functional test for semver feature

[catto]

We might want to support binlink feature of Habitat. It will make easier to use commands such as git and node in tiny build containers. (see screwdriver-cd/scm-github#59)
But I think it would be out of concept of "shared step". What do you think?

[stjohnjohnson]

I'd rather not use binlink because we don't want to affect existing installed versions of those binaries.

[catto]

🙆

[d2lam]

This feature is done. Thanks @s-yoshika @kumada626 @tk3fftk @catto!