Screwdriver: Shared Step Improvement

[stjohnjohnson]

List of things we should probably do:

• [x] Add Readme to https://github.com/screwdriver-cd/sd-step
• [x] Change CLI interface per https://github.com/screwdriver-cd/screwdriver/issues/650
• [x] Add design doc to https://github.com/screwdriver-cd/screwdriver
• [x] Check for root, if not, sudo during install per https://github.com/screwdriver-cd/screwdriver/issues/666
• [x] Capture logs for debugging if needed instead of sending to /dev/null: https://github.com/screwdriver-cd/sd-step/blob/b9cc1857f5d7a39626860193e3da7dddfab1fb99/sd-step.go#L79
• [ ] Provide option for using alternative sources for hab packages (like artifactory)

[yuichi10]

Summary 16/01/2018

I created sd-cmd which can exec binary commands. But there are no command API for now, so I just tried sd-cmd with mock.
I have decided not to use urfave/cli library for the command, because it would receive many kinds of arguments or flags.

I believe that I have done implementing almost all functions for the binary execution feature expect logging feature.
The problem summary is that I could not find the way how to output logs to both file and terminal in right order in real time, in Golang.
It is long sentences, so I posted the detail of this problem to 677-sd-cmd channel on Slack.
https://screwdriver-cd.slack.com/archives/C8F6DMR7W/p1516184037000019

[drewfish]

Some ideas/questions about the commands design:

• does it make sense to add a link to "habitat"? I have no idea what that is
• can the same command be called multiple times within the same build? does each invocation get a separate log file?
• presumably there'll be shared steps to provision/release selenium. which team will author those?
• the $SD_ARTIFACTS_DIR/.sd/commands/namespace/name/version/timestamp.log directory structure seems overly long
  
  
  
  • something like $SD_ARTIFACTS_DIR/.sd/commands/{timestamp}-{namespace}-{name}.log would be nice. would be nice if this log also contains the fully-resolved version of the command as well as it's full invocation (all arguments)
  
  

[drewfish]

Since templates don't ship with code (I've been told) it would be really nice if an sd command could call other sd commands. This would make algorithm composition much easier (more straightforward).

[stjohnjohnson]

@drewfish

• Good point, sorry. https://www.habitat.sh/
• Can be called unlimited times.
• Right now there is no name space restrictions, so it can be anyone. The thing we haven't thought of yet is how to bundle commands or even templates with SD.
• I like it, good idea.

And for composition, it's completely possible. Binary or Habitat package would work fine, Docker commands would not have access to spin up new containers.

[drewfish]

The one trick with "this log also contains.. all arguments" is that that might contain the env vars which have the secrets, e.g. sd_cmd foo/bar@1 --token ${MY_SECRET}. It might be better if there's a best practice for commands suggesting that they "print invocation details (but not secrets) as the first thing you do, to aid debugging".

[jithine]

Can we use sd-cmd shared command to do validation of the shared command rather than installing a separate npm module screwdriver-command-validator

Eg: sd-cmd validate -f sd-command.yaml

[minz1027]

To add to the validation thing ^^

We can have a command/validate api endpoint and have sd-cmd validate hit that endpoint. Something like this: https://github.com/screwdriver-cd/screwdriver/blob/master/plugins/template-validator.js

[tk3fftk]

There are some problem when I try to implement habitat format with mode: local.
.hart file does not have own package information to execute (e.g. python2/python). If we want to obtain the information, we have to parse install log or inspect .hart file with shellscript.

output log exmaple:

$ sudo hab pkg install /hab/cache/artifacts/python2-python-2.7.13-20170523194107-x86_64-linux.hart
☛ Verifying python2/python/2.7.13/20170523194107
→ Using core/bzip2/1.0.6/20170513212938
→ Using core/cacerts/2017.01.18/20170513215104
→ Using core/gcc-libs/5.2.0/20170513212920
→ Using core/gdbm/1.11/20170513213716
→ Using core/glibc/2.22/20170513201042
→ Using core/linux-headers/4.3/20170513200956
→ Using core/ncurses/6.0/20170513213009
→ Using core/openssl/1.0.2j/20170513215106
→ Using core/readline/6.3.8/20170513213506
→ Using core/sqlite/3130000/20170514005747
→ Using core/zlib/1.2.8/20170513201911
✓ Installed python2/python/2.7.13/20170523194107
★ Install of python2/python/2.7.13/20170523194107 complete with 1 new packages installed.

inspect .hart file example:

$ tail -n +6 /hab/cache/artifacts/python2-python-2.7.13-20170523194107-x86_64-linux.hart | xzcat | tar -t | head -1 | sed 's/\hab\/pkgs\/\(.*\)\/$/\1/g'
python2/python/2.7.13/20170523194107

Adding schema to sd-cmd will also fixes this problem but I cannot imagine the use case of mode: local.

Do you have any idea? @stjohnjohnson

[stjohnjohnson]

@tk3fftk Local mode would be if the package wasn't published to the registry. Imagine if there was a shared folder of .hart files (because Depot is so difficult to setup). The intent would be to use that.

Interesting catch. I would say, if we wanted to support local mode, we would need to extend the configuration to replace package with path for local mode. That way we could still use package for referring to the package when executing.

[catto]

A path for logs has been shorten: https://github.com/screwdriver-cd/screwdriver/issues/677#issuecomment-367129362
https://github.com/screwdriver-cd/sd-cmd/pull/17

[catto]

Habitat type has been implemented and now we can use it with launcher v4.0.97 or later.
Currently it only supports habitat with remote repository, not local mode.

[jithine]

@tk3fftk Thanks for this https://github.com/screwdriver-cd/sd-cmd/pull/25 Could also update the docs https://docs.screwdriver.cd/user-guide/commands#creating-a-command with this information ?

[jithine]

Provide option for using alternative sources for hab packages (like artifactory)

@catto @tk3fftk could you add the docs for this functionality in guide if this is completed?

[tk3fftk]

@jithin1987 We completed the feature of using a local habitat package (.hart file). We also added
to the docs about the feature. But other sources (like artifactory) cannot be available.
@catto Do you have any plan?

[jithine]

Perhaps we could leave it as enhancement and address in a different issue ?

[catto]

@jithin1987 That's reasonable. I do not have a plan to implement the feature mentioned yet, so I am closing this and making another issue for the second pass.

[catto]

Closing via #1128