Screwdriver: Build artifacts between jobs

[catto]

I want to build a c++ product, create a package for it and publish it using Screwdriver.

Screwdriver runs main job when Pull Request is opened or updated, so I should specify build and package steps in main job, and a publish step in publish job.

But it won't work. Build artifacts like *.so won't be passed into the next job such as publish.
It works when I also specify build step in publish job. I think it might be a waste of time.

main job (make, test, package) => publish (make, package, publish)

I think we could access the artifacts in publish job in some way if we put these artifacts into "artifacts" directory. It's not easy though.
Is there any way to achieve the subject easily without building the product twice?

[d2lam]

I agree. Currently, there is no way to do this. Or something like npm install needs to be called twice, which is super annoying and slow.

[stjohnjohnson]

This is a great idea. I had two thoughts on ways to implement this.

1. We add a new feature to meta meta get_artifact that can retrieve a file from builds in the current event. Optionally we could rename it artifact get but I could be sold on either one. This gives us the opportunity to convert artifacts into a global namespace like we did with metadata (or we don't have to).
2. On success of the main job, we store the workspace and use it to preload the subsequent jobs in the pipeline.

[catto]

No.1 is simpler and easier to understand than No.2. The only disadvantage for No.1 that I think is to use artifacts directory. For the case of Node.js, every main job has to include a step which copies node_modules directory to artifacts directory. For Java, it would be "target" directory.

I think No.2 is so convenient since we can configure subsequent jobs as if we are using the same workspace as main one.
My concern for No.2 is amount of network traffic and time for storing and preloading. Some repositories are huge, so it would be nice to have an ability to disable or enable it via annotations.

We can implement both 1 and 2 separately. No.1 is suitable for pulling artifacts from previous builds like coverage data or screenshots of an E2E test, No.2 is for reusing whole workspace of main job.

[marian-r]

This would be really useful :)

[drewfish]

Carrying the whole workspace might not be appropriate if for example the template is different in different jobs. As well 2 might cause issues for "fan-in" jobs (requires has AND logic). 2 could instead just carry the artifacts directory to the next build, but that still might have issues for fan-in jobs.

I can see the use cases for both 1 and 2, so perhaps both could be supported opt-in. 1 is by-design opt-in. 2 could perhaps be opt-in by workspace: {previous job name} or something like that.

[matope]

Any progress on this issue? I believe this cool feature helps many users including me.

[catto]

@matope Sorry for keeping the issue open. This feature has been introduced as build cache via https://github.com/screwdriver-cd/screwdriver/issues/1257.
Please refer to https://docs.screwdriver.cd/user-guide/configuration/build-cache for more information.