Azure-devops-docs: How to share files between stages?

[ebashmakov]

Let's say I have Build, Test, Publish, and Deploy stages:

stages:
- stage: Build
  jobs:
  - job: BuildJob
    steps:
    - script: dotnet build
- stage: Test
  jobs:
  - job: TestJob
    steps:
    - script: dotnet test --no-build
- stage: Publish
  jobs:
  - job: PublishJob
    steps:
    - script: dotnet publish src/myProject.csproj --no-build
- stage: Deploy
  jobs:
  - job: DeployJob
    steps:
    - script: az functionapp deployment source config-zip -g myResourceGroup -n myWebApp --src myProject.zip

How can I share the job results (artifacts) between stages? So for example the "Deploy" stage will have "myProject.zip" file from the "Publish" stage.
Do I have to publish artifacts after each stage? Is it possible to share files (pipeline environment) between stages automatically?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 4266f72c-c774-0046-4593-d01eb775d3c3
• Version Independent ID: f20827aa-a6c5-96a8-5969-e576ffbc2e38
• Content: Stages in Azure Pipelines - Azure Pipelines
• Content Source: docs/pipelines/process/stages.md
• Product: devops
• Technology: devops-cicd
• GitHub Login: @vijayma
• Microsoft Alias: vijayma

[shashban]

Stages are logical an independent parts of a pipeline. Publishing artifacts is the supported means to transfer files between stages.

[ebashmakov]

@shashban thanks for the reply! But they're called "Stages", they're kinda dependent by definition. That's super weird. All the provided examples do not make sense for me then - I have to copy/publish and download artifacts over and over again. It seems like a waste of time and resources. Maybe I understand it wrong but what would be a real use-case when you don't need to share data between stages?

[sawatsky]

This would have been really nice. I was hoping to have Build, Compliance, Test, and Publish stages. Even the documentation says this should be possible when it says, "Stages are the major divisions in a pipeline: 'build this app', 'run these tests', and 'deploy to pre-production' are good examples of stages," but I don't see the point if I have to either: a) re-build in every stage, or b) publish and download artifacts in every stage.

I guess I'll just use one big stage then.

[shashban]

The goal is precisely to build in one stage and publish artifacts from the stage, that can be used in multiple other stages. Each job in the stage runs on separate machines (agents) and downloading the required artifacts is a pre-requisite for execution. Publishing artifact is a means to telling the system that the information will be reused.
We understand that no deployment can happen without downloading the artifacts. We'll enable automatic download of all published artifacts in deployment jobs.
You could build multiple packages in your pipeline and have separate test stages for each of them. In those cases, you only need a part of artifacts in the test stage and downloading everything is an overkill. Keeping that in mind, we have a different behavior for normal jobs.
Needless to say, you can always override the default behavior using checkout and download constructs.

As for whether to have one stage or multiple, the reason for separating into stages is more of manageability and control, not so much of a workflow. You get value adds like approvals, checks, execution sequences and status reporting for each stage, in addition to execution of steps in a particular order.

[gtsiou]

The worst thing is that the files are not even shared between jobs of the same stage. So in the same stage, lets say "Build" you have to do all the steps in one job, otherwise you waste time on publishing and downloading artifacts.

Like you say @shashban, the stages are already a logical separator running individually, why do jobs have to be the same and run on separate agents? So now Stages provide clarity from a UI standpoint, but jobs provide nothing useful.

I think this is not intuitive and completely nullifies the potential gains from splitting the work in jobs (and stages).

That plus each job (although you told it to download a published artifact) does checkout of the initial repo over and over again, makes the whole experience quite annoying, and forces us to have single stage / single job setup.

[vijayma]

You have flexibility in order to use steps, jobs, or stages. Steps share the most - same agent, files, etc. Jobs and stages are higher level constructs that are intended for various other scenarios. For instance, if you want to build once and test on multiple platforms, you need these constructs. Or, if you need to build on multiple platforms, then you need these constructs as you need a different agent for each of the platforms. Closing this issue in the docs repo.

[K41eb]

So basically, the only way for stages to share files, is to upload / download them as artifacts like you would do between pipelines? That's very disappointing, and prohibiting.

The app I currently work on generates a 1.7G node_modules folder. I can not afford to upload / download it over and over again. I was planning on using stages and jobs to parallelize some of the work but the precious minutes this would have saved me are wasted away (multiple times over) by the upload / download process. So this forces me, like other people in this thread, not to use that feature at all.

Coming from Gitlab where sharing that kind of "artifact" costs next to nothing, I do not understand why it's such an issue with Azure.

I understand that each stage, and job, runs on a separate machine, but one could have expected that all those machines ran on a single "host" that could easily copy / paste data between volumes, or share said volumes between "guests".

[tuomaskujala]

I think this is a major issue. I recently got started with Azure Devops with a background in CircleCI/Google Cloud Builds, and its very confusing that such a default feature as effortless reuse of code between jobs is missing. Seems highly overkill that, for example, a two-job pipeline such as Checkout-Test requires you to package your code to an artifact.

Right now users are left with a lose-lose decision: either spend extra time working with artifacts, or stick to single stage/job. This is bound to drive new users to competing services, where the same functionality often requires little to no effort at all.