Azure-pipelines-agent: Variable number of step in templates

[4c74356b41]

Agent Version and Platform

Own agents, can upgrade if necessary, but this question predates running builds

VSTS Type and Version

VSTS; http://m-kopa.visualstudio.com

What's not working?

I cannot find information (and\or understand myself) if it is possible to do iteration over input in templates

Additional information

I want to create a templated build:

resources:
  repositories:
  - repository: infrastructure
    type: git
    name: xxx

phases:
- template: vsts/yaml/compile.yml@infrastructure
  parameters:
    packages:
      - package1
      - package2
      - package3

and in compile.yml i'd like to have something like this (jinja syntax):

steps:
- task: blablabla
  ...
  [
    {% for package in parameters.packages %}
- task: octopusdeploy.octopus-deploy-build-release-tasks.octopus-pack.OctopusPack@2
  displayName: "Package {{ package }}"
  inputs:
    other inputs with variable name as well
    {% endfor %}
  ]

if this is not possible, how do i template similar builds where I need to package only some of the stuff? like 1 project from solution, 2 projects, 3 projects?

[4c74356b41]

@ericsciple Hey, no pressure, but can you provide any thoughts? I'd like to work on this during the weekend :) I mean this is either possible or not (at this point in time), if its not, please let me know, I'll work around somehow.

Thanks a ton!

[ericsciple]

We dont have any looping constructs during template expansion today. The workaround today would be to break it into more granular templates (pushes the burden to the template consumer).

We discussed adding a looping construct when we initially added template expansion, but haven't finalized the decision yet.

There are other ways we are also considering solving the same problem. For example, allowing users to express the job itself (the steps) as code instead of a list declared in yaml. There is another alternative similar to this, where you blend the two, list of steps and coded steps mixed together.

Basically, we haven't added a looping construct yet, but are weighing different ways of solving the that problem.

[ericsciple]

In your case it sounds like it's within a job that you want to loop.

There are also cases for looping outside the job. So I'm wondering whether we should go ahead and add a template-expansion-time looping construct. The one thing that makes me hesitant is whether we want tighter control on looping outside of the job (e.g. a job behavior that gets applied at runtime, versus a full blown looping construct at template expansion time).

[4c74356b41]

ok thanks. valueable input. leads to another question, can I mix steps with steps from template? @ericsciple because right now I'm consuming phases, how would I got more granular than that?

as for the looping, i think you want to go for less control, that way you give more control to the end user and its his responsibility to produce proper resulting yaml (which is a lot less frustrating, because when you know this loop works, but the parser rejects it - a lot of frustration). But that's just my opinion.

thanks a lot.

[4c74356b41]

ok, it appears I can still do what I want to do, not as elegant as I wanted, but working:

parameters:
  stepsIncluded: 'path\to.yml' # this is being defined higher, this is just an example.

phases:
- phase: CompilePhase
  queue:
    name: default
  steps:
  - task: gittools.gitversion.gitversion-task.GitVersion@3
    displayName: GitVersion
  - template: ${{ parameters.stepsIncluded }}
  - task: PublishBuildArtifacts@1
    displayName: Publish Artifact
    inputs:
      PathtoPublish: $(build.artifactstagingdirectory)
      ArtifactName: drop 
    condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI','Manual'))
    enabled: false

[ericsciple]

Step templates can have parameters too. Something like this should work, if you wanted to pass a parameter through multiple layers:

# entry.yml
phases:
- template: phase.yml
  parameters:
    foo: asdf

# phase.yml
phases:
- phase: myPhase
  steps:
  - template: steps.yml
    parameters:
      bar: ${{ parameters.foo }}
  - script: echo hi

# steps.yml
steps:
- script: ${{ format('echo {0}', parameters.bar }}

[4c74356b41]

ok, i forgot my yaml definition calling these phases is in another repo and forgot to edit that (was editing the repo that holds all the yaml definitions). solution above works (my solution, your probably work as well, but i didnt test it as i found what was wrong with mine) and is fine with me for the moment.

shall i close this or leave this open to track loops in yaml? @ericsciple

thanks a lot!

[TingluoHuang]

We have created a new repository for all YAML related issues, please move the current issue to there.
https://github.com/Microsoft/azure-pipelines-yaml

[4c74356b41]

@TingluoHuang @ericsciple any eta on release pipelines with yaml, it q4 already...

[ericsciple]

@vtbassmatt ?

[vtbassmatt]

(edit: whoops, reading comprehension for the win)

To the original point of this thread: we have a looping construct coming called each. We'd love to have your input on the design before we finalize it.

To the question about releases in YAML: it's on the backlog, no specific timeframe to announce.