Arcade: Plan to Re-work channel implementation (YAML) creation in Arcade

[JohnTortugo]

Currently whenever we create a new branch in Maestro we need to manually create a PR in Arcade to add a new YAML file representing that channel. This is a error prone process and create a bunch of duplicated files. The channels' YAML are _very_ similar, therefore we can simplify this process by creating a _Channel Implementation Template_ that could be instantiated whenever a new channel should be created.

/cc @mmitche @riarenas

Here are the goals as I see them:

• [ ] UI should not be cluttered
• [x] It must be possible to assign builds after the fact
• [ ] Changes must be back-portable to 3.x
• [ ] Can still make more "integrated" changes (changes that need to match with the BAR publishing steps, like manifest changes).
• [ ] It should not require an arcade rollout to introduce new channels. This is incredibly expensive today. Case and point: Blazor feature branch for 3.1 needs a new channel for feature work. This requires a servicing change and rollout of arcade to that branch, which really stinks.

[mmitche]

@JohnTortugo Possibilities to consider:

• Move channel templates into a separate repo (or arcade branches) and reference as a resource. Would still differ between release/3.x and master versions of arcade. Introducing a new channel would be a single check-in, and immediately available. The interface between publishing and the repo generated info (manifests and artifact drops is pretty stable now)
• Change templates that only differ by feed names to be a single stage and multiplex the feeds and other info using some kind of map at pipeline run time. Reduce clutter in the UI.
• Move publishing into another pipeline altogether so that channel assignment after the fact is possible again.,
• Keep ability to add specific channels in post-build.yml as we do today for publishng that require changes in both manifests/sdk tasks as well as the publishing options.

[riarenas]

Another posibility:
We currently specify in the channel templates the feeds that each channel should publish, but this seems like information that would also make sense to keep in BAR about the channel itself.

• The BAR channels now keep the list of feeds that shipping/non-shipping packages should go to, which is returned by darc-get-channels --include-feeds (or similar)
• The publishing stage during the build pipeline would look at the default channels just as it does today, and when it gets the channel information, it will get the whole list of feeds that it needs to publish to.
• We can have a build pipeline that takes a list of desired channels and a bar build id as input, and we could just run the publishing stage.

As an added advantage, we could make it so eventually, darc-add-build-to-channel also publishes the assets to all the feeds it needs to publish to, which will give us channel assignment after the fact without any build or release pipelines running.

[JohnTortugo]

Much has changed since we posted the previous messages in this issue. I'll post below some of the major things that changed in the meantime and the current issues we are facing (please complement if I forgot any). In a subsequent message I'll post a proposal to approach the problems.

Recently we added:

• Ability to promote builds to any channel after the fact.
• Darc add-build-to-channel now is full featured for publishing builds (including assets + validations) after the fact.

Some issues that we currently face:

• Need to merge changes in Arcade and wait for them to propagate whenever we introduce a new channel.
• Large 'code duplication', tedious and error-prone process whenever we need to add a new channel. I.e., the bottom part of this file.
• UI is immensely cluttered.
• People are confused because the stages are 'activated' but they don't publish anything.
• Symbol publishing happens in more than one stage.
• If the build doesn't have a default channel it's not very clear that nothing happened, since the stages are still activated. This causes a lot of confusion.

[JohnTortugo]

In the message below I tried to merge the proposals that were posted here and the things that we discussed on Teams.

• Instead of having one stage per channel we'll change the infra to have just one publishing stage for the build. The stage will have at least two jobs: 1) publishing symbols and 2) publish packages
  
  
  
  • Publish symbols: is basically what we have today, but will it'll happen only once for the whole build
  
  
  • Publish packages: I explain in more detail below.
  
  

Create a new task in the Tasks.Feed package that will receive a list of parameters similar to what this task receive. Then it'll proceed to:

• Get the list of all default channels for the build and
• Perform the publishing operation to each channel

This change will require us to modify BAR to add the feeds as an associated entity to the Channel(s).

It will probably be a disruptive change but I think we can do this without requiring any changes on the customers build-definitions, because the changes will be in Arcade post-build scripts.

After this change:

• We'll be able to introduce new channels without doing changes in Arcade
• AzDO Build UI won't be cluttered
• We can back port this to 3.x branches
• Since we have a single publishing stage it'll be easier for users see related errors/warnings/logs
• No YAML duplication, simplification of post-build YAMLs since we won't need to pass some parameters around, instantiate channels templates, etc.

What do you think?

/cc @mmitche @riarenas @markwilkie @chcosta

[markwilkie]

You mentioned that this will likely be disruptive, in what way are you thinking?

[mmitche]

I love the idea of removing the channel definitions from arcade. That would be really fantastic. I do have some questions though:

• Today our manifests have been really consistent over time (at least from the late 3.0 timeframe), but the interpretation of those manifests how and where the artifacts in those manifests varies quite a bit. For instance, the 3.x publishing knows nothing about aka.ms links, whereas the 5.x publishing knows how to do that. Eventually 5.x will know nothing about blob feeds, but 3.x will probably stay that way.
• How would we handle the varied parameters between implementations of the publishing infra?
• How would we handle addition of new parameters or subtraction of existing ones?

What if we changed things more radically:

• Unify publishing from 3.x onward onto a single branch of arcade or another repo (arcade-publishing?).
• Unify to a single stage like you say above
• Templates point to 'live' master versions of arcade/arcade-publishing (or maybe a tag that we move around based on the arcade-validation results) so that rollout of new publishing infra is just a matter of moving the tag.
• Add a manifest version number and additional info into the manifest such that the post build yaml templates really take no parameters, only the input artifacts. For example, the "PublishInstallersAndChecksums" parameter would move into the manifest. Today I'd say we have two versions of the manifest: 1 and 2. 1 is 3.x publishing and 2 is 5.x. When we make breaking changes we update the manifest version number. The version number is embedded by the build asset registry publishing, so if we make changes to publishing that require changes in the repo, you'd need to get and react to the corresponding arcade update in order to generate a new manifest.
• Arcade's master branch (or a separate repo) implements all publishing for all releases. The manifest version + the target channels identify how the manifest should be interpreted, and the parameters for each target channel are held in arcade, probably in SetupTargetFeeds.targets or a shared props file.
• You could handle different manifest versions by have a single task that implements each version of manifest publishing, and SetupTargetFeeds.targets is split out into multiple files, one for each manifest version. So maybe this looks like:
• ChannelInfo.props - Shared info about channels. For instance, what feeds are used, etc. Different versions of the manifest could interpret this data differently. For instance, manifest version 1 (for 3.x) publishing might ignore aka.ms data, while version 2 (for 5.x) would ignore blob feed targets.
  
  
  
  • SetupTargetFeeds.v1.targets - References ChannelInfo.props, sets up publishing feed info for the v1 (3.x) publishing
  
  
  • SetupTargetFeeds.v2.targets - References ChannelInfo.props, sets up publishing feed info for the v2 (5) publishing. Includes aka.ms links, no blob feed, etc.
  
  

This would also put us in a position to move publishing into a separate pipeline altogether at some point, like @jaredpar wants, but would also be more robust IMO.

The upshot of this would be that you would still be required to check into arcade or another repo to update channel info, but:

• It wouldn't require a rollout to every repo
• It would get tested in arcade-validation first. We could implement tests for publishing based on a variety of input manifests.

Lastly, I think we should move the symbol publishing into the main publishing task, rather than having it be a separate task. Performance is very important here, so if this ends up adversely impacting publishing perf it should remain split out.

[JohnTortugo]

You mentioned that this will likely be disruptive, in what way are you thinking?

@markwilkie - Since it's a change in a central part of publishing any error doing it will certainly cause considerably noise. But if we do it carefully nothing should break as there is no change required from the customer side.

[JohnTortugo]

• How would we handle addition of new parameters or subtraction of existing ones?

I see that and I was concerned with this too. Please see other comment below.

• Templates point to 'live' master versions of arcade/arcade-publishing (or maybe a tag that we move around based on the arcade-validation results) so that rollout of new publishing infra is just a matter of moving the tag.

You want to introduce this new branch (or repo) so that we can use the templates from that branch as a resource in other pipelines, correct? I imagine that what motivates that is that currently for every channel that we create we need to do changes in the repository. But if that changes become much less often (i.e., not have channels in YAML) then I'm not quite convinced it would justify creating a new repository/branch for this.

I think we need to simplify the overall interface between the components of publishing so that it gets easier to keep this backward compatible or at least easy to migrate to servicing branches. Currently, the publishing flow for a single build is more or less like this:

Perhaps we could simplify it to this:

Which has the following [big] changes:

• The only interface of Arcade.SDK and post-build stages is the AzDO artifact folders "/publish/shipping" and "/publish/non-shipping". Everything on that folder needs to be published somewhere everything else is ignored by the post-build stage(s).
• There is no build manifest. All information about a build should be in BAR.
  
  
  
  • If the user needs to see a manifest we can add something like darc get-build-manifest -format xml -build 123.
  
  
  • The post-build publishing task infers what kind and where an asset should be placed based on the _file extension_ and target channel.
  
  

My thought process here was that simplifying the interface between the Arcade SDK and the post-build stages we can have more confidence that changes that we do in the publishing infra can be back portable. For instance, recently we were 'challenged' because we needed to publish builds produced with Arcade SDK 3.x using the infra that we now have for .Net 5.

This would also put us in a position to move publishing into a separate pipeline altogether...

I think after recent changes we can do that today. Instead of including the post-build.yml stage at the end of his build definition he can do a call to darc add-build-to-channel and it should work the same.

[mmitche]

You want to introduce this new branch (or repo) so that we can use the templates from that branch as a resource in other pipelines, correct? I imagine that what motivates that is that currently for every channel that we create we need to do changes in the repository. But if that changes become much less often (i.e., not have channels in YAML) then I'm not quite convinced it would justify creating a new repository/branch for this.

Yeah, use the templates as resource. The problem here is that the number of parameters that are passed implied by the channel is non-trivial today. It varies based on a wide variety of factors:

• Is the build internal
• Is the build stable
• What, if any are the aka.ms links that we want to generate
• Where do symbols, packages, and blobs go.

You can move channels out of YAML, but you're not changing the complexity. You're just moving it to a different spot (in a DB) which is not really testable.

We could get rid of the manifest, but we'd need to future proof things. How do we move onto a unified publishing infra that will recognize that publishing will change over time and that there may be some changes that will not be able to be applied to older versions of the product. To me that says that we need to specify what "version" the source of the publishing is on (via a manifest perhaps). We have done this in the past btw, we just didn't call it a version. We had PublishUsingPipelines, PublishInstallersAndChecksums, etc. Those were all versions of publishing, though we just made a bunch of booleans instead of a versioned manifest.

We could put the manifest in BAR itself as part of the upload, but it really needs to be in a form that allows for a great deal of flexibility over time. We cannot assume that the same set of fixed parameters will be in use. A file based manifest gives a lot of flexibilty there.

My thought process here was that simplifying the interface between the Arcade SDK and the post-build stages we can have more confidence that changes that we do in the publishing infra can be back portable. For instance, recently we were 'challenged' because we needed to publish builds produced with Arcade SDK 3.x using the infra that we now have for .Net 5.

Yes, I do think we should simpify this. In my approach above (versioned manifest, props + target file containing parameters, and a shared task that supports all manifest versions), the same infra would end up used for every release of .net core, even though there might be slight variations in implementation based on the manifest version.

We could 'simplify' that and remove the manifest altogether, but I think no matter what you will need to version the build info in BAR somehow. Otherwise rollout of changes to publishing will be extremely risky,

Let's say in BAR you add a manifest version table. You start with two versions, 1 and 2. Then in BAR you invent a json blob of some sort for each channel. The channel data. It contains info about target feeds, aka.ms links, etc. You then have versions of that data for every manifest version. SO when you add a new manifest version, you add a new set of parameters for each channel.

Then we remove the publish to build asset registry job, move it into a new stage, and remove the manifest. The publish to BAR job publishes more info than before. It now publishes info about public/private builds, stable vs. non stable, and the input data version.

Then the publishing stage or job will read the BAR data, determine how it should interpret the input artifacts, and pass the corresponding set of configuration data for the target channels to the publishing task.

[jaredpar]

To me that says that we need to specify what "version" the source of the publishing is on (via a manifest perhaps)

+100

The manifest version + the target channels identify how the manifest should be interpreted, and the parameters for each target channel are held in arcade, probably in SetupTargetFeeds.targets or a shared props file.

Like this a lot.

so that rollout of new publishing infra is just a matter of moving the tag.

What is a "tag" here?

[mmitche]

What is a "tag" here?

In this case a git tag or a branch or something along those lines. Basically you can set up a pipeline to point to templates as a resource, and those templates could come from another repo at a specific commit, branch, or tag. We want to validate that whatever changes we make to those templates work, so you wouldn't want to reference arcade@master directly. If we broke arcade with a check-in you'd immediately break anyone referencing it. Instead, you'd move a tag when you validated publishing.

This is basically the same thing as having a separate repo for publishing and merging master into some target production branch automatically every time you validated things worked. Automatic deployment. The difference here is that you could use arcade as the publishing repo.

[JohnTortugo]

Quick update since my write up on a proposal is taking longer than I expected: _I talked with @mmitche and we came up with a plan; I'm writing it down and I'll post it here or in a PR probably today by EOD._

[JohnTortugo]

This is the plan that we currently have:

The build UI and the publishing flow

• YAML publishing stages will be converted to a single stage with a single publishing job.
• Maestro Channels won't have an implementation in YAML.
• There will be a publishing task in Tasks.Feed that iterate over the default channels for the build and does the publishing work.
• The parameters to a channel will be passed to the publishing task directly or stored in the file below.
• There will be a ChannelInfo.props file with shared information concerning channels. For instance, target feeds, aka.MS paths, etc. _I'm still deciding if we really need this file._ @mmitche are you OK with adding this information to the Channels table in BAR?

The customer side

• Builds will be able to opt to use the _build promotion pipeline_ (BPP) to promote just created builds or inline the single publishing stage (as is today) in their builds.
• To be discussed: We still need to figure out how to handle publishing failures in the BPP. I.e., who/what will be responsible for monitoring the promotion builds and alerting about failures.

The manifest

• We'll keep the Build Manifest and we'll add a version attribute to it's root element.
• Arcade will store the BPP yamls in a separate branch (e.g., a /publishing branch).
• Arcade-validation will validate changes in the publishing YAMLs and copy them from the Arcade working branch to the Arcade/publishing branch.
• The BPP implementation will be from the Arcade/publishing branch.

How will be the workflow for common scenarios

Adding a new channel

• Today:

  1. Create channel in BAR.
  2. Duplicate a few lines of YAML in one of the Arcade post-build files.
  3. Create PR in Arcade and wait for it to be merged and files propagated by Maestro.
  4. One more stage would show up in every repository build UI.

• Proposed:

  1. We'll just need to add it to BAR and perhaps update the ChannelsInfo.props file in Arcade. No need for YAML. No additional stage in Build UI.
  2. May need to create a PR to update ChannelsInfo.props.

Upgrading the publishing infra. Scenario 1.

Let's say that we want to publish to publish all .nupkgs to an additional feed.

• Today:

  1. Patch the publishing task in Tasks.Feed to recognize a new parameter+key for the new feed & be able to publish to it.
  2. Create+merge PR with changes to Tasks.Feed
  3. Wait for every repo that wants to publish to new feed to receive updated Tasks.Feed + Arcade.SDK
  4. Now that we have a task that recognizes the new parameter we need to go to every channel implementation (in YAML) and add the _new parameter_ (we already have a lot of parameters) containing path of the new feed + key to the feed.
  5. Create PR & wait for it to be propagated.

• Proposed:

  1. Add new channel in BAR and perhaps update the ChannelsInfo.props file in Arcade.
  2. May need to create a PR to update ChannelsInfo.props.

Upgrading the publishing infra. Scenario 2.

Lets say that we want to sign files in the same job that does the publishing. To do that we'll store signing information in the build manifest together with the assets information. I'll assume the info is already in the manifest.

• Today:

  1. Modify the existing publishing task in Tasks.Feed to be able to parse the new manifest, call the signing server and only then publish the assets.
  2. Create & merge PR with the above changes.
  3. Once that change is propagated by Maestro++ every repo will be using the new version of the task, even builds that don't want to sign+publish in the same step. The potential for breaking builds is high.

• Proposed:

  1. Modify the existing publishing task in Tasks.Feed to be able to parse a new version of manifest that contains signing information, call the signing server and only then publish the assets.
  2. Create & merge PR with the above changes.
  3. Repos that wish to sign+publish in the same step then upgrades their build definition to produce the new version of the manifest.

[mmitche]

• There will be a publishing task in Tasks.Feed that iterate over the default channels for the build and does the publishing work.

I would say this should be a set of input channels, and we call the task with the default channels (e.g. maybe with another task that gathers the default target channels). This keeps the task generic.

[mmitche]

• There will be a ChannelInfo.props file with shared information concerning channels. For instance, target feeds, aka.MS paths, etc. _I'm still deciding if we really need this file._ @mmitche are you OK with adding this information to the Channels table in BAR?

As long as its generic and versioned, yes. Meaning that:

• If you produce version 1 of a manifest and want to publish to channel A, the publishing goes and looks up channel A at version 1, gets the blob of config info, and uses that. That would mean we could change what the input parameters are over time without bloating the overall config.

[mmitche]

• Arcade will store the BPP yamls in a separate branch (e.g., a /publishing branch).

What do you think about separate branch vs. separate repo? What happens if you need a new arcade to update publishing? A couple options I can think of:

• We have a subscription which updates arcade in arcade from master->publishing
• We put publishing in a separate repo altogether

[mmitche]

• Arcade will store the BPP yamls in a separate branch (e.g., a /publishing branch).

What do you think about separate branch vs. separate repo? What happens if you need a new arcade to update publishing? A couple options I can think of:

• We have a subscription which updates arcade in arcade from master->publishing
• We put publishing in a separate repo altogether

Ahh I see...I think arcade-valdiation could do a dependency update, including a new arcade, to the publishing branch.

[mmitche]

• Add new channel in BAR and perhaps update the ChannelsInfo.props file in Arcade.
• May need to create a PR to update ChannelsInfo.props.

This doesn't seem to quite match up with the scenario.

[mmitche]

1. Repos that wish to sign+publish in the same step then upgrades their build definition to produce the new version of the manifest.

And they'd get this new version by pulling a new version of arcade, which would update the publish to BAR task.

[mmitche]

I am liking this plan quite a bit. /fyi @jaredpar

[JohnTortugo]

• There will be a publishing task in Tasks.Feed that iterate over the default channels for the build and does the publishing work.

I would say this should be a set of input channels, and we call the task with the default channels (e.g. maybe with another task that gathers the default target channels). This keeps the task generic.

Sounds good. The promotion pipeline already works like that.

And they'd get this new version by pulling a new version of arcade, which would update the publish to BAR task.

Correct.

• Arcade will store the BPP yamls in a separate branch (e.g., a /publishing branch).

What do you think about separate branch vs. separate repo? What happens if you need a new arcade to update publishing? A couple options I can think of:

• We have a subscription which updates arcade in arcade from master->publishing
• We put publishing in a separate repo altogether

Ahh I see...I think arcade-valdiation could do a dependency update, including a new arcade, to the publishing branch.

TBH my initial thought was that arcade-validation would just make a push to the "/publishing" branch once it decided that the changes were good. Having a subscription looks better, though.

• If you produce version 1 of a manifest and want to publish to channel A, the publishing goes and looks up channel A at version 1, gets the blob of config info, and uses that. That would mean we could change what the input parameters are over time without bloating the overall config.

I see what you're getting at here - same as before, the .props file is much more flexible / "future safe" and easy to version. I'd say let's stick with a .props file then.

I've some ideas about how arcade-validation would perform the validation of the publishing changes. I'll write a few paragraphs and post here.

[riarenas]

I like this plan. Thanks for putting it all together @JohnTortugo

[JohnTortugo]

So, here is my thoughts on what we need to change in arcade-validation scripts to validate the publishing infra.

What we currently have: Arcade-validation creates a branch of a customer repo (let's say Roslyn) and uses darc to update that branch with latest changes from Arcade/master repository. Arcade-validation sets up a default-channel for the customer repo and triggers a build. If the build passes then Arcade build is "promoted" otherwise error is reported.

Why it's not sufficient to test the new infra: With the plan proposed in this issue customer repos will use the build promotion pipeline or import templates from Arcade to implement the post-build stages. In both cases the source files come from Arcade/publishing branch, not Arcade master. Therefore, if we just trigger a build of the customer repo it'll not test the files in Arcade/publishing but in Arcade/master.

What I propose: That we change Arcade-validation to instead of setting up a default-branch for the customer repo it'll instead use darc add-build-to-channel --build xyz --source-branch Arcade/master to promote the build using the sources from Arcade _master_ branch.

If arcade-validation build succeeds then we update Arcade/publishing branch with the changes merged in master. I think just a direct push to the publishing branch (instead of a PR) will speedup things.

[mmitche]

Therefore, if we just trigger a build of the customer repo it'll not test the files in Arcade/publishing but in Arcade/master.

I think you meant the other way around.

Otherwise looks fine. I am a bit concerned that we're basically using another branch of arcade like a separate repo. The version of arcade used for publishing will be the one in global.json in the publishing branch, so it may not be obvious to some people what files are "active" in the publishing branch and which files are basically just there because it's the arcade repo.

[JohnTortugo]

Therefore, if we just trigger a build of the customer repo it'll not test the files in Arcade/publishing but in Arcade/master.

I think you meant the other way around.

Yeap. It's the other way around.

Otherwise looks fine. I am a bit concerned that we're basically using another branch of arcade like a separate repo. The version of arcade used for publishing will be the one in global.json in the publishing branch, so it may not be obvious to some people what files are "active" in the publishing branch and which files are basically just there because it's the arcade repo.

Sounds good if we start with the branch approach and then we keep talking along the way to see if things are still looking good? TBH.. it just seems that creating a separate repo just to store the publishing files seems an over complication.. but I'm probably not seeing the whole thing as clear as you are.

[JohnTortugo]

I'll start breaking this down to issues then. Thank you all for the comments.