Arcade: Plan to Improve Arcade Reliability

[missymessa]

Short Term

• [ ] Disable automatic Arcade promotion to Latest after Arcade-Validation passes.
• [ ] Manually run Arcade builds through a set of repositories and manually promote Arcade if all looks good. (May need to do this daily or every other day).
• [ ] Determine telemetry needed to collect to keep a pulse on Arcade's reliability. Add any telemetry that may be missing; create reports to view the data collected.

Mid Term

• [ ] Have an automated way to include or remove repositories to validate Arcade as Arcade-Validation does.

Long Term

• [ ] Improve Arcade-Validation's robustness of validation coverage so that it better mirrors our customers' use cases and repositories.

[missymessa]

/cc @markwilkie @mmitche

[mmitche]

I still cannot find my silly comment.

[riarenas]

Not sure it's silly, but this comment seems super relevant to this:

https://github.com/dotnet/arcade/issues/4748#issuecomment-582003555

[mmitche]

Yes! That's the one.

[missymessa]

This has been broken down into smaller tasks. Closing.