Arcade: Improve documentation for arcade-services verification

[mmitche]

The documentation for the process by which arcade-services can be verified isn't really clear. We should improve this documentation. Specifically I am looking for:

• [x] How different types of changes should be verified
  
  
  
  • [x] Deployment changes that cannot be verified locally should run deployments against int and then roll back the cluster
  
  
  • [x] Functionality changes should run the unit tests and the scenario tests (introducing new tests as necessary)
    
    ~- [ ] How to run scenario tests locally~ Moved to MORE Validation Epic - https://github.com/dotnet/core-eng/issues/9103
    
    ~- [ ] How to debug the cluster~ Moved to Deployment Phase 2 Epic - https://github.com/dotnet/core-eng/issues/9106
  
  
• [x] How to roll out a private branch to the int cluster.

[mmitche]

/fyi @sunandabalu @mjanecke

Not expecting you to do this, just a heads up that this appears to be a gap in our arcade-services verification and deployment process.

[mmitche]

/cc @ChadNedzlek @riarenas

[ChadNedzlek]

I think we probably need another cluster rather than let private branches stomp all over int. For arcade-services that shouldn't be too expensive (we don't need a ton of pools of machines behind them), and it would help keep the int environment from getting broken in case someone's PR makes stateful (and irreversable) changes.

[mmitche]

I also think we should take the opportunity and bring up pre-production cluster in preparation for your and the TLS changes going out. I am not confident at the moment that they will roll out seamlessly.

[markwilkie]

pre-production might be the right answer - but I'd also like to discuss this more first.

[mmitche]

Pre-production specifically to guard against the changes that would be rolled out in the next iteration. The deployment process changed quite a bit and there were quite a few commits that did not roll out properly (in fact it's still broken because of the the intermediate commits managed to put it in a bad state: #4356)

[missymessa]

/cc @sunandabalu to manage.

[sunandabalu]

https://github.com/dotnet/arcade-services/commit/50a6fae92044adeb6a11555b5664e47957352b36

[missymessa]

Documentation complete. Closing.