Conda-forge.github.io: Document how to set up self-hosted Azure agents

[leofang]

Conda-Smithy is a core utility for Conda-Forge, but it can do things beyond CF's purposes, and I think we should get more undocumented Conda-Smithy capabilities documented somewhere. For examples, it allows setting up self-hosted agents for non-CF infrastructures, see https://github.com/conda-forge/conda-smithy/pull/1296.

For a sample config see https://github.com/nsls-ii-forge/cupy-feedstock/blob/e14c4c9ba6f75cde55ee3b6956968318b5526c05/conda-forge.yml#L15-L29. In this particular case, it allowed us (NSLS-II-Forge) to test GPU packages using our local machines through the same Azure CI.

[leofang]

cc @viniciusdc

[Aaishpra]

hey, @leofang are we going to add this information to this page.
Also, just a suggestion for a better introduction about conda-smithy "Conda-Smithy is the core utility for Conda-Forge, it contains a lot of tools for building and deploying continuous integration (CI) infrastructure on a variety of different platforms and architectures! It can also be used to set up self-hosted agents for non-CF infrastructures" just got this after reading the description of this issue.

[Aaishpra]

@viniciusdc @leofang i would like to take this up.

[viniciusdc]

hey, @leofang are we going to add this information to this page.
Also, just a suggestion for a better introduction about conda-smithy "Conda-Smithy is the core utility for Conda-Forge, it contains a lot of tools for building and deploying continuous integration (CI) infrastructure on a variety of different platforms and architectures! It can also be used to set up self-hosted agents for non-CF infrastructures" just got this after reading the description of this issue.

This might https://conda-forge.org/docs/maintainer/infrastructure.html?highlight=smith be better

[leofang]

You also need to add the configs relevant for setting this up to https://conda-forge.org/docs/maintainer/conda_forge_yml.html.

[PrernaSingh587]

Is this issue being tackled?

[Aaishpra]

Yes, it is, I am working on it.

[Aaishpra]

Hi @leofang @viniciusdc
I checked this PR about self-hosted Azure agents and found a lot of changes in files, and couldn't understand what configs are required to set up the self-hosted azure agents. Can any one of you help?

[leofang]

@Aaishpra Please check my issue description. I did link to a working example (although I commented it out).

[Aaishpra]

@leofang This is just my initial implementation about configs required for setting up self-hosted azure, I have added this to https://conda-forge.org/docs/maintainer/conda_forge_yml.html
Need your suggestion for improving this

Also, shall I remove the org name and project name from the code, and can you suggest some comments to be added to this.

[mariusvniekerk]

I would also recommend mentioning https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops for autoscaling agents that could be cheaper to run that permanently active agents

[leofang]

@Aaishpra Looks great overall, thank you for working on this. Could you please send a PR and ping me so that I can make some minor suggestions there? It's easier to suggest changes in a PR.

[Aaishpra]

Sure @leofang I have submitted the PR and have also added @mariusvniekerk suggestions into that.