Azure-devops-docs: GitHub Enterprise Connectivity

[chrpai]

We have a corp on-prem GitHub Enterprise that's not accessible from the internet. When I try to create the service connection I get:

Failed to query service connection API: https://github.somecompany.com/api/v3. Error Message: A task was canceled..
Note: Verification may fail if the URL is not accessible from Visual Studio Team Services. However, it might still be valid and accessible from the agent that executes the build or release.

Ok, I get that it'll be accessible from the agent but I can't get that far. I'm blocked when I try to create my pipeline as I can't browse my org/repos/branches. Is there a certain firewall rule (port 443 from some ip/subnet source?) that I'm supposed to request to be implemented? Do I have to stand up a Azure Devops Server in side the network to use all of this?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 72b90dd0-49f9-4ed4-3010-9a64dde6ae5a
• Version Independent ID: 6e679d50-4e94-eee3-8793-af884cc909ca
• Content: Service connections in Azure Pipelines and Team Foundation Server
• Content Source: docs/pipelines/library/service-endpoints.md
• Product: devops
• GitHub Login: @alexhomer1
• Microsoft Alias: ahomer

[alexhomer1]

Thank you for your feedback. I have passed this on to the product team for advice.

[chrpai]

I'm wondering how this could have ever worked? Perhaps MSFT tested hosting GitHub Enterprise in Azure? If the source IP for the API request isn't static, I'm wondering if some API gateway relay/proxy agent is needed to relay the interactions and get around the firewall.

[davidstaheli]

Hi @chrpai. Sorry for the trouble. Just to make sure I understand... The GitHub Enterprise server isn't accessible to the internet, but you want to set up an on-premises agent that runs builds triggered by Azure Pipelines. Is that right?

If so, creating the service connection to GitHub Enterprise won't work because Azure Pipelines uses it to access GitHub Enterprise over the internet. Azure Pipelines does this to create a webhook that lets it know when code changes and a CI build should be triggered. To address this, you can open an incoming firewall rule on port 443 for HTTPS. Make sure that GitHub Enterprise is secured with HTTPS and requires credentials.

If you aren't able to expose GitHub Enterprise to the internet on port 443, you can set up an on-premises agent that connects to Azure Pipelines and performs manually-triggered builds, but it won't be able to perform CI builds when code changes occur because Azure Pipelines won't be able to contact GitHub Enterprise to know when code changes occur.

Finally, you could work around this by installing Team Foundation Server 2018, a slightly older, on-premises version of Azure Pipelines. Since this runs behind your firewall, it will have full access to your GitHub Enterprise server and you won't need to use Azure Pipelines.

Will any of these options work for you?

Thanks,
David

[chrpai]

I can make a request with my firewall team for tcp/443 but they are going to want to know the source ip range. They aren't going to allow incoming 443 from the entire internet. Is there a list of virtual ip or subnets that Azure would be making these API calls from depending on which datacenter the Azure DevOps org is hosted in?

[chrpai]

If the previous request for source IP firewall rules isn't possible, it would be good if an on-prem build agent could be used as a proxy to make the magic happen. 1) Attempt API call and get time out 2) Send request for build agent to try 3) Build agent gets the request including the GitHub API URL, PAT and requested action and relays the call to the API and returns the response to the Azure DevOps API so you can browse repos/org/branch from Azure DevOps API WebUI. Same thing for enabling continous integration. On Prem Build Agent relays request to create a WebHook in GitHub Enterprise telling it to deliver the checkin message to the Azure DevOps API to enable CI builds.

[davidstaheli]

That agent proxy flow is a really great idea Christopher! We'll look into doing that. It would make things much simpler.

First, if your outgoing firewall rules prevent the agent from communicating with Azure Pipelines, here are the outbound URLs that need to be allowed.

For IP address ranges to allow inbound, they can possibly change. We have an XML document released every Wednesday that contains all IP ranges for Azure datacenters broken out by region. It can be downloaded from here. This file contains IP address ranges used by the Azure datacenters where Azure Pipelines is hosted. A new XML file is uploaded every Wednesday (pacific time) with the new planned IP address ranges grouped by region. New IP address ranges will be effective on the following Monday (pacific time). The new XML file must be downloaded for any necessary changes on your site before Monday. You’ll only need to whitelist the IP address ranges for your Azure DevOps organization's region which you can get from the link above. To verify your region in Azure DevOps, navigate to the Settings page at https://dev.azure.com/YOUR_ORGANIZATION_NAME/_settings/overview. Under Overview you will see a field for the organization's region.

[cbitter78]

Why cant you just publish the needed webhooks that are configured in GHE. And then I can manually configure my GHE. This would be an outbound to the internet call which in most GHE setups is allowed. And no firewall or agent is required.

[jcrowthe]

May we reopen this request? So far there has been great discussion, and I believe there is a great need here that is not well addressed by the various CI/CD products on the market. I think many would love it and Azure Pipelines is uniquely positioned to do so.

To throw out another idea here: DNS is also a problem. Even if a company punches a hole on 443 to allow Pipelines to access a corporate git server, the DNS name for that server often may resolve to a 10.0.0.0/8 address, making the DNS record unusable from Pipelines.

Potential solutions:

• Allow access to Git to happen over a VNET Peer. Many companies are already familiar with peering of VNET's and this can provide another way to access a corporate git without exposing git publicly.
• Use the agents as a proxy to corporate git. This could be a good solution, both from an access perspective and a DNS resolution perspective. Agents can already be self-managed, which means they can be run wherever a customer knows proper access can be found.
• Publish the Github webhooks necessary to trigger builds and assume the agents have the necessary connectivity to pull the codebase. There is probably some features that would be lost by this method though, and may not be possible since the Pipelines control plane needs deeper knowledge of the metadata around a repo (commit hashes, releases, tags, etc.)

In all, I think there are options here that are viable, but the community is dependent on Azure to provide them. Can we re-open this ticket to not forget this feature request?

[davidstaheli]

We've also been testing and creating instructions for using Azure App Services Hybrid Connections as a solution for this.

[davidstaheli]

Closing this issue because it's not related to documentation. However, we are still actively looking for ways to accomplish this using Azure App Services Hybrid Connections. This is a high priority for us. The reason the simple webhooks are insufficient is because Azure Pipelines still needs to query GHES for the branches, tags, users, etc. at runtime. Disabling this would remove quite a bit of functionality.

[busbyk]

@davidstaheli is there a feature request/issue I can follow along for progress on using Azure App Services Hybrid Connections to connect to an on-prem GHE server not accessible to the internet from Azure Pipelines? I would be interested in attempting to POC some ideas as I'm actively working on a project where I need this functionality.

[jcrowthe]

+1 Agreed, I'd like at least a ticket to track, as well as access to any POC that might be made available.

[eddieramirez]

+1 Does Azure App Services Hybrid Connections work out of the box or does something need to be done on the Azure Pipelines side to make it work?

[davidstaheli]

It still requires some sticky configuration and it isn't secure in that it allows public access to the endpoint exposed by the App Service. We want to secure and restrict the communication between Azure Pipelines and the App Service endpoint. We're working out a schedule/plan to address these and then I'll post a link here that can be tracked.

[devblackops]

I’d very much like to see how that hybrid connection would work. We have thought about ways to securely allow Pipelines access on-prem GitHub Enterprise but haven’t come up with something great. IMHO an on-prem build agent acting as a proxy would be OK as that pattern is how most on-prem agents interact with Azure (data gateways, integration runtimes, etc.).