Azure-docs: Health probe status unhealthy when using wildcard certificate on backend host (CN doesn't match host header)

[oising]

This seems like a bug -- "The Common Name (CN) of the backend certificate does not match the host header of the probe" is my error when I'm using a wildcard certifcate on the backend host. I'm using a custom probe because it's APIM (need custom path) and everything is fine when working directly with APIM. However, obviously the CN doesn't strictly match (myapim.mydomain.private against *.mydomain.private) -- but this is obviously incorrectly behaviour. Is this a known issue? I read everywhere else that widlcard certs are supported in appgw v2.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 77a317f1-deef-4ba7-5839-f39ce3733655
• Version Independent ID: d85aa8fe-7270-d073-ea56-d1c0759383b8
• Content: Enabling end to end SSL on Azure Application Gateway
• Content Source: articles/application-gateway/ssl-overview.md
• Service: application-gateway
• GitHub Login: @amsriva
• Microsoft Alias: victorh

[oising]

OK, so it's not a "bug" but the documentation is SEVERELY lacking. You cannot use a _self-signed_ wildcard certificate -- you must create a root self-signed root CA cert, then create a wildcard certificate signed with the self-signed root CA. You can then upload the CA root CER file (public key) to the http setting, and bind the APIM gateway with the wildcard PFX (full chain PFX). Man, this took days to figure out.

[TravisCragg-MSFT]

@oising this is discussed Here in the End to End SSL with the V2 SKU. Where would you recommend this requiremet be in the future to make this easier to troubleshoot?

[oising]

@TravisCragg-MSFT The language in this article ( https://docs.microsoft.com/en-us/azure/application-gateway/ssl-overview#end-to-end-ssl-with-the-v2-sku ) needs to be overhauled and made more specific. A self-signed certificate - by definition - has no chain: It is its own signer. So notes like "_The self-signed certificate must be a part of a certificate chain. A single self-signed certificate with no chain is not supported in V2 SKU_." are misleading. I think the writer is positing that "self-signed" means "you are not using a public CA", or perhaps that your root signing certificate is self-signed (but this is redudant - a root CA is always self-signed, regardless of origin.)

So, be more specific:

• If you want to create your own certificate for SSL (wildcard or not), it CANNOT be self-signed: i.e it is not its own signatory
• You must create a self-signed "root" cert, and then use that to sign the certificate destined for SSL usage (wildcard or not.)
• You must create a CER from the root CA, and add that to the app gateway as a "trusted root" cert.
• You must take the PFX of the SSL certificate (signed by the root cert) - (with full chain? end entity only?) and bind that as a custom domain for your API management instance.

Also, provide scripts to do the same using the official PKI module from Microsoft. Here, I'll even get you started:

#Requires -Module @{modulename="pki";moduleversion="1.0";guid="cf094c6b-63d1-4dda-bf70-15a602c4eb2b"}

# create root signing cert (expiration: 5 years, change as required)
$cert = New-SelfSignedCertificate -Type Custom -KeySpec Signature `
    -Subject "CN=MyCompany-internal-web-signing-root" -KeyExportPolicy Exportable `
    -HashAlgorithm sha256 -KeyLength 4096 `
    -CertStoreLocation "Cert:\CurrentUser\My" -KeyUsageProperty Sign -KeyUsage CertSign `
    -NotAfter (get-date).AddYears(5)

# create wildcard cert for SSL, and sign with root (expiration default: 1 year, change as required)
$ssl = New-SelfSignedCertificate -Type Custom -DnsName "*.mycompany.private","mycompany.private" `
    -KeySpec Signature `
    -Subject "CN=*.mycompany.private" -KeyExportPolicy Exportable `
    -HashAlgorithm sha256 -KeyLength 2048 `
    -CertStoreLocation "Cert:\CurrentUser\My" `
    -Signer $cert

# export SSL cert (pfx) for api management custom domain
Export-PfxCertificate -Cert $ssl -FilePath mycompany.private-fullchain.pfx `
    -ChainOption BuildChain -Password (read-host -AsSecureString -Prompt "password")

# export public key (CER) of root for app gateway trusted root config
Export-Certificate -Type CERT -Cert $cert -FilePath .\mycompany-internal-web-signing-root.cer

# export PFX of signing root to be kept secure for signing future SSL certs
 Export-PfxCertificate -Cert $cert -FilePath mycompany-internal-signing-web.pfx `
    -Password (read-host -AsSecureString -Prompt "password")

This should save a ton of people pain.

[oising]

Maybe as a follow on, show people how to store these certificates safely in azure keyvault. Though app gateway cannot use keyvault, api management can.

[TravisCragg-MSFT]

@oising I am moving this issue over to the End to End SSL with the V2 SKU page and assigning this issue to the doc author to evaluate and update as appropriate.

[vhorne]

reassign:winthrop28

[Tbohunek]

We have a CA-signed cert with multiple alternate names. Root cert is trusted and Backend health is properly healthy.
But if we perform the built-in Test on the Health probe blade, we always receive The Common Name (CN) of the backend certificate does not match the host header entered in the health probe configuration regardless of what is the hostname in the probe.

Is this result of the same underlying issue or do we have wrong config?

[bhushang19]

Documentation is hugely confusing when it comes to configuring the SSL with back-end specially when using the self signed certificates with V2 tier. @oising has done a fantastic job of explaining it here on this thread and hope MS makes the documentation clear with instructions to get this working.

Tried all the steps mentioned by @oising as below

• generated self signed wildcard domain certificate with root CA.
• Installed the self signed pfx on back-end (local machine cert store).
• Configured back-end endpoint with this wildcard self signed certificate.
• Installed the root signing certificate (.cer) on application gateway in the HttpSetting that talks to back-end

but still stuck with same error message as below when tried creating the health probe to back-end

_The Common Name (CN) of the backend certificate does not match the host header entered in the health probe configuration. Either change the probe configuration or add a valid certificate on the backend._

Theoretically configurations seems all ok, but what is it that we are missing? Is it that the wildcard is not yet supported?

Update: [2020-06-05]: Tried invoking the back-end SSL endpoint directly from the Postman (root certificate uploaded in the Postman) and could get the response without any issue and without any SSL error.
Can anyone confirm if something is missing on app gateway or could this be a potential bug in health probes implementation? Or where can we find the exact error message as health probe constantly seem to report the same error of CN mismatch message every time.

[oising]

@bhushang19 As a last resort, I would suggest to delete the certificates from AppGw's store, then delete the listener and recreate it.

[bhushang19]

@oising Yes, have tried it but still the same. Point to note here is that If you add the custom probe without 'Test' and then associate it with the HTTP setting - it works but issue occurs only when you try and create the custom health probe and 'Test' is before creating.

[Tbohunek]

I believe there is just a bug with the Test mechanism. The Probe itself works, the Backend Health also shows as healthy and everything works. Just that any time you decide to run the Test within the Probe config, it always fails with CN mismatch, no matter what you enter in the Host header or any other parameter.

[bhushang19]

@Tbohunek yes thought the same, seems like a potential bug.

[bhushang19]

Is there any update on this as this seems to be more than doc enhancement!

[varnav]

Application gateway gives me an error:

'Backend health': 'The Common Name (CN) of the backend certificate does not match the host header of the probe.' for both backend instances.

But if I try to access scale set instances directly, the certificate is valid:

https://apitestss000000.mycorp.local/ and https://apitestss000001.mycorp.local/

look fine and valid in browser

HTTP setting is set to 'Pick hostname from backend target.' If I set it to 'Override with specific domain name' and type in one of the hostnames like 'apitestss000001.mycorp.local', there is no error for this host, but still error for the other. Looks like 'Pick hostname from backend target' picks the wrong hostname from the scale set.

[Flodu31]

Hello,
I have exactly the same error from backend health: The Common Name (CN) of the backend certificate does not match the host header of the probe.' for both backend instances.
The certificate is valid because it works on many websites.
It's a wildcard certificate, with many DNS name inside (wildcard also).
Anyone has the same setup that works?
Thanks.
Florent

[mayanksingh-ops]

I am also facing the same issue with the application gateway. First I got the Common Name does not match with backend issue for that I have created dns name for backend server after that this issue got fixed by adding domain name of backend in the custom probe. After that I am getting below error
'The root certificate of the server certificate used by the backend does not match the trusted root certificate added to the application gateway. Ensure that you add the correct root certificate to whitelist the backend'

For that I have added same root CA cert which had been used by my backend instance in http settings but still I am getting this error.
Can this be checked or am I missing something ?
Thanks
Mayank

[oising]

I believe there is just a bug with the Test mechanism. The Probe itself works, the Backend Health also shows as healthy and everything works. Just that any time you decide to run the Test within the Probe config, it always fails with CN mismatch, no matter what you enter in the Host header or any other parameter.

I am seeing this behaviour also. Standard_V2

@TravisCragg-MSFT

[danielriosb]

@mayanksingh-ops Did you figure out a solution for that message? I'm fancing the same one.