When configuring ASP.NET MVC Core application to use X509Certificate2 that is not in computer's certificate store, application throws CryptographicException: Keyset does not exist.
I am running ASP.NET MVC Core on top of .NET Framework 4.7.1, but I suspect that is not an issue. More details bellow.
This is sample code where I configure MVC service for DataProtection in startup class:
var x509Certificate2 = CertificateLoader.Load();
services.AddDataProtection()
.SetApplicationName("MyApp")
.ProtectKeysWithCertificate(x509Certificate2)
.PersistKeysToFileSystem(new DirectoryInfo("D:\\Temp\\DataProtection"));
Reason for CryptographicException seems to be that private class EncryptedXmlWithCertificateKeys calls base class method EncryptedXml.DecryptEncryptedKey. Base class has no knowledge of private _certificates collection of subclass, so it probably goes to certificate store by default. Bellow are lines from EncryptedXmlWithCertificateKeys class.
public override byte[] DecryptEncryptedKey(EncryptedKey encryptedKey)
{
byte[] key = base.DecryptEncryptedKey(encryptedKey);
if (key != null)
{
return key;
}
Relevant stack trace:
System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, bool randomKeyContainer)
System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, bool randomKeyContainer, int dwKeySize, ref SafeProvHandle safeProvHandle, ref SafeKeyHandle safeKeyHandle)
System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
System.Security.Cryptography.RSACryptoServiceProvider..ctor(int dwKeySize, CspParameters parameters, bool useDefaultKeySize)
System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)
System.Security.Cryptography.CngLightup.GetRSAPrivateKey(X509Certificate2 cert)
System.Security.Cryptography.Xml.EncryptedXml.DecryptEncryptedKey(EncryptedKey encryptedKey)
Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor+EncryptedXmlWithCertificateKeys.DecryptEncryptedKey(EncryptedKey encryptedKey)
System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, string symmetricAlgorithmUri)
System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey+<>c__DisplayClass1_0.b__0()
System.Lazy.CreateValue()
System.Lazy.get_Value()
Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing+KeyHolder.GetEncryptorInstance(out bool isRevoked)
Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.get_DefaultAuthenticatedEncryptor()
Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(byte[] plaintext)
Microsoft.AspNetCore.Authentication.SecureDataFormat.Protect(TData data, string purpose)
Microsoft.AspNetCore.Authentication.SecureDataFormat.Protect(TData data)
Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.WriteNonceCookie(string nonce)
Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleChallengeAsync(AuthenticationProperties properties)
Microsoft.AspNetCore.Authentication.AuthenticationHandler.ChallengeAsync(AuthenticationProperties properties)
Microsoft.AspNetCore.Authentication.AuthenticationService.ChallengeAsync(HttpContext context, string scheme, AuthenticationProperties properties)
Microsoft.AspNetCore.Mvc.ChallengeResult.ExecuteResultAsync(ActionContext context)
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultAsync(IActionResult result)
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAlwaysRunResultFilters()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
See https://github.com/aspnet/Home/issues/2884
You need to additionally configure with .UnprotectKeysWithAnyCertificate()
@Rick-Anderson I don't think we ever documented this, probably because I forgot to ask ...
I added .UnprotectKeysWithAnyCertificate() as suggested by @blowdart , but getting exactly the same error . This is sample configuration code as it is now:
var x509Certificate2 = CertificateLoader.Load();
services.AddDataProtection()
.SetApplicationName("MyApp")
.ProtectKeysWithCertificate(x509Certificate2)
.UnprotectKeysWithAnyCertificate()
.PersistKeysToFileSystem(new DirectoryInfo("D:\\Temp\\DataProtection"));
Here is error message and beginning of stack trace. Full stacktrace is identical to the I originally posted.
CryptographicException: Keyset does not exist
System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, bool randomKeyContainer)
Just to make sure, I tried extracting RSA key from my certificate (2nd line) and it is available:
var x509Certificate2 = Certificates.Certificate.Load();
var privateKey = x509Certificate2.GetRSAPrivateKey();
services.AddDataProtection()
.SetApplicationName("MyApp")
.ProtectKeysWithCertificate(x509Certificate2)
.UnprotectKeysWithAnyCertificate()
.PersistKeysToFileSystem(new DirectoryInfo("D:\\Temp\\DataProtection"));
Just for completion, this is XML file in DataProtection folder:
<?xml version="1.0" encoding="utf-8"?>
<key id="63903c99-8a40-4b80-802d-638a547914bb" version="1">
<creationDate>2018-06-13T07:15:15.3249054Z</creationDate>
<activationDate>2018-06-13T07:15:15.0158935Z</activationDate>
<expirationDate>2018-09-11T07:15:15.0158935Z</expirationDate>
<descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
<descriptor>
<encryption algorithm="AES_256_CBC" />
<validation algorithm="HMACSHA256" />
<encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection">
<EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#">
<EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
<EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<X509Data>
<X509Certificate>MIIDYTCCAk2gAwIBAgIQZ1i9fosfYJ1D/63SZAANIjAJBgUrDgMCHQUAMDIxMDAuBgNVBAMTJ3Z3di5pbW1vdmFsdWF0aW9uLmNvbSAtIGRhdGEgcHJvdGVjdGlvbjAeFw0xODA2MTIxMjM2NDlaFw0zOTEyMzEyMzU5NTlaMDIxMDAuBgNVBAMTJ3Z3di5pbW1vdmFsdWF0aW9uLmNvbSAtIGRhdGEgcHJvdGVjdGlvbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMkMn30WO5OghutSvs+5e6X5ib1c3fbaU8IefOMP88AVLfvuppzOC8BDlOYjv3AQLWQWN3IKBWYmvzmG7ZpBdh64zfSFWSXSbpKWMvkSPjKlbP0Mp+7FFJ6CdWoVYAB3Xs9vc9KO6c4AEluKExkpnr27Bn2uIoAEYiYNrnXMm2Oh/rJnf1dcaWmAPlFehhS0UOPH/ja58rfmJvJ6+yrhq/6nr2vCmMF4vyhdVv6rdsx7qxSKUway6OcJ37wGFuUuJHVDsv4hRKwMsqH+MyYbc1yuBZp60vnvhJW0JkMyBmDBP3M1+TbpGkFUOmuRR5STOM0PRgxJzuZ2xikbtp8a+NkCAwEAAaN7MHkwEgYDVR0lBAswCQYHKoZIhvcNAzBjBgNVHQEEXDBagBBHIi5MLKWsxmJ+k8YGpmfnoTQwMjEwMC4GA1UEAxMndnd2LmltbW92YWx1YXRpb24uY29tIC0gZGF0YSBwcm90ZWN0aW9ughBnWL1+ix9gnUP/rdJkAA0iMAkGBSsOAwIdBQADggEBACNo/3ryImyoxeA6UE6aWFwym/dCQRCWf5TU83/Qmi0cQv2J8p5FFvwSMMf1eQGa2kxeBTCFRmv1L6oxa0E85XeljEcVbKBVt1Edv9a95XEOMjHZwqJjNBOvaHCmlhCvyvnDnWDwfW3LKJJkaktjyCHuDMXPlk2n7Fpo6WU1ljL+Zx08ed9rikhZkzimp9sS5ItZRbsoULd/MS1hLScjmjeIELwqMA0rpr3BdXVm3wBIQjKyevxDVP2a2/E/kEbJuQXM6CLZJpK+uJt7Mf5mXdf8G/lp+WY3yVE3j7LYoH3HDz7PVrtSATbM7UCDkGcG1pOYxI/l5HBnz01lD/XtQ9o=</X509Certificate>
</X509Data>
</KeyInfo>
<CipherData>
<CipherValue>f4qagRAVmwhkcs3Y4iu9R1LtDnapOUt+SckoLVBJKZiVpf/x+shA8/24uTbqgPvKaYlVGmsoApMK/aNjo1SeRVLfpMZvJxEu4vySExOjiFKxOxnJA1yt1zCHnkKPzxz4kawAaU2+Aqc41zPavBg72r3TaBeCH3bm3AFy/tbpytG8yPybG+WYTGd3lHGIps0ne3L21MWWv0ZtTjuQHjhMcm8dAyuz9bIPBlVliOLf14ORwsgCymSct9sNPBfsa8JI+bPEe1phxPpyzLqQmkDKYIkJA56ypO0A9S0WqMupLLWiIDW4QD9UW9jqDuMWgnaDo/NPL+9vEn/N8LCYtDLiUA==</CipherValue>
</CipherData>
</EncryptedKey>
</KeyInfo>
<CipherData>
<CipherValue>WLlxWTdnxD6i+HeChsJ72ucOnyVlW0xoCFRnUZ+mmSk0M7u1mNcqwQ+f0rrikN766wF6yORZzW2LQfX8Oyt7bIxANY12iM//Nw8w7CwT5ITbb6Csac5lO9mMXPKidoPTEhdgyhJJCiq30ZUxSiVXLFcd0l6uPeEzfa0qhG5Y8ssoPS+R61cFYHdO6Hoj2SiRriW5AyvYMtWjMhqf+wMRfGW/ei+Wfacb6Hzi4goBC9fZPTrM2lDbvzRWsbxyJW7HAC94ze40z5TFRCIHqK1yhHXP7s9nwe1d9jv+K2QxPWiyHD0Lg8gDGICT9nywQicbgmHhv5HgsojqcXl2vd2oqqo3GHQcYvfW9tne3pVUPOdQ5o3yIhjkFtt7VtVbLJCFfFyJjpQBfFN+0+piorrJdg==</CipherValue>
</CipherData>
</EncryptedData>
</encryptedSecret>
</descriptor>
</descriptor>
</key>
@natemcmaster Can you take a look?
Can you confirm that the X509Certificate2 object you are loading matches the expected certificate from your XML key data?
```c#
var x509Certificate2 = Certificates.Certificate.Load();
Console.WriteLine(x509Certificate2.Thumbprint);
// this content came from the
var expectedCert = new X509Certificate2(Convert.FromBase64String("MIIDYTCCAk2gAwIBAgIQZ1i9fosfYJ1D/63SZAANIjAJBgUrDgMCHQUAMDIxMDAuBgNVBAMTJ3Z3di5pbW1vdmFsdWF0aW9uLmNvbSAtIGRhdGEgcHJvdGVjdGlvbjAeFw0xODA2MTIxMjM2NDlaFw0zOTEyMzEyMzU5NTlaMDIxMDAuBgNVBAMTJ3Z3di5pbW1vdmFsdWF0aW9uLmNvbSAtIGRhdGEgcHJvdGVjdGlvbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMkMn30WO5OghutSvs+5e6X5ib1c3fbaU8IefOMP88AVLfvuppzOC8BDlOYjv3AQLWQWN3IKBWYmvzmG7ZpBdh64zfSFWSXSbpKWMvkSPjKlbP0Mp+7FFJ6CdWoVYAB3Xs9vc9KO6c4AEluKExkpnr27Bn2uIoAEYiYNrnXMm2Oh/rJnf1dcaWmAPlFehhS0UOPH/ja58rfmJvJ6+yrhq/6nr2vCmMF4vyhdVv6rdsx7qxSKUway6OcJ37wGFuUuJHVDsv4hRKwMsqH+MyYbc1yuBZp60vnvhJW0JkMyBmDBP3M1+TbpGkFUOmuRR5STOM0PRgxJzuZ2xikbtp8a+NkCAwEAAaN7MHkwEgYDVR0lBAswCQYHKoZIhvcNAzBjBgNVHQEEXDBagBBHIi5MLKWsxmJ+k8YGpmfnoTQwMjEwMC4GA1UEAxMndnd2LmltbW92YWx1YXRpb24uY29tIC0gZGF0YSBwcm90ZWN0aW9ughBnWL1+ix9gnUP/rdJkAA0iMAkGBSsOAwIdBQADggEBACNo/3ryImyoxeA6UE6aWFwym/dCQRCWf5TU83/Qmi0cQv2J8p5FFvwSMMf1eQGa2kxeBTCFRmv1L6oxa0E85XeljEcVbKBVt1Edv9a95XEOMjHZwqJjNBOvaHCmlhCvyvnDnWDwfW3LKJJkaktjyCHuDMXPlk2n7Fpo6WU1ljL+Zx08ed9rikhZkzimp9sS5ItZRbsoULd/MS1hLScjmjeIELwqMA0rpr3BdXVm3wBIQjKyevxDVP2a2/E/kEbJuQXM6CLZJpK+uJt7Mf5mXdf8G/lp+WY3yVE3j7LYoH3HDz7PVrtSATbM7UCDkGcG1pOYxI/l5HBnz01lD/XtQ9o="));
Console.WriteLine(expectedCert.Thumbprint);
```
@natemcmaster, I ran your code in the app. It is the same thumbprint:
Website2> D717BCF9CEEC51D377E6CF10AEF338AAECF56CD2
Website2> D717BCF9CEEC51D377E6CF10AEF338AAECF56CD2
And it makes sense, because folder I configured new DirectoryInfo("D:\\Temp\\DataProtection") was initially empty. My app created XML in the first place.
I think that baseclass method EncryptedXml.DecryptEncryptedKey that I mentioned in more details in the 1st post on this issue should be investigated.
One more thing, can you confirm this cert is not installed to any local machine or current user cert store? You can check quickly by running this command in powershell
Get-ChildItem -Recurse Cert:\* | ? { $_.SerialNumber -eq '6758bd7e8b1f609d43ffadd264000d22' }
I cannot reproduce this error on my own. If you have a standalone repro, please share.
Without a repro, the best I can offer is a guess, and my guess is that your certificate is actually being loaded from the cert store. For certificates _not_ in the store, what is supposed to happen is this:
Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor+EncryptedXmlWithCertificateKeys and requests to decrypt an XML file.EncryptedXmlWithCertificateKeys.DecryptDocument first needs to load the private key. This calls EncryptedXmlWithCertificateKeys.DecryptEncryptedKeyEncryptedXmlWithCertificateKeys.DecryptEncryptedKey will first check if the key can be decrypted using the X509Store by calling base.DecryptEncryptedKey. This should return null if the X509Certificate is not in the CurrentUser\My and LocalMachine\My X509Store.EncryptedXmlWithCertificateKeys.DecryptEncryptedKey then searches its own dictionary of X509Certificate2 objects which come from DataProtectionBuilder.ProtectKeysWithCertificate and .UnprotectKeysWithCertificate.Your stack trace suggestions the problem is coming from step 3 -- checking the X509Store for the cert. Your stack trace indicates it threw from https://github.com/dotnet/corefx/blob/6d571f70a69a3a1dcf54f73ab828c545de40690a/src/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/EncryptedXml.cs#L438. If I've inspected the code correctly, the only way you get to this point is if your X509Certificate is in the LocalMachine\My and/or CurrentUser\My cert store.
Yes, my certificate was in the store. MakeCert command adds it there automatically. I export it afterwards, without deleting it. I did not expect copy of the certificate in the store to cause the issue.
At the end, after deleting certificate from the store, my original code works:
var x509Certificate2 = CertificateLoader.Load();
services.AddDataProtection()
.SetApplicationName("MyApp")
.ProtectKeysWithCertificate(x509Certificate2)
.PersistKeysToFileSystem(new DirectoryInfo("D:\\Temp\\DataProtection"));
@blowdart There was no need to add .UnprotectKeysWithAnyCertificate() option.
@natemcmaster Just from curiosity, why step 3 blows up, when it finds same certificate in the store?
Thank you guys for quick responses and awesome clarification. Really appreciate.
The error says "Keyset does not exist" and is thrown from X509Certificate2.get_PrivateKey(), so I'm assuming this failed because the certificate imported into your cert store did not contain the private key.
I'm going to leave this open, however, because I think we can do better. You're not the first to report issues working with keys. The messages in the CryptographicException are unclear and don't make it obvious how to resolve the issue. I think a better error message in this case would have said something like "Could not decrypt the data protection key material in (xml path) because this file was encrypted and the private key for X509Certificate (serial #) could not be found".
@bartonjs - what would you say to adding something like an ErrorCode enum to CryptographicException? My thinking is that if we could check CryptographicException.ErrorCode == ErrorCodes.PrivateKeyDoesNotExist, we could wrap the original exception with a new one whose message is more specific to DataProtection.
cc @blowdart
The error says "Keyset does not exist" and is thrown from X509Certificate2.get_PrivateKey()
That could mean that the key is being held by CNG, instead of CAPI. Don't use the PrivateKey property (pretty much ever), use Get[Algorithm]PrivateKey() instead.
Since this usage came from GetRSAPrivateKey it means that the key is supposed to be stored in CAPI, and the cert knows what the name of the key should be, but someone deleted it (which probably means that an instance created by loading a PFX (or cloned from such an instance) got Disposed / Finalized, which deletes the private key that the PFX load put on disk.
what would you say to adding something like an ErrorCode enum to CryptographicException?
It's a concept I've toyed with. But I usually reject it because a large number (perhaps the majority) of the exceptions are "a native layer gave me an error code, just ask the native layer for the text of that code and throw". Personally, as a platform consumer, I'd rather have no error code than error codes be wrong, and the unwillingness to be wrong has been what usually stopped me from figuring out how we'd make it work.
I wonder if var x509Certificate2 = CertificateLoader.Load(); gets disposed at some awkward point during runtime. Maybe we should clone the certificate inside ProtectKeysWithCertificate() and use the clone?
@blowdart I don't think that's what's happening here. The X509Certificate2 instance EncryptedXml is using came from the X509Store. It's not the same instance we received from the ProtectKeysWithCertificate call. In theroy, this could happen, so we could investigate creating a deep-copy of the X509Certificate2 cert, but I'm also ok telling users not to dispose objects they give to our API.
@bartonjs I agree a wrong error message is worse than no error message, but IMO a vague error message is just as bad. I don't think I'm alone on this one (one data sample: 44,000 views on this StackOverflow question).
Just to clarify two details of my use case. It might narrow possible reasons for the error.
1) Certificate was created using MakeCert command and was automatically added to current user's certificate store. Later it was exported to *.pfx file, including private key. This is exact command I used:
Makecert -r -pe -n CN="my certificate" -sky exchange -eku 1.2.840.113549.3 -ss my -sr localmachine -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12 -len 2048
Both certificate instances (one in the store and *.pfx file) contain private key. That's where message CryptographicException: Keyset does not exist really confuses me.
2) Method var x509Certificate2 = CertificateLoader.Load(); uses following X509Certificate2 constructor:
public X509Certificate2(byte[] rawData, string password)
Method just returns certificate instance. I am not disposing that object instance anywhere explicitly. My assumption is that disposing is not an issue here.
Thanks for the additional info @nenadvicentic. @blowdart I recommend closing as I don't think there is any action for us here. I don't think we can provide a better error message for users, and the source of the problem appears to be the result of misconfiguring the cert store. We can reopen if you think there is something actionable here.
@natemcmaster Allow me to disagree here. I am configuring data-protection with "standalone" certificate loaded from file. I am not expecting that if I copy certificate in the Windows certificate store application suddenly blows up. Fact that there is dependency to the certificate store is hidden implementation detail.
At least, all this quirks should be clearly documented. We even had advise here to add UnprotectKeysWithAnyCertificate() which turned out not to be needed at all. It just proves to me that there is solid confusion how to configure this feature properly.
At least, all this quirks should be clearly documented.
cc @guardrex
@nenadvicentic PR's welcome in the doc repo.
That's a good point. Your comment gave me an idea I don't think we've explored. We could look into changing the precedence of the where we look for private keys. Presumably, if you pass us a X509Certificate2 object with a private key, that could come first, before we look at the Windows cert store.
Thoughts @blowdart ?
If the XML classes allow then then yes, that might work. But I don't know if they do.
It's really easy to fix this. Basically, we swap these lines so they come last, after we check certs given to us explicitly first.
I think this may help: https://github.com/aspnet/DataProtection/pull/314.
This should be resolved in 2.2 with this change: https://github.com/aspnet/DataProtection/pull/314