Azure-docs: Can Service Fabric configuration settings be encrypted on a local dev cluster?

[rjmiller-revint]

I've put the cert in the HKLM/My and HKCU/My, and given Everyone access to the private key in HKLM/My. So I don't know what else to do.

I'm trying to test this on my local development cluster and it is failing with the following error:
"There was an error during activation.Failed to configure certificate permissions. Error E_FAIL."

I also see the following errors in the event viewer under "Apps & Services Logs -> Microsoft-Service-Fabric -> Admin"

• <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
• <System>
  <Provider Name="Microsoft-ServiceFabric" Guid="{cbd93bc2-71e5-4566-b3a7-595d8eeca6e8}" />
  <EventID>256</EventID>
  <Version>2</Version>
  <Level>2</Level>
  <Task>1</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8000000000000001</Keywords>
  <TimeCreated SystemTime="2019-04-17T20:56:58.170500200Z" />
  <EventRecordID>57856</EventRecordID>
  <Correlation />
  <Execution ProcessID="332" ThreadID="19704" />
  <Channel>Microsoft-ServiceFabric/Admin</Channel>
  <Computer>COMPUTERNAME</Computer>
  <Security UserID="S-#-#-##" />
  </System>
• <EventData>
  <Data Name="id" />
  <Data Name="type">CryptoUtility</Data>
  <Data Name="text">CryptAcquireCertificatePrivateKey failed. Error:0x80090014</Data>
  </EventData>
  </Event>

• <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
• <System>
  <Provider Name="Microsoft-ServiceFabric" Guid="{cbd93bc2-71e5-4566-b3a7-595d8eeca6e8}" />
  <EventID>257</EventID>
  <Version>2</Version>
  <Level>3</Level>
  <Task>1</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8000000000000001</Keywords>
  <TimeCreated SystemTime="2019-04-17T20:56:58.170502500Z" />
  <EventRecordID>57857</EventRecordID>
  <Correlation />
  <Execution ProcessID="332" ThreadID="19704" />
  <Channel>Microsoft-ServiceFabric/Admin</Channel>
  <Computer>COMPUTERNAME</Computer>
  <Security UserID="S-#-#-##" />
  </System>

• <EventData>
  <Data Name="id" />
  <Data Name="type">CryptoUtility</Data>
  <Data Name="text">Can't get private key filename for certificate. Error: 0x80090014</Data>
  </EventData>
  </Event>

• <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

• <System>
  <Provider Name="Microsoft-ServiceFabric" Guid="{cbd93bc2-71e5-4566-b3a7-595d8eeca6e8}" />
  <EventID>257</EventID>
  <Version>2</Version>
  <Level>3</Level>
  <Task>1</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8000000000000001</Keywords>
  <TimeCreated SystemTime="2019-04-17T20:56:58.170504000Z" />
  <EventRecordID>57858</EventRecordID>
  <Correlation />
  <Execution ProcessID="332" ThreadID="19704" />
  <Channel>Microsoft-ServiceFabric/Admin</Channel>
  <Computer>COMPUTERNAME</Computer>
  <Security UserID="S-#-#-##" />
  </System>

• <EventData>
  <Data Name="id" />
  <Data Name="type">CryptoUtility</Data>
  <Data Name="text">All tries to get private key filename failed.</Data>
  </EventData>
  </Event>

• <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

• <System>
  <Provider Name="Microsoft-ServiceFabric" Guid="{cbd93bc2-71e5-4566-b3a7-595d8eeca6e8}" />
  <EventID>256</EventID>
  <Version>2</Version>
  <Level>2</Level>
  <Task>1</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8000000000000001</Keywords>
  <TimeCreated SystemTime="2019-04-17T20:56:58.170516700Z" />
  <EventRecordID>57859</EventRecordID>
  <Correlation />
  <Execution ProcessID="332" ThreadID="19704" />
  <Channel>Microsoft-ServiceFabric/Admin</Channel>
  <Computer>COMPUTERNAME</Computer>
  <Security UserID="S-#-#-##" />
  </System>

• <EventData>
  <Data Name="id" />
  <Data Name="type">SecurityUtility</Data>
  <Data Name="text">Failed to get the Certificate's private key. Thumbprint:<certthumbprint_here>. Error: E_FAIL</Data>
  </EventData>
  </Event>

• <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

• <System>
  <Provider Name="Microsoft-ServiceFabric" Guid="{cbd93bc2-71e5-4566-b3a7-595d8eeca6e8}" />
  <EventID>256</EventID>
  <Version>2</Version>
  <Level>2</Level>
  <Task>1</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8000000000000001</Keywords>
  <TimeCreated SystemTime="2019-04-17T20:56:58.170519500Z" />
  <EventRecordID>57860</EventRecordID>
  <Correlation />
  <Execution ProcessID="332" ThreadID="19704" />
  <Channel>Microsoft-ServiceFabric/Admin</Channel>
  <Computer>COMPUTER</Computer>
  <Security UserID="S-#-#-##" />
  </System>

• <EventData>
  <Data Name="id" />
  <Data Name="type">SecurityUtility</Data>
  <Data Name="text">Failed to get private key file. x509FindValue:<certthumbprint_here>, x509StoreName: My, findType: FindByThumbprint, Error E_FAIL</Data>
  </EventData>
  </Event>

• <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

• <System>
  <Provider Name="Microsoft-ServiceFabric" Guid="{cbd93bc2-71e5-4566-b3a7-595d8eeca6e8}" />
  <EventID>23041</EventID>
  <Version>2</Version>
  <Level>3</Level>
  <Task>90</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8000000000000001</Keywords>
  <TimeCreated SystemTime="2019-04-17T20:56:58.170521400Z" />
  <EventRecordID>57861</EventRecordID>
  <Correlation />
  <Execution ProcessID="332" ThreadID="19704" />
  <Channel>Microsoft-ServiceFabric/Admin</Channel>
  <Computer>COMPUTERNAME/Computer>
  <Security UserID="S-#-#-##" />
  </System>
• <EventData>
  <Data Name="id" />
  <Data Name="type">ProcessActivationManager</Data>
  <Data Name="text">ACLing private key filename for thumbprint <certthumbprint_here>. ErrorCode=E_FAIL</Data>
  </EventData>
  </Event>

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 65744c73-e7a2-6d5b-004c-22f980730bd0
• Version Independent ID: d3366f95-1038-0e0c-4f98-9ce25b96c928
• Content: Set up an encryption certificate and encrypt secrets on Azure Service Fabric Windows clusters
• Content Source: articles/service-fabric/service-fabric-application-secret-management-windows.md
• Service: service-fabric
• GitHub Login: @vturecek
• Microsoft Alias: vturecek

[mimckitt]

@rjmiller-revint what exactly is the issue? Are you trying to deploy a local secured dev cluster and getting this error?

What command are you running that is presenting you with this error?

[rjmiller-revint]

I’m trying to have encrypted configuration parameters. But for some reason the local dev cluster doesn’t seem to be able to load the private key for the certificate. I’ve tried everything.

It’s clearly able to decrypt the string because it is reading the certhumbprint from the ciphertext. But for some reason it can’t read the private key for the cert.

ApplicationManifest.xml

Settings.xml

Local.1Node.xml

From: Micah [e-mail]
Sent: Wednesday, April 17, 2019 2:50 PM
To: MicrosoftDocs/azure-docs [e-mail]
Cc: Randy Miller [e-mail]; Mention [e-mail]
Subject: Re: [MicrosoftDocs/azure-docs] Can Service Fabric configuration settings be encrypted on a local dev cluster? (#29612)

@rjmiller-revinthttps://github.com/rjmiller-revint what exactly is the issue? Are you trying to deploy a local secured dev cluster and getting this error?

What command are you running that is presenting you with this error?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/29612#issuecomment-484274377, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALBPN3GFCIE3I6C72PRYTLLPQ6LPJANCNFSM4HGX3MJQ.
Confidentiality Notice: This email message and any accompanying documents is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, distribution, or copying is prohibited. If you are not the intended recipient, please contact our office by email or telephone at 1-855-252-7606 and immediately destroy all copies of the original message.
Confidentiality Notice: This email message and any accompanying documents is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, distribution, or copying is prohibited. If you are not the intended recipient, please contact our office by email or telephone at 1-855-252-7606 and immediately destroy all copies of the original message.

[mimckitt]

@rjmiller-revint I don't think I have seen a case where a user is trying to encrypt the configuration settings... I will have to check to see if this is supported. What is your use case for this?

CC @aljo-microsoft

[rjmiller-revint]

Here is the link to the document that I'm trying to use: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-application-secret-management-windows. We want to encrypt the configuration settings so that we aren't putting plain text secrets into our source code repository when running on our local dev cluster.

[mimckitt]

thanks @rjmiller-revint

I am checking on this offline and will update you shortly.

[mimckitt]

@rjmiller-revint I checked with the SF team on this. Here was the response:

The error mentioned in the event is NTE_BAD_PROV_TYPE, which indicates an attempt to access a K(ey)S(torage)P(rovider)-managed private key using a (legacy) C(rypto)S(ervice)P(rovider) object. In other words, the code is using the wrong API to access the certificate’s private key.

This is a known issue/limitation in managed SF code which handles certificates, as .net does not support out of the box CNG object. That is, an extension exists, we just haven’t picked it up.

Can you please confirm that the provider of the self-signed certificate created according to our documentation is, indeed, “Microsoft Enhanced Cryptographic Provider v1.0”. Unless otherwise specified, the PSh New-SelfSignedCert will generate a CNG cert (provider = “Microsoft Software Key Storage Provider”).

To confirm:
Cd cert:\localmachine\my
$cert = gci
$cert.HasPrivateKey -> should return ‘true’
$cert.PrivateKey -> should be non-null
$cert.PrivateKey.CspKeyContainerInfo.ProviderName -> should return Microsoft Enhanced Cryptographic Provider v1.0

Another option is certutil -v -store my

CC @dragav

[mimckitt]

@rjmiller-revint any update on this?

[mimckitt]

@rjmiller-revint I will close for now. If you have further questions let me know and we can reopen and continue the discussion.

[rjmille2]

Here is the output from certutil for the certificate in question which clearly shows the legacy provider not the CSP provider.

================ Certificate 2 ================
X509 Certificate:
Version: 3
Serial Number: 76c44c83a51a649842d0f723a89aee81
Signature Algorithm:
Algorithm ObjectId: 1.2.840.113549.1.1.5 sha1RSA
Algorithm Parameters:
05 00
Issuer:
CN=ServiceFabricConfigEcryption-Local/Dev
Name Hash(sha1): b03a7d15177326c244488e6982e78a1e75dd9cee
Name Hash(md5): 5e13969420029c9025bf4ec3edc74c0a

NotBefore: 4/17/2019 5:25 PM
NotAfter: 1/1/2070 12:00 AM

Subject:
CN=ServiceFabricConfigEcryption-Local/Dev
Name Hash(sha1): b03a7d15177326c244488e6982e78a1e75dd9cee
Name Hash(md5): 5e13969420029c9025bf4ec3edc74c0a

Public Key Algorithm:
Algorithm ObjectId: 1.2.840.113549.1.1.1 RSA
Algorithm Parameters:
05 00
Public Key Length: 2048 bits
Public Key: UnusedBits = 0
0000 30 82 01 0a 02 82 01 01 00 dd 61 3c 7a 60 5e df
0010 ef 78 8e f0 e4 04 89 7b ff d4 80 85 c6 40 26 c8
0020 12 05 f4 d3 18 f2 5c d2 83 c6 a7 77 db 4a d7 a3
0030 bc 76 be 97 3b 85 56 7d f4 c2 73 ae 00 dc f3 78
0040 73 41 5e 05 25 2e e8 05 98 cb 45 1e c8 06 6a b6
0050 39 f5 8d dc ba d5 fb 66 9e 04 d8 4f 19 fb fc 56
0060 86 f2 ae e4 9a d5 89 82 c2 03 95 14 6f 93 ab 4e
0070 93 28 76 a0 69 cc bf e1 c4 0f 13 bd 93 36 8f f9
0080 78 79 10 3e dd 15 f6 b7 1c ac 4f 70 85 ff bd 05
0090 ce 61 02 07 83 95 a4 9b 6f 6d 0e 10 13 6b 7a d8
00a0 08 8c 52 9c 0f f0 fc a6 4a 5e 0b c4 40 53 28 de
00b0 aa 7f 10 1c 75 2e ff 3c 65 56 ef 6f 29 a9 8d a8
00c0 0b 35 91 59 43 91 ce ed 29 a4 43 4d 36 e8 a0 44
00d0 09 47 e7 29 90 eb 73 01 9a 09 1a 69 6e 42 a3 9b
00e0 40 a6 ed 79 ec be a4 09 3e 9e 7e 4d 5a 6b ec fa
00f0 4f 0d 2d 66 8c ae 5f 02 66 17 48 e8 1f 35 5e 4c
0100 37 22 43 7f a4 2e c9 6e 71 02 03 01 00 01
Certificate Extensions: 3
2.5.29.15: Flags = 1(Critical), Length = 4
Key Usage
Data Encipherment (10)

2.5.29.37: Flags = 0, Length = d
Enhanced Key Usage
    Document Encryption (1.3.6.1.4.1.311.80.1)

2.5.29.14: Flags = 0, Length = 16
Subject Key Identifier
    b1147f813d2fcde4504f16435b14c878382b5384

Signature Algorithm:
Algorithm ObjectId: 1.2.840.113549.1.1.5 sha1RSA
Algorithm Parameters:
05 00
Signature: UnusedBits=0
0000 4a bf 44 47 95 ed f7 d8 49 d0 62 8b 8d 37 1b 73
0010 59 9a ea cf 1d 9d cd 2c 43 6b f3 af 8e 1c 65 cd
0020 84 3b 1f e8 64 e5 30 6b 8b ce 1c bf c5 7e d9 f8
0030 75 f9 5a ab 33 b6 6a c4 df 0c 72 8e 60 ad b1 95
0040 d7 2f 8d 7a 57 8b 2a 88 96 d3 46 b0 ac 7d 5c 3f
0050 f6 a9 85 8f cb 4b 06 43 89 47 2b 6f 5d 76 33 7e
0060 02 c7 31 04 eb 21 f6 8e dd 3c 3a 3c 0c 5a d2 e8
0070 21 68 39 93 a1 d3 69 40 24 23 32 45 a8 60 b7 ac
0080 95 f5 39 33 ea 4c 65 85 7b 34 d9 ec 7a ec 3a e2
0090 b8 4b 43 47 3d 88 98 97 16 4b af 88 92 a2 9a 5e
00a0 13 79 16 78 1b ed 1f 5b c3 98 9c 91 e4 c1 c6 20
00b0 8e 5f f6 7a de e2 71 60 e7 39 42 3e 0b fb d9 8c
00c0 a2 db 9d 3a ba a0 fc df cb 16 11 f1 87 60 4d bd
00d0 e0 c8 50 02 99 99 aa 23 81 11 2a 16 f6 6e ea 77
00e0 a9 af 78 d8 ac c3 f8 16 e6 23 ab 8f 93 c6 a1 2d
00f0 7e 02 5a e2 83 13 d9 98 95 ac 8c 53 42 58 71 d4
Signature matches Public Key
Root Certificate: Subject matches Issuer
Key Id Hash(rfc-sha1): b1147f813d2fcde4504f16435b14c878382b5384
Key Id Hash(sha1): 0889f2950565ae2f052fb1403fc2d9e3c955e364
Key Id Hash(bcrypt-sha1): 46ca4b7ecb045f0fc23ff67f5a14442cbfd15099
Key Id Hash(bcrypt-sha256): ba97889a872e981f3b3f639e0beb4d20af8ab614567aad8311476a5e3d57f15c
Key Id Hash(md5): 8d5bf656bb34465654818097fa9faa4b
Key Id Hash(sha256): ce3b58a5d786aef72e103b9f82817e3c3a2cbd316d9312b2c5febd5ddd444372
Key Id Hash(pin-sha256): TYhKbOs15qbXQyh5CACY1jAET6cNscJtBcbJpV9VhUg=
Key Id Hash(pin-sha256-hex): 4d884a6ceb35e6a6d7432879080098d630044fa70db1c26d05c6c9a55f558548
Cert Hash(md5): e1356d342bda5deb3a0b882fee1e74ea
Cert Hash(sha1): c9b4a907d2117b9aea8915bf2cf7d6c1508587ae
Cert Hash(sha256): 743c221a7c085f930d7bf3e87d6ba58e0c62f82267c0f2881d73bb82c8189efa
Signature Hash: 91ac8133265c2b97f8ac190da3b2004e0aae614c

CERT_REQUEST_ORIGINATOR_PROP_ID(71):

CERT_KEY_PROV_INFO_PROP_ID(2):
Key Container = 8dd6d20c579fded1532ba9e8f570432f_95cdac0e-e168-4a2c-bc67-c551fa66d6df
Simple container name: te-1a4f1abf-0ffd-45f1-8351-e197453ce876
Provider = Microsoft Enhanced Cryptographic Provider v1.0
ProviderType = 1
Flags = 20 (32)
CRYPT_MACHINE_KEYSET -- 20 (32)
KeySpec = 1 -- AT_KEYEXCHANGE

CERT_SUBJECT_PUB_KEY_BIT_LENGTH_PROP_ID(92):
0x00000800 (2048)

CERT_SHA1_HASH_PROP_ID(3):
c9b4a907d2117b9aea8915bf2cf7d6c1508587ae

CERT_SUBJECT_PUBLIC_KEY_MD5_HASH_PROP_ID(25):
8d5bf656bb34465654818097fa9faa4b

CERT_KEY_IDENTIFIER_PROP_ID(20):
b1147f813d2fcde4504f16435b14c878382b5384

CERT_SIGNATURE_HASH_PROP_ID(15) disallowedHash:
91ac8133265c2b97f8ac190da3b2004e0aae614c

CERT_MD5_HASH_PROP_ID(4):
e1356d342bda5deb3a0b882fee1e74ea

CERT_ACCESS_STATE_PROP_ID(14):
AccessState = 6
CERT_ACCESS_STATE_SYSTEM_STORE_FLAG -- 2
CERT_ACCESS_STATE_LM_SYSTEM_STORE_FLAG -- 4

Provider = Microsoft Enhanced Cryptographic Provider v1.0
ProviderType = 1
Simple container name: te-1a4f1abf-0ffd-45f1-8351-e197453ce876
RSA
PP_KEYSTORAGE = 1
CRYPT_SEC_DESCR -- 1
KP_PERMISSIONS = 3f (63)
CRYPT_ENCRYPT -- 1
CRYPT_DECRYPT -- 2
CRYPT_EXPORT -- 4
CRYPT_READ -- 8
CRYPT_WRITE -- 10 (16)
CRYPT_MAC -- 20 (32)

D:PAI(A;;GAGR;;;BA)(A;;GAGR;;;SY)

Allow Full Control  BUILTIN\Administrators
Allow Full Control  NT AUTHORITY\SYSTEM

Private Key:
PRIVATEKEYBLOB
Version: 2
aiKeyAlg: 0xa400
CALG_RSA_KEYX
Algorithm Class: 0xa000(5) ALG_CLASS_KEY_EXCHANGE
Algorithm Type: 0x400(2) ALG_TYPE_RSA
Algorithm Sub-id: 0x0(0) ALG_SID_RSA_ANY
0000 52 53 41 32 RSA2
0000 ...
048c
Encryption test passed

[mimckitt]

Thanks. Looking into it. Will update shortly.

[dragav]

@rjmille2 thanks for the details. I assume you are still seeing the same errors in the event log as before?

I appears the problem is one of access to the certificate's private key; I would have expected a different error code, but let's go with that.

The ACL on the certificate's private key shows Fabric would not have access to it; we ACL certificates declared in the cluster or application manifests. This cert, in particular, should be declared as a Secrets Certificate in the application manifest, as follows:

An alternative/quick fix would be to manually ACL the cert's private key to SFAdministrators (if you're using encrypted environment variables) or SFAllowedUsers (if you're using encrypted settings).

@MicahMcKittrick-MSFT please open an internal doc bug against aljo to fix this: Manage encrypted settings in SF apps; it's missing the SecretsCertificate reference.

[mimckitt]

@rjmiller-revint @dragav is there any further action in this issue?

[rjmiller-revint]

I haven’t had a chance to try it again yet. I’ll try to do that today.

From: Micah [e-mail]
Sent: Monday, April 29, 2019 10:47 AM
To: MicrosoftDocs/azure-docs [e-mail]
Cc: Randy Miller [e-mail]; Mention [e-mail]
Subject: Re: [MicrosoftDocs/azure-docs] Can Service Fabric configuration settings be encrypted on a local dev cluster? (#29612)

@rjmiller-revinthttps://github.com/rjmiller-revint @dragavhttps://github.com/dragav is there any further action in this issue?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/29612#issuecomment-487676937, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALBPN3GXS572GLZ2QFMDFYDPS4YA3ANCNFSM4HGX3MJQ.
Confidentiality Notice: This email message and any accompanying documents is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, distribution, or copying is prohibited. If you are not the intended recipient, please contact our office by email or telephone at 1-855-252-7606 and immediately destroy all copies of the original message.
Confidentiality Notice: This email message and any accompanying documents is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, distribution, or copying is prohibited. If you are not the intended recipient, please contact our office by email or telephone at 1-855-252-7606 and immediately destroy all copies of the original message.

[mimckitt]

Hi @rjmiller-revint any luck?

[mimckitt]

@rjmiller-revint I will close this for now. If you try the steps again and still have issues just let us know and we can always reopen and continue the discussion.