Azure-docs: Unable to verify certificates

[MaWa19]

I'm trying to set up a Raspberry Pi 3 as a transparent gateway following this How-to. Everything seems to work until it comes to the verification of certificates.

Testing the connection with openssl leads to "unable to get local issuer certificate" and the following log. The returned error is the same when trying to connect to the IoT Hub instead of the gateway (localhost).

One thing I noticed when inspecting the certs in /etc/ssl/certs is that the filename extension of the CA cert is added twice: "azure-iot-test-only.root.ca.cert.pem.pem". However, I'm not sure if thats relevant. Please tell me if I need to provide any more information.

openssl s_client -connect localhost:8883 -CAfile /home/pi/work/certs/azure-iot-test-only.root.ca.cert.pem -showcerts
CONNECTED(00000003)
depth=2 CN = Test Edge Device CA
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/CN=iotedgegateway
   i:/CN=iotedged workload ca
-----BEGIN CERTIFICATE-----
MIIDwzCCAaugAwIBAgIEOmkVejANBgkqhkiG9w0BAQsFADAfMR0wGwYDVQQDDBRp
b3RlZGdlZCB3b3JrbG9hZCBjYTAeFw0xODA4MTQxNDMxMzhaFw0xODExMTIxNDMw
MzhaMBkxFzAVBgNVBAMMDmlvdGVkZ2VnYXRld2F5MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAvt/im65NNFPOZG2RS2v5CRNvhocLnWU5Q+lx5BqQ1zUD
/5OTiLsPKB7gUI+6SessX/vgfG+phnw70PadhtxAaMV27uSvLFaV/w7A61zrFCw3
EM1yxAjWqgqYzTNWviozQ0I7pfEeTDubY7rqx5Whp2ke+bljIY4+zCTb7BcpBQZk
bt/IX81R05FKZXTwW9EjRujlFwBfTt6Gc1/c0KlgWdnzG8lAxhNLv226gUSXK6am
kkGJuxg0XmMeefKNTBBxCz1Uk3p8K1XUcetdCvJGwhO6Ma7CpquqfV6cDTjHOioK
KznHAXR7Bm19la+CCl6QhkW3twdOUAKiMWrp0qZJ+QIDAQABow0wCzAJBgNVHRME
AjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAcEWVeVMdiVu24YtRD+LqqYjSwjBA4ONJC
7Q7Pw0XMmIpZXhx+zNeV20Wbeqfdw/wWMek76c+Wxo7T+LaacHKD11OeDzghPjoS
o3XLiCWbmk1hYSeUq362ruOEliiTq48f8qcYQM8mnfVPeDUdB/u0LXSDhBw22Zfi
yF/WjvhbfMu2dSxAKn5iPJZUDs2uYTtJG8a+n0nNPOle7gb7QVjy/dCtrXBRkldk
fW0zzvCRfxwZGAjsDxlJhw9NnOu15tLgyQ2hZK8isnqIAwp25qT3YQK4+jYOyZSg
kXc0Lhvr/bE/FuTCGA+YqnUu3hJmHg7tC0CW5qaKUD+4NdRguCXagIIv1oI9yVJ7
KOU2+Rkkc9ht5PMYk2mjcwVI7Tks4j7dy8izVHQiAo/+furMFxqWy9WR5TZ8GRw9
dGgjzL/ayENab0onm15L9GVjMuBXB9PtA1aoODyNOIk0+X2P6OxMEVcwnHmWIqwY
ZcqliiNRokZyG7sr0SXW3zI8R4u2a0nfHxiYgKHbyOiJ7s0VAwXUW4l0Ey6l9kd4
7ZNxwV3nbQ2MsWePRvBnSEl9BLtHts4z/T57TnZOsPfOsfFCPKRaQZNi23umw+0x
cymGKiWwAl5fwpTDQee4s9a6DZGhVHf9HlWiit5+d+1LFQLAZbWT1V0nkyx159QI
zeZAuA7YIw==
-----END CERTIFICATE-----
 1 s:/CN=Test Edge Device CA
   i:/CN=Test Edge Owner CA
-----BEGIN CERTIFICATE-----
MIIEzzCCAregAwIBAgIEMnsjxjANBgkqhkiG9w0BAQsFADAdMRswGQYDVQQDDBJU
ZXN0IEVkZ2UgT3duZXIgQ0EwHhcNMTgwODE0MTQzMDQ0WhcNMTgxMTEyMTQzMDM4
WjAeMRwwGgYDVQQDDBNUZXN0IEVkZ2UgRGV2aWNlIENBMIICIjANBgkqhkiG9w0B
AQEFAAOCAg8AMIICCgKCAgEAyju2mdNAddvZdIbWQNxPciR572tk2tlcb1UiHY0S
Tlt5e7mNDpbA5+KkspsYnSkFwcw/vDiNXEYrXIYDe5j/GMrkePYLuOgR4rBq+2DS
slTQvcM6eN7fR2FqfPrYJ45+oeIO2LOu86B9wYjF53jPz2FtEdnYHwxxCZNgn5n0
EJ8TFjP77mLV65vPA8e8ARIKP2bpRpbYsSSSwoKomo7P6wuf4Owxh8obd9IkeFVb
tZWJQ2E90H1QP8GiXOs1Fs+Q2YqjzWU456GkH7aSPfYijTjaHHTxPy4hQHciRfpZ
j0CjC77MXI3YC0Zi4BeRHOApaAtUgldMmHGTb0LJoqpeWiWsUR9LfXOBafYScRdZ
mFP+Ey6v6VL0teFNqc6wgThdIEWwHHb+hMx4Slr87IbOr+vbMMNptZ7/Z3tUxILx
1eRyhmzyh+MGJBx/lxY5oMD0vgrfQNx6tcCsGDHmBxUXIJrPqsodk+PXPT4PQnRT
CkNYiNUyJSmlGl/mFQGTYWKcOI/LPf/8bIqJF3DKlVuIuM8hWVhJSdcjEq/+e+AY
E3+BxV4FGvdtOQJeSvXgTqIJxyUXgx1Z5dMmAUT5E9c+NmFQlm0IH2wDcogg9jQD
mUj164vsl84Ce1taflj49uRL6HT/PPT9XXcniGuWHM6ArDsVM6ZQYo7HMsQkF3Dv
bVECAwEAAaMWMBQwEgYDVR0TAQH/BAgwBgEBAQIBAjANBgkqhkiG9w0BAQsFAAOC
AgEAmdBLyHY8DfccPSvRQrwSHUQTu9Z0/aaSIS6Ngk3zjyodY/4piNsZnW/XppT+
LDXbtwDM70WXflOEZkbSO2bSA2zfCbcp2KLn/ZwTY8eM7RQr/rH7+L+LRlUa54We
SqpPXHuZe1mIji8adQC0hF+I8x6ofNwhbpwXmbIeAGolCBTY/tm8mCwmDkeRLCRY
+9DtPkC03wpzlQJGUeCUd9eEAYs+YCF6kKmEnN0Rq1ceNf/nYInzPLjlU0fwNaU7
lHNzoAKUIW+oNMMp0Qi8zgAzMTOEAbAUOvwlvSypc7nBPKF+T2gY65U4wpbLV3PU
4S2Apt4PdybvpuLIGfOYlm2sEQyAQJw45nU0/dKPNRgbeGmsKN4DCfPhK6uGuv92
Zpu9ycf48lz4SmAInkxengCNVSj2QPuazbAR6mXHrDf+KlmTnR5UD9/I4WKYZG4g
bDUVTg03a1rUKre3jnaaytVGxtegukfK1bikH45lPaA/6tl+W4LubztFDRKHj12k
1C2tyawLYaLn8U5R1WiQBxEf/MiZjYm38+5EWPn5FulUp54sSqNnGOVsIey0NM4+
J3PZz7wxC3R+JwyRjhN+JSwiUhpwF0WaqC1FLPr6sA8w5GzrgwHkpl6nK4hdjxzT
f4TZKAElsdW9NUgGOpIDYPhochDov7dVZMSNp17pcNJWeXE=
-----END CERTIFICATE-----
 2 s:/CN=iotedged workload ca
   i:/CN=Test Edge Device CA
-----BEGIN CERTIFICATE-----
MIIE0TCCArmgAwIBAgIEHaeK8jANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDDBNU
ZXN0IEVkZ2UgRGV2aWNlIENBMB4XDTE4MDgxNDE0MzEwNFoXDTE4MTExMjE0MzAz
OFowHzEdMBsGA1UEAwwUaW90ZWRnZWQgd29ya2xvYWQgY2EwggIiMA0GCSqGSIb3
DQEBAQUAA4ICDwAwggIKAoICAQDJBVJvwX7XAlj1S5/kWkGRCgCKKhA7+FCJmlUv
+NELdQCEERU73Szm3hf1BOSSZVj3zX3SKbrw7+nOAHTYwvENpMfszmlutodQbnS7
4U0sQdSVWdHY1jyjG1bB0rgzpcwJFrBsPJtYaA/SnLkmp94c75wAJ//HCa12Xzru
LGZ7RKChFuTrChplj0SjxWtDkGlDJoAAfk9DkPBVcYdIPLkLl6fvQ/MZtEb1RCM1
VlOMjutBsSWi9F42oMl1b1fU5xtij9fGhE3+OOIUlQZeg9lyUppls7sLL5U5yjFN
UjmOkm6RD5gS8BJKdDpVOm7Q+zkAs4zuU4K5o9akvn5JO3RHcI1/gRMnqq4x1hv5
AB48Gv/PkSwRVp/DYofJUy4nUpsOR29IzZ7tDFd4/Q8uOqignvhwL9zPgE1fO8LP
AM7r4V3n0sg1lgBIg/dimzyrIRDu4yc82QIPIGXxOMXK1Soqai/wBnrG9P7Y+vss
Eri87jO8ms13GZw1qvAmLMKP2zL3ymQpP2uP6pw06lqM3BYo6s4VaavEXevY4HiM
vUN476fE6qGkPacGN+aNpwafCwaIcuryFAmiZjOGiJ3j8fPU/Gquns/eVIdiQeb2
hiIXQPWV3GcPqqT+1y600pYVEz9nlwmYtSYgpsqDQstdBB28IqePKKXC+CIMo4+k
2GzsiwIDAQABoxYwFDASBgNVHRMBAf8ECDAGAQEBAgEAMA0GCSqGSIb3DQEBCwUA
A4ICAQBXcrPzLgQm/Jdj57AzvvT6PK//10CFGNdYi4/E7cIM3xok+YRgP0e+6VTh
M5jg+8wjHokW7MTs6FRTQS6d5jObFGBO29AANMWSZNeRsU/aXTGK5XKuXM3UcZGk
x4p0lQYsaNPEiVjtZL2PeZ3+KNldAO2cXhWGiXIBQiYe7VBRI9VSw/wkH3VP7WtG
NOaPFZ3vYmCqUUwSsgDQ8La3Bwj+0X1gTnyqG6yMsUhWWk0YsnN+0PsQ/n8/8nat
4eb/nh9X98jU88506Fd2yZaRj4NYG5RvS0kQqyDYSbODsipKwVtTpgeS2Ozm6QGv
a6Z8RPn77TNeLJHyO2SyvhQpetowsYdPLP2uSvFoT6kdqAnU3Ttr4ehTBCaccDjQ
3p4wOxInVqOzreO+zLlMIoTCNGZN1HfeF0MRiXb2jyBE+pJHOeaYZFdaigW+o5Hv
1V1eX5V1mThnGnsweqrIRdo1VXYWWHpDeRaCi9SBzqp0t1IV37NqCe0fpXPNgxmj
CGGREnMn+Fk6tmBK19ni5RBONTV6+RyOBs2CdClIjLflPWi1bttres4qDvFI2B8M
fKW6/3K6bA3ZXPtLMiTJdhikrtN/WBEtibBuPt/MxpD7zQBXnxYjPbF9FsVeF8R5
4/nHDHPTSrcD4zWM7aN4LRm59VYWJsMYN+/PKlEVtmoCNOBAFA==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=iotedgegateway
issuer=/CN=iotedged workload ca
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4099 bytes and written 302 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 7947C8E2DF4B97416F4BEFFE633149BBB2755C01F93D136C828B732CBBB138A4
    Session-ID-ctx:
    Master-Key: 5BC2A4C12B44B423AFA1C7C8E088DA784029FBD34CE898EF67D55158A12F3D670902CD1B8D975522109459E244856AA2
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - b5 ac b9 47 89 49 05 8d-90 5c dc 3a c0 00 20 05   ...G.I...\.:.. .
    0010 - e5 64 a3 38 f8 d2 02 ae-fb 72 d8 e6 9b 2a 47 e2   .d.8.....r...*G.
    0020 - 70 c8 ed 32 73 34 bb 3d-0e 85 95 1f a3 a5 e9 b5   p..2s4.=........
    0030 - 04 aa 6f 74 a6 01 ad 64-5e 41 5d ee 97 55 83 f3   ..ot...d^A]..U..
    0040 - c7 8b 0f 82 81 fe f2 4e-1e 98 23 e9 1f 52 6e c2   .......N..#..Rn.
    0050 - 85 be 8f 43 85 da fd d3-87 22 4c 24 6d ad c0 b1   ...C....."L$m...
    0060 - 1f bf de d1 75 f7 ac 64-e4 67 f6 53 d2 b9 17 a8   ....u..d.g.S....
    0070 - 96 98 f2 25 bb 7d 2f d0-67 ec f5 c5 62 4a 89 0c   ...%.}/.g...bJ..
    0080 - b0 39 37 35 f8 ed 36 ac-6b ee 87 52 33 ed ed 9c   .975..6.k..R3...
    0090 - e2 a9 5d 7d e1 c5 d4 49-ca d9 1e 5d ec 45 3c 15   ..]}...I...].E<.

    Start Time: 1534259155
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
---
read:errno=0

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 71acc49c-a23e-493f-c3f5-4ec46846616c
• Version Independent ID: bd3abb8d-1b48-0efd-f6a4-1b3f7a12c039
• Content: Create a transparent gateway with Azure IoT Edge - Linux
• Content Source: articles/iot-edge/how-to-create-transparent-gateway-linux.md
• Service: iot-edge
• GitHub Login: @kgremban
• Microsoft Alias: kgremban

[TravisCragg-MSFT]

@MaWa19 Thanks for the feedback. We are actively investigating and will get back to you soon.

[asergaz]

@MaWa19 can you please follow the instructions from here and let us know if you were successfull? Thank you.

[MaWa19]

@sergaz-msft Thanks for your quick reply. Unfortunately the linked instructions didn't solve the problem. In fact, I did the proof of ownership already while trying to solve this issue. Are there any debug files I could inspect to see where the error is coming from?

[asergaz]

@MaWa19 you are welcome.

In fact, I did the proof of ownership already while trying to solve this issue.

What is the error that you see now? When you perform "openssl s_client -connect localhost:8883 -CAfile /home/pi/work/certs/azure-iot-test-only.root.ca.cert.pem -showcerts" you should see "Verify return code: 0 (ok)"

What is the ‘hostname’ in config.yaml ? It should be "localhost" in order for it to work.

[stevebus]

I would actually recommend using something other than 'localhost' for the hostname in config.yaml. That name in config.yaml should be whatever you plan to use for your downstream 'leaf' devices to connect (in other words, whatever you want your "GatewayHostName" parameter to be if you are using our SDKs), and that's unlikely to be 'localhost' (in fact, that might cause some really weird problems). You should name it something totally different, like "mygateway.local" or something like that. If you don't have a real DNS server, you may have to add an entry into your hosts file (/etc/hosts on Linux and \windows\system32\drivers\etc\hosts on Windows) to resolve the name.

Once you update your hostname in config.yaml, you'll need to have Edge regenerate your TLS certs. there's currently a bug in this process that would cause edge to not regenerate your TLS certs, so run the following commands to restart Edge after updating your hostname in config.yaml

sudo rm -rf /var/lib/iotedge/hsm/certs
sudo rm -rf /var/lib/iotedge/hsm/cert_keys
sudo systemctl restart iotedge

Let us know if that helps with the cert validation

[MaWa19]

@sergaz-msft the error was the same as before..

@stevebus I guess the error was in fact related to the hostname. Currently I don't have a real DNS server since this is just an exemplary implementation for now. Today I got a little bit further and was able to establish a connection between the leaf device and the gateway using the owner certificate in /var/lib/iotedge/hsm/certs. I also changed the gateway hostname within the certificate creation to the IP address of the gateway. However, I was still not able to set up a connection using the azure-iot-test-only.root.ca.cert.pem.

How are the certs in /var/lib/iotedge/hsm/certs related the the ones I created along the process? I won't be able to test your suggestions before Sunday but they sound promising so far.

[stevebus]

This is an implementation detail that is always subject to change, but regarding how the certs are related.... the certificate that you create with the certGen create_edge_device_cert command actually created an intermediate certificate. That intermediate certificate, which really you can think of as being a 'machine level' cert is used, by the iotedge runtime (specifically, the iotedge daemon) to generate another cert that we refer to as the "workload CA cert". That cert is used internally by the edge runtime for several thing, one of which is to then generate the actual cert that is presented by edgeHub to connecting downstream leaf devices (let's call that the 'edgeHub server cert'). So that final cert is the cert that is presented to clients when they try to establish a TLS connection with edgeHub. When the edge runtime generates that cert, it uses the 'hostname' entry in the config.yaml file as the Common Name (CN) of the edgeHub server cert. So, the hostname in config.yaml must be the same name that your downstream leaf devices will use to connect to IoT Edge (for example, in the GatewayHostName parameter on the connection string if you are using our SDKs) and, as an aside, the name that you would use in your openssl command above. So, you can't use localhost for that name as downstream devices would not be able to resolve localhost to the Edge device.

So, the certificate chain is

root ca cert* -> intermediate ca cert* -> device ca cert* -> workload cert^ -> edgeHub server cert

• 
  
  • generated for dev/test using our developer scripts (certGen.sh)
    
    ^ - generated under the covers by the edge security daemon
  
  

does that make sense?

If this is a POC/sample/etc (i.e not production, in production you should use a real FQDN), you can just make up a name to use in your hostname entry in config.yaml. For example, mygateway.local or something like that. then, in the absence of real DNS you can add an entry in your hosts file (/etc/hosts on Linux, \windows\system32\drivers\etc\host on Windows) on both your edge box (so you can test with your openssl command) and then on your downstream leaf device. Just make sure you can ping that name from the various boxes after you make the entry.

[asergaz]

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.