Openenclave: Remote attestation testing

[wintersteiger]

I'm trying to test remote attestation, but I'm not on an Azure VM (local installation with prototype machines). I have the az-dcap-client installed, but it fails to produce usable certificates, causing sgx_qe_get_target_info to fail. Is there a way to switch to a debug-client or a way to populate AZDCAP_BASE_CERT_URL and AZDCAP_CLIENT_ID with local certificate information?

[Liaojinghui]

I got some problem with the remote attestation testing too. My device can run the intel/sgx-ra-sample in hardware, which means my device is capable to do remote attestation, but when I run the remote attestation sample of openenclave, I got error
21Enclave: ***/home/compass/LNSGX/SGX/SGXEVM/openenclave/samples/remote_attestation/common/attestation.cpp(55): oe_get_report failed.
Also, may I ask how to setup my SPID, Primary key and Secondary key into remote attestation sample?

[soccerGB]

@wintersteiger
Could you run the remote attestation sample with environment OE_LOG_LEVEL=INFO and share our the output?

[wintersteiger]

Of course!

Without az-dcap-client installed:

cwinter@msrc-cc03:~/openenclave-master/samples/remote_attestation/build$ OE_LOG_LEVEL=INFO make run
...
Host: Enclave library /home/cwinter/openenclave-master/samples/remote_attestation/build/enclave_a/enclave_a.signed
12:18:01:362763 tid(0x7f84fef49d40) (H)[INFO]Processor supports AVX instructions[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/xstate.c _is_xgetbv_supported:33]
12:18:01:362777 tid(0x7f84fef49d40) (H)[INFO]Value of XFRM to be set in enclave is 7
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxload.c _detect_xfrm:135]
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(76): mbedtls initialized.
Host: Enclave successfully created.
Host: Enclave library /home/cwinter/openenclave-master/samples/remote_attestation/build/enclave_b/enclave_b.signed
12:18:01:518866 tid(0x7f84fef49d40) (H)[INFO]Processor supports AVX instructions[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/xstate.c _is_xgetbv_supported:33]
12:18:01:518869 tid(0x7f84fef49d40) (H)[INFO]Value of XFRM to be set in enclave is 7
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxload.c _detect_xfrm:135]
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(76): mbedtls initialized.
Host: Enclave successfully created.
Host: requesting a remote report and the encryption key from 1st enclave
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/enclave_a/ecalls.cpp(43): enter get_remote_report_with_pubkey
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/dispatcher.cpp(96): get_remote_report_with_pubkey
12:18:01:700971 tid(0x7f84fef49d40) (H)[INFO]oe_load_quote_provider libdcap_quoteprov.so
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:25]
12:18:01:701083 tid(0x7f84fef49d40) (H)[ERROR]sgxquoteprovider: libdcap_quoteprov.so not found 
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:77]
12:18:01:701087 tid(0x7f84fef49d40) (H)[ERROR]oe_initialize_quote_provider failed (oe_result_t=(null))[OE_QUOTE_PROVIDER_LOAD_ERROR /home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c:1519549348]
12:18:01:701089 tid(0x7f84fef49d40) (H)[ERROR]:OE_QUOTE_PROVIDER_LOAD_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/quote.c sgx_get_qetarget_info:156]
12:18:01:709852 tid(0x7f84fef49d40) (H)[ERROR]quote3_error_t=0xe00f
 (oe_result_t=OE_PLATFORM_ERROR)[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquote.c oe_sgx_qe_get_quote_size:35]
12:18:01:709857 tid(0x7f84fef49d40) (H)[ERROR]:OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/quote.c sgx_get_quote:201]
12:18:01:709886 tid(0x7f84fef49d40) (E)[ERROR]enclave_a.signed::OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/enclave/core/sgx/report.c oe_get_remote_report:237]
12:18:01:709895 tid(0x7f84fef49d40) (E)[ERROR]enclave_a.signed::OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/enclave/core/sgx/report.c _oe_get_report_internal:311]
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/attestation.cpp(54): oe_get_report failed.
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/dispatcher.cpp(137): get_remote_report_with_pubkey failed.
Host: verify_report_and_set_pubkey failed. OE_OKHost: Terminating enclaves
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(90): mbedtls cleaned up.
Host: Enclave successfully terminated.
Enclave: ***/home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(90): mbedtls cleaned up.
Host: Enclave successfully terminated.
Host:  failed 
...
make: *** [run] Error 2

With az-dcap-client:

```
cwinter@msrc-cc03:~/openenclave-master/samples/remote_attestation/build$ OE_LOG_LEVEL=INFO make run
...
Host: Creating two enclaves
Host: Enclave library /home/cwinter/openenclave-master/samples/remote_attestation/build/enclave_a/enclave_a.signed
12:18:52:329424 tid(0x7f1bbcf55d40) (H)[INFO]Processor supports AVX instructions[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/xstate.c _is_xgetbv_supported:33]
12:18:52:329440 tid(0x7f1bbcf55d40) (H)[INFO]Value of XFRM to be set in enclave is 7
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxload.c _detect_xfrm:135]
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(76): mbedtls initialized.
Host: Enclave successfully created.
Host: Enclave library /home/cwinter/openenclave-master/samples/remote_attestation/build/enclave_b/enclave_b.signed
12:18:52:824771 tid(0x7f1bbcf55d40) (H)[INFO]Processor supports AVX instructions[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/xstate.c _is_xgetbv_supported:33]
12:18:52:824790 tid(0x7f1bbcf55d40) (H)[INFO]Value of XFRM to be set in enclave is 7
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxload.c _detect_xfrm:135]
Enclave: */home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(76): mbedtls initialized.
Host: Enclave successfully created.
Host: requesting a remote report and the encryption key from 1st enclave
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/enclave_a/ecalls.cpp(43): enter get_remote_report_with_pubkey
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/common/dispatcher.cpp(96): get_remote_report_with_pubkey
12:18:52:988939 tid(0x7f1bbcf55d40) (H)[INFO]oe_load_quote_provider libdcap_quoteprov.so
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:25]
12:18:52:992358 tid(0x7f1bbcf55d40) (H)[INFO]sgxquoteprovider: provider.get_revocation_info = 0x7f1bba60a5f6
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:37]
12:18:52:992363 tid(0x7f1bbcf55d40) (H)[INFO]sgxquoteprovider: provider.free_revocation_info = 0x7f1bba60b6a6
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:40]
12:18:52:992365 tid(0x7f1bbcf55d40) (H)[INFO]sgxquoteprovider: Installed log function
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:47]
12:18:52:992368 tid(0x7f1bbcf55d40) (H)[INFO]sgxquoteprovider: provider.get_qe_identity_info = 0x0
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:67]
12:18:52:992370 tid(0x7f1bbcf55d40) (H)[INFO]sgxquoteprovider: provider.free_qe_identity_info = 0x0
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/linux/sgxquoteproviderloader.c oe_load_quote_provider:70]
12:18:53:002143 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [INFO]: Using default base cert URL 'https://global.acccache.azure.net/sgx/certificates'.
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:002155 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [INFO]: Using default client id 'production_client'.
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:002227 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [INFO]: Fetching quote config from remote server: 'https://global.acccache.azure.net/sgx/certificates/2608b9fafbba0e5ffdf4b2fe44e1d3a3/0202ffffff8002000000000000000000/0700/0000?clientid=production_client&api-version=2018-10-01-preview'.
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:467887 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [ERROR]: Encountered CURL error 22 in curl_easy_perform
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:468582 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [ERROR]: error thrown, error code: 16: curl_easy_perform
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:468614 tid(0x7f1bbcf55d40) (H)[ERROR]quote3_error_t=0xe011
(oe_result_t=OE_PLATFORM_ERROR)[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquote.c oe_sgx_qe_get_target_info:21]
12:18:53:468625 tid(0x7f1bbcf55d40) (H)[ERROR]:OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/quote.c sgx_get_qetarget_info:157]
12:18:53:468969 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [INFO]: Using default base cert URL 'https://global.acccache.azure.net/sgx/certificates'.
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:468997 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [INFO]: Using default client id 'production_client'.
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:469103 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [INFO]: Fetching quote config from remote server: 'https://global.acccache.azure.net/sgx/certificates/2608b9fafbba0e5ffdf4b2fe44e1d3a3/0202ffffff8002000000000000000000/0700/0000?clientid=production_client&api-version=2018-10-01-preview'.
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:568991 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [ERROR]: Encountered CURL error 22 in curl_easy_perform
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:569060 tid(0x7f1bbcf55d40) (H)[INFO]dcap_quoteprov: [ERROR]: error thrown, error code: 16: curl_easy_perform
[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquoteprovider.c oe_quote_provider_log:35]
12:18:53:569071 tid(0x7f1bbcf55d40) (H)[ERROR]quote3_error_t=0xe011
(oe_result_t=OE_PLATFORM_ERROR)[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/sgxquote.c oe_sgx_qe_get_quote_size:35]
12:18:53:569075 tid(0x7f1bbcf55d40) (H)[ERROR]:OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/host/sgx/quote.c sgx_get_quote:201]
12:18:53:569363 tid(0x7f1bbcf55d40) (E)[ERROR]enclave_a.signed::OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/enclave/core/sgx/report.c oe_get_remote_report:237]
12:18:53:569392 tid(0x7f1bbcf55d40) (E)[ERROR]enclave_a.signed::OE_PLATFORM_ERROR[/home/jenkins/work_dir/workspace/OpenEnclave-v0.6.x_packages/enclave/core/sgx/report.c _oe_get_report_internal:311]
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/common/attestation.cpp(54): oe_get_report failed.
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/common/dispatcher.cpp(137): get_remote_report_with_pubkey failed.
Host: verify_report_and_set_pubkey failed. OE_OKHost: Terminating enclaves
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(90): mbedtls cleaned up.
Host: Enclave successfully terminated.
Enclave: /home/cwinter/openenclave-master/samples/remote_attestation/common/crypto.cpp(90): mbedtls cleaned up.
Host: Enclave successfully terminated.
Host: failed
12:18:53:579379 tid(0x7f1bbcf55d40) (H)[INFO]_unload_quote_provider libdcap_quoteprov.so
...
make: ** [run] Error 2

[radhikaj]

@soccerGB could you please follow up on this?

[yentsanglee]

I got some problem with the remote attestation testing too. My device can run the intel/sgx-ra-sample in hardware, which means my device is capable to do remote attestation, but when I run the remote attestation sample of openenclave, I got error
21Enclave: ***/home/compass/LNSGX/SGX/SGXEVM/openenclave/samples/remote_attestation/common/attestation.cpp(55): oe_get_report failed.
Also, may I ask how to setup my SPID, Primary key and Secondary key into remote attestation sample?

'SPID', 'Primary key' and 'Secondary key' are for Intel EPID attestation. OE SDK does not need them.

[yentsanglee]

@wintersteiger

We are aware of the issue that OE SDK remote attestation does not work no non-Azure VMs. We are fixing the issue.

[yufjiang]

The root cause was that the OE was deeply coupled with Azure DCAP client. Azure DCAP client retrieves certs from Azure PCK certs caching service which only stores the certs for Azure Confidential Computing (ACC) nodes. The non-ACC machines' remote attestation will fail because their certs cannot be retrieved.

This issue has been resolved as we have supported the Intel DCAP client. With proper setup, any SGX-DCAP-enabled machine is able to retrieve the cert. The document was also added. Please find the document at https://github.com/openenclave/openenclave/blob/master/docs/GettingStartedDocs/Contributors/NonAccMachineSGXLinuxGettingStarted.md

@wintersteiger and @Liaojinghui, in case you still need to run workloads on non-Azure machines, you might try it by following that document. Thanks!

[yentsanglee]

Issue resolved.