Aws-sdk-cpp: Deprecated OpenSSL APIs

[jkuszmaul]

I'm trying to pull parts of the SDK into our build system so that we can make use of it. In trying to get things running, I tried seeing if I could compile against boringssl instead of OpenSSL, hoping that it would work. It didn't but the errors I encounter seem to suggest that this library is using currently deprecated parts of the OpenSSL API.
I ran into issues where the following symbols were not defined in aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:

OPENSSL_add_all_algorithms_noconf
EVP_MD_CTX_FLAG_NON_FIPS_ALLOW
EVP_CTRL_CCM_GET_TAG

Some of these appear to be artifacts of attempting to use boringssl; however, ERR_load_crypto_strings appears to be deprecated, and while poorly documented, it appears that EVP_MD_CTX_FLAG_NON_FIPS_ALLOW may be a nop. The other two look like issues that are probably a result of using boringssl, although from glancing at boringssl's code they probably aren't doing anything.

I'm not at all familiar with interfacing with OpenSSL or any of these flags/macros/functions, so there may be some good reason to continue using them; on the other hand, it would also be a nice property to have this library fully compatible with boringssl.

This is all in version 1.7.148 of the AWS C++ SDK.

[jkuszmaul]

The EVP_CTRL_CCM_GET_TAG define is an issue with an old version of boringssl on my part; the rest still stand, although from looking at OpenSSL, you may want to prefer EVP_CTRL_AEAD_GET_TAG, I'm having trouble telling.

[jkuszmaul]

Just to add some notes, for future reference (not sure if they'll help with any changes to the code-base, but if anyone else is trying to use boringssl):
I am able to use at least some basic parts of the SDK with boringssl without issue (I haven't tested anything more complex, so there may be some hidden subtle issues), by making the following changes:

• Commenting out ERR_load_CRYPTO_strings (because it is deprecated, and boringssl in general doesn't tend to need this sort of intitialization)
• Commenting out OPENSSL_add_all_algorithms_noconf, because https://github.com/google/boringssl/commit/83042a829260f53b0f0f49e06d33bc3a73401bf0 makes me think that these initialization functions are unnecessary for boringssl, because it manages initialization slightly differently than OpenSSL.
• Commenting out the lines that set the EVP_MD_CTX_FLAG_NON_FIPS_ALLOW flag because, as I already mentioned, it appears to be a nop.
• Changing EVP_CTRL_CCM_GET_TAG to EVP_CTRL_AEAD_GET_TAG because they seem to refer to the same number.

[KaibaLopez]

Hi @jkuszmaul, thanks for bringing this up and we appreciate the suggestions.
We are planning on doing some cleanup for these kind of functions that are deprecated for openssl, I cannot say about making the library compatible with boringssl though, that's a bit out of scope at the moment.
I'll keep the issue open for now as a reminder and to consider your suggestions on boringssl.

[jkuszmaul]

Sounds good. I mostly bring up compatibility with boringssl because (a) it would make me not have to maintain a slight fork of upstream and (b) ostensibly it shouldn't be too hard. I've also filed a bug against boringssl to see if they'll update their libraries to handle these symbols correctly.

[jkuszmaul]

@KaibaLopez FYI I got a response from upstream boringssl about these symbols, see here--basically, they may add the OPENSSL_add_all_algorithms_noconf symbol, and suggest that the rest of the symbols being used here should probably be used differently, even if just sticking to OpenSSL (see the comment I linked).

[KaibaLopez]

Hi @jkuszmaul,
I've implemented some of these changes on version 1.7.155 of the SDK.
In short:

• ERR_load_CRYPTO_strings is now only used if the openssl version is below 1.1.x, otherwise we use OPENSSL_init_crypto as recommended.

• Changed EVP_CTRL_CCM_GET_TAG to EVP_CTRL_GCM_GET_TAG.

Hope this helps to make our sdk more compatible, and thanks again for bring this up to us.

[jkuszmaul]

Thank you!

[DennisHeimbigner]

I am using branch 1.7.9 and am still seeing these failures using openssl 1.1.1
under ubuntu 18.
Is there a specific branch that has these fixes so I can recommend to my users?

[KaibaLopez]

not really, it should've been part of main in all subsequent versions.
Are you sure that you are using openssl 1.1.1 when compiling?
Cmake tends to get confused when you have multiple openssl versions installed unless explicitly stated.

[DennisHeimbigner]

Possible, but I believe I uninstalled openssl libssl-dev and libssl1.0
and reinstalled and it failed.

[DennisHeimbigner]

So to review, I am trying to build branch 1.7.9, which I assume is the stable 1.7 branch. I am using ubuntu18 with its standard tool set. CMake is version 3.15.

I checked if the following was set by using some #error preprocessor commands.

• OPENSSL_VERSION_LESS_1_1 -- it wasn't set => version greater/equal 1.1

The build still failed. Below I describe in detail what I changed to get compile to work, but loading failed as shown below. Any suggestions?
=Dennis Heimbigner
Unidata

==================================================

So at compile time, I had following errors in the file
.../aws-sdk-cpp/aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp
(line numbers are approximate because some changes affected line no. of later changes).

• line 59: ERR_load_CRYPTO_strings was undeclared -- fixed by adding #include
• line 60: undeclared symbol: OPENSSL_add_all_algorithms_noconf() -- fixed by changing to OpenSSL_add_all_algorithms()
• line 627: EVP_CTRL_CCM_GET_TAG undefined -- fixed by changing to EVP_CTRL_AEAD_GET_TAG

At this point everything compiled.

But when trying to build the first executable: aws-cpp-sdk-s3-integration-tests

I got the following undefined references on loading. I think they all occur in CryptoImpl.cpp.
../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to `OpenSSL_add_all_algorithms' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to `EVP_CIPHER_CTX_cleanup' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to `EVP_MD_CTX_destroy' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to `EVP_MD_CTX_create' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to `EVP_CIPHER_CTX_init'
Apparently, my load command was this:
/usr/bin/c++ -rdynamic @CMakeFiles/aws-cpp-sdk-s3-integration-tests.dir/objects1.rsp -o aws-cpp-sdk-s3-integration-tests -Wl,-rpath,/home/dmh/git/aws-sdk-cpp/build/aws-cpp-sdk-s3:/home/dmh/git/aws-sdk-cpp/build/testing-resources:/home/dmh/git/aws-sdk-cpp/build/aws-cpp-sdk-core:/usr/local/lib ../aws-cpp-sdk-s3/libaws-cpp-sdk-s3.so ../testing-resources/libtesting-resources.so ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so /usr/local/lib/libaws-c-event-stream.so.1.0.0 /usr/local/lib/libaws-c-common.so.1.0.0 -lm -pthread -lrt /usr/local/lib/libaws-checksums.so -lpthread
This appears to be missing the openssl libs.

So, I modified .../aws-sdk-cpp/aws-cpp-sdk-s3-integration-tests/CMakeLists.txt as follows (line 32)
-target_link_libraries(${PROJECT_NAME} ${PROJECT_LIBS}) +target_link_libraries(${PROJECT_NAME} ${PROJECT_LIBS} ${CRYPTO_LIBS})
and got the same set of undefined references.

The modified load command was as follows:
/usr/bin/c++ -rdynamic @CMakeFiles/aws-cpp-sdk-s3-integration-tests.dir/objects1.rsp -o aws-cpp-sdk-s3-integration-tests -Wl,-rpath,/home/dmh/git/aws-sdk-cpp/build/aws-cpp-sdk-s3:/home/dmh/git/aws-sdk-cpp/build/testing-resources:/home/dmh/git/aws-sdk-cpp/build/aws-cpp-sdk-core:/usr/local/lib ../aws-cpp-sdk-s3/libaws-cpp-sdk-s3.so ../testing-resources/libtesting-resources.so ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so -lssl -lcrypto -lz /usr/local/lib/libaws-c-event-stream.so.1.0.0 /usr/local/lib/libaws-c-common.so.1.0.0 -lm -pthread -lrt /usr/local/lib/libaws-checksums.so -lpthread
Note that -lssl and -lcrypto are present (though perhaps not in the right order?).