Aws-sdk-cpp: AWS SDK static linking: libcrypto.so.1.0.0: cannot open shared object file

[ashu0992sharma]

I am trying to build AWS SDK for c++ with static linking, so I can use it as a binary inside AWS Lambda function.

Steps which I took are as follows:

1. git clone https://github.com/aws/aws-sdk-cpp.git.
2. mkdir build && cd build
3. cmake .. -DBUILD_SHARED_LIBS=OFF -DBUILD_ONLY="s3" -DENABLE_TESTING=OFF -DFORCE_SHARED_CRT=OFF (which created libaws-cpp-sdk-s3.a inside aws-cpp-sdk-s3 directory)
4. Now my source CMakeLists.txt looks like below

cmake_minimum_required(VERSION 3.1)

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_BUILD_TYPE "Release")
project(executable LANGUAGES CXX)

add_executable(${PROJECT_NAME} "execute_code.cpp")

set(OPENSSL_USE_STATIC_LIBS TRUE)
find_package(OpenSSL REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE OpenSSL::Crypto)

set(aws-sdk-cpp_DIR "<path to aws-sdk-cpp/build>")
find_package(aws-sdk-cpp)

link_libraries(aws-cpp-sdk-core)

target_link_libraries(executable PRIVATE aws-cpp-sdk-s3 aws-cpp-sdk-core)

target_compile_features(${PROJECT_NAME} PRIVATE "cxx_std_11")
target_compile_options(${PROJECT_NAME} PRIVATE
    "-Wall"
    "-Wextra"
    "-Wconversion"
    "-Wshadow"
    "-Wno-sign-conversion")
#set(CMAKE_EXE_LINKER_FLAGS "-static -static-libgcc -static-libstdc++")
include_directories(${PROJECT_SOURCE_DIR})

I am aware that AWS SDK for c++ requires OpenSSL and I have added that inside CMakeLists and when I run make after Cmake command, tt shows

-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.a (found version "1.0.2g") 

so project is able to find static libcrypto. But when I am deploying it on AWS lambda, it gives error error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory\n.

Can you tell me how do I debug this or If I am missing anything? I tried searching for it, but couldn't find anything useful for static linking.

Here is my execute_code.cpp:

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/Bucket.h>
#include<iostream>

int main(int argc, char** argv) {

  Aws::SDKOptions options;
  Aws::InitAPI(options);
  {
      // snippet-start:[s3.cpp.list_buckets.code]
      Aws::S3::S3Client s3_client;
      auto outcome = s3_client.ListBuckets();

      if (outcome.IsSuccess())
      {
          std::cout << "Your Amazon S3 buckets:" << std::endl;

          Aws::Vector<Aws::S3::Model::Bucket> bucket_list =
              outcome.GetResult().GetBuckets();

          for (auto const &bucket : bucket_list)
          {
              std::cout << "  * " << bucket.GetName() << std::endl;
          }
      }
      else
      {
          std::cout << "ListBuckets error: "
              << outcome.GetError().GetExceptionName() << " - "
              << outcome.GetError().GetMessage() << std::endl;
      }
      // snippet-end:[s3.cpp.list_buckets.code]
  }
  Aws::ShutdownAPI(options);

SDK version number
Found AWS SDK for C++, Version: 1.7.307

Platform/OS/Hardware/Device
Ubuntu 16.04

Please let me know if you require any other information related to this.

[webbnh]

I realize that it's not the answer to the problem that you are posting, but you _can_ build Lambda functions whose executables use shared libraries -- you just have to include them in the zip-bundle which you use for deployment.

(We build a Lambda function using the AWS Lambda runtime which uses _lots_ of shared libraries -- if the Lambda build doesn't find them on it's own, just create a directory or symlink lib which contains/points to the shared library that you want, and add it to the zip bundle (so that it shows up with the path lib/libcrypto.so) and it should find it when your Lambda executes.)

[ashu0992sharma]

@webbnh I am planning to use the same executable in different lambda functions (i.e python, java etc runtimes), that's why I am trying to use static libraries as much as possible. I am able to add other libraries with static linking but this gives me this error.
Also, I am not sure if it's a recommended approach to use static linking with some shared libs.
Still, If you can share some examples on how do I bundle the shared libraries. I would be happy to look into them.

[webbnh]

I don't understand what you're trying to achieve, but that's probably not important.

As for why something in your deployment is referencing the shared version of libcrypto, I presume it's because the reference is coming from another shared library...now, why something in your deployment is referencing _that_ is an interesting question, if you've asked everything to link against static libraries. Depending on where you are able to try to run your code (e.g., whether you can run it locally, or how much control you have over the Lambda environment), you should be able to get the loader to print out the executables, either a priori (if you have local access) or as they load (in the Lambda environment). This should give you a lead on where the reference to the shared libcrypto is coming from. (See man pages on ld(1) and dyld(1), et al.)

As for the recommended approach regarding mixing shared and static libraries, the answer is highly dependent upon the libraries. Some libraries (e.g., libpthreads) depend on having exactly one copy in the process. Thus, if you link each of two shared libraries against it statically, and then link your main executable against both shared libraries, then you'll end up with two copies of the libpthread library in your process, and badness will result. The same is true for any other function which maintains static context (e.g., strtok(3), or, to some extent, the standard I/O functions). However, in the latter case, it's only a problem if the calls to those functions are expected to be made alternately from separate shared executables -- if the calls are always made from a single executable, then they generally work as expected, and, in certain instances, they actually work _better_, since they don't suffer from interference from activities in the other executable(s). So, the answer is, yes, it is best to use all static or all shared libraries, to ensure that, whatever behavior you get, it's consistent; however, for many cases, it works fine or even best to have a given shared library include all of its dependencies (i.e., to link _it_ against static libraries, but have the dependencies on it link against shared libraries). And, the overall reasons for and against using shared libraries over static libraries apply: if anything in the application is using shared libraries, then you'll likely be better off if everything uses shared libraries, as the resulting disk and memory footprints will be smaller, because the shared libraries are _shared_ not only between the components in the process but potentially with other processes on the system, as well.

As for examples, we use CMake to generate the build for our project, and we use the aws-lambda-runtime and the AWS C++ SDK so that we can write our Lambda in C++. So, our CMakeLists.txt has lines like the following:

    find_package(aws-lambda-runtime)
    add_executable(mylambda mylambda.cpp myAwsIOStream.cpp)
    target_link_libraries(mylambda mylib AWS::aws-lambda-runtime ${AWSSDK_LINK_LIBRARIES})
    aws_lambda_package_target(mylambda)

Then we build our Lambda bundle like this:

ln -sf myObjsDir lib && \
  make aws-lambda-package-mylambda && \
  zip mylambda.zip lib/lib*"

The aws-lambda-package-mylambda target is generated by the aws_lambda_package_target CMake directive, and it builds mylambda and creates a zip file of its executable and all the shared libraries that it (statically) depends on. However, in our case, we have some libraries which we dynamically load at runtime, so the extra zip command adds them to the bundle. The final detail is that, in the AWS Lambda execution environment, one of the places that the loader looks for shared libraries is in ./lib, and so adding the additional libraries via the symbolic link ensures that they will be found properly when the bundle is unpacked in the Lambda environment.

[KaibaLopez]

@webbnh that's a great explanation thanks, super appreciate the support!

@ashu0992sharma , just want to add a couple things:
First, combined library types are very problematic with our sdk specifically, not that it's impossible to do but just weird and tricky because of cmake limitations regarding that, so I would personally recommend to not go there, just my opinion though.
Second, as webbnh said:

why something in your deployment is referencing that is an interesting question

I would just add, a common place to miss this is curl or whatever your crypto or openssl configuration is. The cmake flags don't extend to these libraries configurations, so if you have for example cURL configured to look for openssl.1.0.2g.so it will still look for that regardless if you built the sdk to take static.
It would definitely help if we knew if your code runs locally or fails that too.

[ashu0992sharma]

@KaibaLopez code runs perfectly in local, in lambda environment it gives the error. How should I proceed further if I want the static linking or if you can help me how can I debug further?
@webbnh I will definitely try out bundling the lib directory. One question though, is myObjsdir your source directory and for lib directory, are you referring to /usr/lib or it is a directory within your source bundle where you have compiled everything from source?

[webbnh]

@ashu0992sharma, given that your code can be run locally, you should be able to get the loader to tell you what the shared library dependencies are (one way or another), and then you can see which shared libraries are being referenced by the linker, and, hopefully, armed with that, you can figure out how to modify your build to change those dependencies to use static libraries.

myObjsDir is the directory in which my build places all of the shared libraries that it produces. In my case, the AWS Lambda build target automatically finds all the system libraries for me. If you use it, I expect it will find (and bundle) libcrypto for you, unless it is some sort of exception (but, I don't remember there being any exceptions). Nevertheless, you are free (more or less) to add any libraries that you like to the zip bundle, but you have to do it in such a way that, when the bundle is unpacked, they land in one of the places where the LD_LIBRARY_PATH environment variable (in the Lambda execution environment) points.

So, you can modify your Lambda (or write a new one) to tell you what your choices are, and you can be clever about how you build the zip bundle (e.g., using the symlink trick), and the result should be that all the libraries you need end up in places where they can be found.

[ashu0992sharma]

@webbnh thank you for the detailed explanation, I am able to fix this issue.
I was working on the Ubuntu machine, now when I built on Amazon Linux, it fixed my issue.

Although I might need the bundling of libraries and deploy on lambda in the near future. I am a kind of a newbie in this. Can you please point me to the documentation or resources which can give me a starting point in this?

[webbnh]

I'm glad you were able to get resolution. We, too, work on Ubuntu (on 16.04!...), and there are some adjustments required to make the code also work on Amazon's Linux (mostly, IIRC, around the SSL CA certificates' location). If you can build and test in the environment that you're going to deploy in (e.g., by using Docker), then you'll probably find it easier.

As for documentation, now you're asking a hard question!

The aws-lambda-cpp Github repo README.md file is the principal resource for building, bundling, and deploying C++-based Lambda functions. I presume that you've already found it, but check the section and subsection in it on "packaging".

Here is the list of environment variables which are set up for you in the AWS Lambda deployment (but, I found it more useful to just make my code ask and print them out).

And, here's my favorite resource for the C++ runtime library (if you poke it hard enough, it will show you the header files, which is where I found the (lack of) virtual keywords).

As I was looking for other things for you, I also found this fascinating bit, which might be of more use to you than the TransferManager!

[ashu0992sharma]

@webbnh I can't thank you enough :). Really thankful for helping me out.