Aws-sdk-cpp: Errors building a hello world example using C++ SDK on OSX

[dblock]

How do I build a basic hello world example?

What platform/OS are you using?

MacOS Mojave

Which version of the SDK?

1.7.278

What compiler are you using? what version?

cmake version 3.16.4
GNU Make 3.81

Building the SDK

Downloading and building SDK works.

git clone [email protected]:aws/aws-sdk-cpp.git
cd aws-sdk-cpp
git checkout '1.7.278'
mkdir build & cd build
cmake ../ -D CMAKE_BUILD_TYPE=Debug
make

Scanning dependencies of target aws-cpp-sdk-core
... a few hours later
Scanning dependencies of target polly_sample
[100%] Building CXX object aws-cpp-sdk-polly-sample/CMakeFiles/polly_sample.dir/main.cpp.o
[100%] Linking CXX executable polly_sample
[100%] Built target polly_sample

List S3 Buckets Sample

Trying to follow this doc or this blog post.

mkdir list-s3-buckets & cd list-s3-buckets

main.cpp

#include <iostream>
#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/Bucket.h>

int main(int argc, char** argv)
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    // more code here

    Aws::ShutdownAPI(options);
}

CMakeLists.txt

cmake_minimum_required(VERSION 2.8)
project(s3-sample)

find_package(aws-sdk-cpp)

add_executable(s3-sample main.cpp)

target_link_libraries(s3-sample aws-cpp-sdk-s3)

Reading https://aws.amazon.com/blogs/developer/using-cmake-exports-with-the-aws-sdk-for-c/ I run into missing dependencies, similar to https://github.com/aws/aws-sdk-cpp/issues/406.

osx ~/source/aws-sdk/list-s3-buckets/build$ cmake -Daws-sdk-cpp_DIR=/Users/dblock/source/aws-sdk/aws-sdk-cpp/build ..
-- The C compiler identification is AppleClang 11.0.0.11000020
-- The CXX compiler identification is AppleClang 11.0.0.11000020
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning at /usr/local/Cellar/cmake/3.16.4/share/cmake/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
  By not providing "Findaws-c-event-stream.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "aws-c-event-stream", but CMake did not find one.

  Could not find a package configuration file provided by
  "aws-c-event-stream" with any of the following names:

    aws-c-event-streamConfig.cmake
    aws-c-event-stream-config.cmake

  Add the installation prefix of "aws-c-event-stream" to CMAKE_PREFIX_PATH or
  set "aws-c-event-stream_DIR" to a directory containing one of the above
  files.  If "aws-c-event-stream" provides a separate development package or
  SDK, be sure it has been installed.
Call Stack (most recent call first):
  /Users/dblock/source/aws-sdk/aws-sdk-cpp/build/aws-cpp-sdk-core/aws-cpp-sdk-core-config.cmake:2 (find_dependency)
  /Users/dblock/source/aws-sdk/aws-sdk-cpp/build/aws-sdk-cpp-config.cmake:1 (include)
  CMakeLists.txt:4 (find_package)


CMake Warning at CMakeLists.txt:4 (find_package):
  Found package configuration file:

    /Users/dblock/source/aws-sdk/aws-sdk-cpp/build/aws-sdk-cpp-config.cmake

  but it set aws-sdk-cpp_FOUND to FALSE so package "aws-sdk-cpp" is
  considered to be NOT FOUND.  Reason given by package:

  aws-sdk-cpp could not be found because dependency aws-c-event-stream could
  not be found.



-- Configuring done
-- Generating done
-- Build files have been written to: /Users/dblock/source/aws-sdk/list-s3-buckets/build

and

osx ~/source/aws-sdk/list-s3-buckets/build$ make
Scanning dependencies of target s3-sample
[ 50%] Building CXX object CMakeFiles/s3-sample.dir/main.cpp.o
/Users/dblock/source/aws-sdk/list-s3-buckets/main.cpp:2:10: fatal error: 'aws/core/Aws.h' file not found
#include <aws/core/Aws.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/s3-sample.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/s3-sample.dir/all] Error 2
make: *** [all] Error 2

Seems like that blog post is out of date and these instructions don't work. Same problem on SO.

Tried various other things. How do I fix this?

[KaibaLopez]

Hi @dblock ,
So, continuing from your comment on the previous issue, you don't have to install the sdk globally but you have to install it somewhere for cmake to be able to use find_package().

So, the step by step would be:

1. follow the instructions to build the sdk, but when you do cmake add the parameter CMAKE_INSTALL_PREFIX, something like:

cmake {sdk-source-location} -DBUILD_ONLY="s3;polly" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX="{install location}"
make

1. when you run cmake on your app add the -DCMAKE_PREFIX_PATH parameter with the installed location, so:

cmake {app location} -DCMAKE_PREFIX_PATH="{install location}
make

Remember to update the CMakeLists.txt to use AWSDK instead of aws-sdk-cpp, since the blog post is outdated at this point, and if you're not planning on using the whole sdk for this app I recommend adding the "-DBUILD_ONLY" with the services you're going to need (in this case s3) to make building much faster.
Let me know if you have any more problems

[dblock]

I made some progress with this.

git clone [email protected]:aws/aws-sdk-cpp.git
cd aws-sdk-cpp
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=$HOME/source/aws-sdk/install
make install

Using find_package(AWSSDK REQUIRED COMPONENTS) it still fails on make.

~/source/aws-sdk/list-s3-buckets/build$ cmake -DCMAKE_PREFIX_PATH=$HOME/source/aws-sdk/install ..
-- The C compiler identification is AppleClang 11.0.0.11000020
-- The CXX compiler identification is AppleClang 11.0.0.11000020
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found AWS SDK for C++, Version: 1.7.278, Install Root:/Users/dblock/source/aws-sdk/install, Platform Prefix:, Platform Dependent Libraries: pthread;curl
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/dblock/source/aws-sdk/list-s3-buckets/build

then ...

~/source/aws-sdk/list-s3-buckets/build$ make
Scanning dependencies of target s3-sample
[ 50%] Building CXX object CMakeFiles/s3-sample.dir/main.cpp.o
/Users/dblock/source/aws-sdk/list-s3-buckets/main.cpp:2:10: fatal error: 'aws/core/Aws.h' file not found
#include <aws/core/Aws.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/s3-sample.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/s3-sample.dir/all] Error 2
make: *** [all] Error 2

[dblock]

Also what's the recommended default for the SDK? Just make install into default directories and use CMAKE_PREFIX_PATH=/usr/local?

Feels like everything should work with defaults without any additional path options.

[dblock]

Furthermore, the doc (source) is incorrect, right? There's no more aws-sdk-cpp_DIR setting?

[KaibaLopez]

yea it's outdated.
It does work with no additional paths if everything is in default. To be clear this is a cmake limitation than the actual sdk, it is cmake's find_package() that needs to be told where to look for the libraries so it can link them.
Can I see the CMakeLists.txt you're using?
Also, tryfind_package(AWSSDK REQUIRED COMPONENTS s3) or find_package(AWSSDK)

[dblock]

I built and make install-ed the C++ SDK with defaults. It went into /usr/local.

With find_package(AWSSDK) no errors in cmake .., but 'aws/core/Aws.h' file not found. With find_package(AWSSDK REQUIRED COMPONENTS s3) errors above.

https://github.com/dblock/aws-sdk-cpp-list-s3-buckets/blob/master/CMakeLists.txt

Again, it can't find the include. How does it know to look in /usr/local/include?

[dblock]

Related, https://github.com/aws/aws-sdk-cpp/issues/1307

[KaibaLopez]

Change the line:
target_link_libraries(s3-sample aws-cpp-sdk-s3)
to
target_link_libraries(s3-sample ${AWSSDK_LINK_LIBRARIES})
And let me know if that works.

[dblock]

That doesn't help because the problem is at compile time, not link time.

I figured some of it out, but am stuck at linking.

Issue 1: missing /usr/local/include in include search path

CMake cannot find /usr/local/include/aws/core/Aws.h, but it exists. Turns out, CMake on OSX does not include /usr/local by default, and we need to tell it to. This is discussed in https://stackoverflow.com/questions/23905661/on-mac-g-clang-fails-to-search-usr-local-include-and-usr-local-lib-by-def.

Workaround 1: Set CPLUS_INCLUDE_PATH

export CPLUS_INCLUDE_PATH=/usr/local/include

Workaround 2: Include in CMakeFiles.txt

include_directories(/usr/local/include)

Probably safe on all *nix platforms, I would recommend.

Issue 2: C++ Standard

The next set of problems look like this:

In file included from /Users/dblock/source/aws-sdk/list-s3-buckets/main.cpp:2:
In file included from /usr/local/include/aws/core/Aws.h:17:
In file included from /usr/local/include/aws/core/utils/logging/LogLevel.h:20:
In file included from /usr/local/include/aws/core/utils/memory/stl/AWSString.h:20:
In file included from /usr/local/include/aws/core/utils/memory/stl/AWSAllocator.h:21:
In file included from /usr/local/include/aws/core/utils/memory/AWSMemory.h:20:
/usr/local/include/aws/core/utils/memory/MemorySystemInterface.h:35:52: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
                virtual ~MemorySystemInterface() = default;

We just need to enable C++ 11 extensions in CMakeFiles.txt.

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

Issue 3: Missing libraries at link time

[100%] Linking CXX executable s3-sample
Undefined symbols for architecture x86_64:
  "Aws::ShutdownAPI(Aws::SDKOptions const&)", referenced from:
      _main in main.cpp.o
 ...

Looks like my AWSSDK_LINK_LIBRARIES is blank. Still trying to resolve.

If you have a mac, you'll have all these problems @KaibaLopez.

[KaibaLopez]

@dblock I do not, I have tested this in mac, windows and linux, with the default install locations as well with a custom locations, and I can run the sample code with no problems, well no problems after changing the CMakeLists.txt.
There is a another thing I forgot to mention though but it causes a different error on cmake, but it looks like you got around this somehow?
We changed how the dependencies are built recently, so normally you'd need to add:
-DBUILD_SHARED_LIBS=ON on your cmake command when building your app.

I did share my CMakeLists.txt on the previous issue, so I assume it also didn't work for you.

[dblock]

Well then, this fixed that. I added s3 in find_packages and BUILD_SHARED_LIBS options. Here's my final CMakeFiles.txt.

cmake_minimum_required(VERSION 3.2)
project(s3-sample)

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

option(BUILD_SHARED_LIBS "Build shared libraries" ON)

# on OSX this is not the default for CMake
# see https://stackoverflow.com/questions/23905661/on-mac-g-clang-fails-to-search-usr-local-include-and-usr-local-lib-by-def
include_directories(/usr/local/include)

find_package(AWSSDK REQUIRED COMPONENTS s3)
add_executable(s3-sample main.cpp)

target_link_libraries(s3-sample ${AWSSDK_LINK_LIBRARIES})

This was unnecessarily time consuming. I put everything in https://github.com/dblock/aws-sdk-cpp-list-s3-buckets and can keep it up-to-date as things change. Will cut a ticket to transfer it to aws-samples.

I'll go and try to make some updates to the linked issues/blog posts/docs unless someone beats me to it.

[dblock]

I've PRed fixes, so closing this. Thanks for your support @KaibaLopez.

[dblock]

https://code.dblock.org/2020/03/06/working-with-aws-cpp-sdk-and-data-exchange.html