Aws-sdk-cpp: Can only build custom client if serviceName is "custom-service"

[mattcwill]

What platform/OS are you using?

Microsoft Windows 10 Enterprise 1709 Build 16299.1146

Which version of the SDK?

1.7.117

What compiler are you using? what version?

Microsoft Visual Studio 2017 (cl.exe version 19.16.27027.1)

What are your CMake arguments?

-DBUILD_ONLY="s3" -DADD_CUSTOM_CLIENTS="serviceName=petstore, version=2017-11-03"

I'm trying to generate a custom client for my API Gateway service, but the Cmake process fails unless my service is named "custom-service". You can reproduce the problem with the sample PetStore custom service as follows:

1. Copy custom-service-2017-11-03.normal.json from code-generation/api-descriptions/custom-service to code-generation/api-descriptions
2. Rename custom-service-2017-11-03.normal.json to petstore-2017-11-03.normal.json
3. Run CMake using -DADD_CUSTOM_CLIENTS="serviceName=petstore, version=2017-11-03"

I can see code generated in <source_dir>/aws-cpp-sdk-petstore, and a folder called aws-cpp-sdk-petstore in my build directory, but CMake fails with the error:

get_property could not find TARGET aws-cpp-sdk-petstore. Perhaps it has not yet been created.

If I look in aws-cpp-sdk-petstore in my build directory, I see a file called aws-cpp-sdk-custom-service-config-version.cmake. It seems like the file is always named with "custom-service" no matter what I name my custom service .json file. Everything seems to work if I leave the service name as "custom-service". I'm wondering if in 7d60ff63bc5532e897b372f45b748132a8025880 the project name was maybe inadvertently hard-coded to "aws-cpp-sdk-custom-service"? Or is there a step I'm missing?

[wps132230]

Hi, there.
It was hard coded by mistake. We will fix that.
If the protocol is "api-gateway", it will always have "aws-cpp-sdk-custom-service" as project name.
https://github.com/aws/aws-sdk-cpp/blob/9007ea3f1eeae02179336146ed8d58899fc3efea/code-generation/api-descriptions/custom-service/custom-service-2017-11-03.normal.json#L7
https://github.com/aws/aws-sdk-cpp/blob/9007ea3f1eeae02179336146ed8d58899fc3efea/code-generation/generator/src/main/resources/com/amazonaws/util/awsclientgenerator/velocity/cpp/CMakeFile.vm#L5
And for your use case, would you like to build your service with SDK at the same time? Or build your service after you have installed SDK?

[mattcwill]

Thanks for the reply! It'd be nice to be able to build my service at the same time as the SDK, and then be able to re-build just the service if I need to make changes to it. Currently it seems like I need to build and install the SDK without my custom service (i.e. without ADD_CUSTOM_CLIENTS) first, as the custom service build process expects the aws-sdk-core libraries to already be installed.

This is probably a separate issue, but on the topic of generating custom clients, I also needed to use the workaround described in this comment for the AWSAuthV4Signer to work. Namely, setting the endpoint prefix to "execute-api" and then setting the endpointOverride to "abcdefg123.execute-api.us-east-1.amazonaws.com" (or similar) in my custom client's ClientConfiguration.

[wps132230]

Yes, currently you have to build and install SDK first.
And for your separate issue, we will append "execute-api" to the original endpoint, if the protocol is "api-gateway":
https://github.com/aws/aws-sdk-cpp/blob/11e9a78009eea36d219be20988f65b5ad74bdee0/code-generation/generator/src/main/java/com/amazonaws/util/awsclientgenerator/transform/C2jModelToGeneratorModelTransformer.java#L112

[mattcwill]

I think the endpoint prefix part of the code generation is fine as you stated. My issue was that in the generated code (i.e. PetStoreClient.cpp), SERVICE_NAME was being set to my API's endpoint prefix (i.e. "z2z37qum61") when the signer wants it to be "execute-api". In JsonServiceClientSource.vm line 43 you can see that this is set based on the signingName from metadata:

static const char* SERVICE_NAME = "${metadata.signingName}";

And here in the Java code you can see that the signing name is set to the endpointPrefix if it's not explicitly set in the JSON definition:

https://github.com/aws/aws-sdk-cpp/blob/11e9a78009eea36d219be20988f65b5ad74bdee0/code-generation/generator/src/main/java/com/amazonaws/util/awsclientgenerator/transform/C2jModelToGeneratorModelTransformer.java#L107

So ultimately my solution was to set "signingName" : "execute-api" in my service's .json file. Perhaps this should also be set in the PetStore example file, and/or have the Java code set signingName to "execute-api" if not explicitly set and the protocol is "api-gateway"?

[mattman71]

I'm not sure if I should pile on a "me too" for this thread.

The version mentioned by OP, 1.1.117, is extremely old -- unless that's a typo. In any case, I am unable to build my custom client (protocol api-gateway) using the latest release 1.7.146.

I'm building static libraries on "Visual Studio 15 2017 Win64" platform, with CPP_STANDARD=17.

The last version I successfully compiled my custom client was 1.5.17, and it was working fine then. I have not changed my api gateway at all since then. I did change my build process slightly, based on this thread. Above it was mentioned to build/install the sdk first, then come back to build the custom client. I did make that change, although previously, with 1.5.17, I was able to do both in one step. Even with this change to build the sdk first, then come back for another pass to build only my custom client, with version 1.7.146 it is failing.

Here is the end of the process where it is failing (some of these names have been changed, for obvious reasons)...

-- serviceName=myCustomAPI, version=2018-07-13
-- Clearing existing directory for myCustomAPI to prepare for generation.
-- generating client for myCustomAPI version 2018-07-13
Generating myCustomAPI api version 2018-07-13.
-- Found AWS SDK for C++, Version: 1.7.146, Install Root:C:/Users/myUserName/AppData/Local/Temp/awsdepinstall/Debug, Platform Prefix:, Platform Dependent Libraries: Userenv;version;ws2_32;Bcrypt;Wininet;winhttp
-- Components specified for AWSSDK: core
-- Try finding aws-cpp-sdk-core
-- Found aws-cpp-sdk-core
-- Building custom service as static objects
-- Updating version info to 1.7.146
-- Custom memory management disabled
CMake Error at cmake/dependencies.cmake:15 (get_property):
  get_property could not find TARGET aws-cpp-sdk-myCustomAPI.  Perhaps it has
  not yet been created.
Call Stack (most recent call first):
  cmake/dependencies.cmake:59 (compute_links)
  cmake/sdks.cmake:226 (sort_links)
  CMakeLists.txt:288 (add_sdks)

CMake Error at cmake/dependencies.cmake:70 (get_target_property):
  get_target_property() called with non-existent target
  "aws-cpp-sdk-myCustomAPI".
Call Stack (most recent call first):
  cmake/sdks.cmake:226 (sort_links)
  CMakeLists.txt:288 (add_sdks)

One additional bit of info, I do see that source code has been generated under aws-cpp-sdk-myCustomAPI, just like it was in previous versions. I'm not savvy enough to know if it's correct, but it appears to be complete. Somehow the build appears to be broken and failing to find the target.

So, is this the same issue as OP, or should I start a new issue? Is there a work-around? Changing my service name to "custom-service", as deployed in the cloud, is not an option.

[mattcwill]

@mattman71 Oops you are right - the version was a typo, should have been 1.7.117. I edited the original post with the correct version I was using.

And yes that is the exact error I get if I call my service anything other than "custom-service". If you go to the aws-cpp-sdk-myCustomAPI directory in your build folder, you will see a file called "aws-cpp-sdk-custom-service-config-version.cmake". This should be generated as "aws-cpp-sdk-myCustomAPI-config-version.cmake", but it's not. Seems like the bug may have been introduced in 1.7.98.

[mattman71]

@wps132230 , after playing with this, I concur with your identification above about CMakefile.vm line 5. If I do this patch:

< project(aws-cpp-sdk-custom-service)
---
> project(aws-cpp-sdk-${metadata.projectName})

then the build continues successfully with all the proper names.

I can't comment about the execute-api endpoint situation mentioned, as I'm not sure what I'm looking for there yet to determine if I'm having that issue.

[wps132230]

Hi, @mattman71 , you are right, we are about to fix that. And the code is under review.

[wps132230]

Hi, we've fixed this issue: https://github.com/aws/aws-sdk-cpp/commit/5cae5d29c91994c6dabcffe9c15098738e2c6c96
And by using -DADD_CUSTOM_CLIENTS, you can build your custom client along with C++ SDK now.
Otherwise, you can follow this step to build custom client as a separate package: https://github.com/aws/aws-sdk-cpp/blob/master/scripts/build_custom_client.sh

[mattman71]

I've confirmed that the build completes successfully now...

I was able to build with the SDK (by using -DADD_CUSTOM_CLIENTS along with sdk build, using -DBUILD_ONLY="stuff;I;want" -- and not having to specify my custom client in the -DBUILD_ONLY).

I also was able to build the client by itself, by pointing the install dir to the previously build SDK libraries, and using -DADD_CUSTOM_CLIENTS on the subsequent build run and giving -DBUILD_ONLY="myCustomClient" (though I'm not sure that was necessary, it did work).

I haven't tested the execution, that's a different story, but the build is working.

Thank you!

[mattcwill]

I can also confirm that the service name issue is resolved, thank you!

As for the endpoint issue, my difficulty was that I was using Lambda Proxy integration in API Gateway, and therefore had no Method Response models defined. This meant when I tried to generate a c2j file using the aws apigateway get-export ... --export-type c2j command, I got an error saying "Must have at least one 2xx response." Same error if I tried to generate a Java SDK. So I ended up switching to Lambda Integration and defining Method Response models for my endpoints, and now I can generate a c2j file for my API. In the generated version, the signingName is set correctly to "execute-api", and so I don't need to set the endpointOverride in my client as a workaround.

So the lesson learned was: if you want to automatically generate a c2j file for your API, it seems like you need to define Method Response models in API Gateway. If you don't want to define response models, you can still start from the petstore custom-service example file, but you may need to set some additional metadata fields depending on things like signature version (in my case, I needed to set signingName to "execute-api"). The generated version is also far more detailed than the one I was trying to build by hand, so I'll probably stick with defining response models for now. If you're using Serverless Framework, the serverless-aws-documentation plugin was helpful with this.