Aws-sdk-cpp: Unable to get custom client working, missing signer?

[mattman71]

Problem summary

I am able to generate the library for my custom client using the sdk (1.4.39 is my latest attempt). To do that, I used the aws apigateway get-export method to pull my c2j and then placing that in the required filename format into code-generation/api-descriptions. That is all working now (I have 4 static libs: aws-cpp-sdk-{core,mySanitizedApi,s3,transfer}.lib). Everything compiles and links without error. But when I try to use it (just in a test for now), I'm getting a debug assertion and a failure deep in the auth client mechanism -- it is attempting to look up the authorizer name for my custom api in a list of "signers", and isn't finding it.

See sample code and trace level log below.

This API works with direct http URL crafting in a browser (passing the required request variable as a query parameter), and it also works without any special setup using C#. The api does not require credentials -- it is intended to be anonymous. Its purpose is to retrieve some temp credentials on the server side based on the SanitizedRequestVariable.

I have tried initializing my client using the AnonymousAWSCredentialsProvider (as well as not specifying a provider at all, letting it default) -- nothing is working (at least not in the c++ sdk generation of my custom client.

What platform/OS are you using?

Windows

What compiler are you using? what version?

Visual Studio 2015

What's your CMake arguments?

-DBUILD_ONLY="s3;transfer" -DADD_CUSTOM_CLIENTS="serviceName=mySanitizedApi, version=2018-04-23" -DBUILD_SHARED_LIBS=OFF -DFORCE_SHARED_CRT=OFF -DENABLE_UNITY_BUILD=ON -DTARGET_ARCH="WINDOWS" -G "Visual Studio 14 2015 Win64"

Can you provide a TRACE level log? (sanitize any sensitive information)

[INFO] 2018-05-01 10:16:48 Aws_Init_Cleanup [14356] Initiate AWS SDK for C++ with Version:1.4.35
[INFO] 2018-05-01 10:16:48 WinHttpSyncHttpClient [14356] Creating http client with user agent aws-sdk-cpp/1.4.35 Windows/10.0.15063.608 AMD64 with max connections 25 request timeout 10000,and connect timeout 3000
[INFO] 2018-05-01 10:16:48 WinHttpSyncHttpClient [14356] Http Client is using a proxy. Setting up proxy with settings scheme http, host mysanitizedproxy.net, port 8080, username 
[DEBUG] 2018-05-01 10:16:48 WinHttpSyncHttpClient [14356] Adding proxy host string to winhttp HTTPS=http://mysanitizedproxy.net:8080
[DEBUG] 2018-05-01 10:16:48 WinHttpSyncHttpClient [14356] API handle 000001A9DC465AF0
[INFO] 2018-05-01 10:16:48 ConnectionPoolMgr [14356] Creating connection pool mgr with handle 000001A9DC465AF0, and max connections per host 25, request timeout 10000 ms, and connect timeout in 3000 ms.
[TRACE] 2018-05-01 10:16:48 AWSClient [14356] No content body, content-length headers
[ERROR] 2018-05-01 10:16:48 AuthSignerProvider [14356] Request's signer: 'mySanitizedApiAuthorizer' is not found in the signer's map.
[DEBUG] 2018-05-01 10:16:54 WinSyncHttpClient [14356] Cleaning up client with handle 0000000000000000
[INFO] 2018-05-01 10:16:54 WinHttpConnectionPoolMgr [14356] Cleaning up conneciton pool mgr.
[INFO] 2018-05-01 10:22:42 Aws_Init_Cleanup [504] Initiate AWS SDK for C++ with Version:1.4.35
[INFO] 2018-05-01 10:22:42 WinHttpSyncHttpClient [504] Creating http client with user agent aws-sdk-cpp/1.4.35 Windows/10.0.15063.608 AMD64 with max connections 25 request timeout 10000,and connect timeout 3000
[INFO] 2018-05-01 10:22:42 WinHttpSyncHttpClient [504] Http Client is using a proxy. Setting up proxy with settings scheme http, host mysanitizedproxy.net, port 8080, username 
[DEBUG] 2018-05-01 10:22:42 WinHttpSyncHttpClient [504] Adding proxy host string to winhttp HTTPS=http://mysanitizedproxy.net:8080
[DEBUG] 2018-05-01 10:22:42 WinHttpSyncHttpClient [504] API handle 000001C3C40BDC90
[INFO] 2018-05-01 10:22:42 ConnectionPoolMgr [504] Creating connection pool mgr with handle 000001C3C40BDC90, and max connections per host 25, request timeout 10000 ms, and connect timeout in 3000 ms.
[TRACE] 2018-05-01 10:25:20 AWSClient [504] No content body, content-length headers
[ERROR] 2018-05-01 10:29:12 AuthSignerProvider [504] Request's signer: 'mySanitizedApiAuthorizer' is not found in the signer's map.
[DEBUG] 2018-05-01 10:29:23 WinSyncHttpClient [504] Cleaning up client with handle 0000000000000000
[INFO] 2018-05-01 10:29:23 WinHttpConnectionPoolMgr [504] Cleaning up conneciton pool mgr.

Sample code

#define TAG "mySanitizedApiTest"

TEST_METHOD(mySanitizedApiPass)
{
    auto config = Aws::MakeShared<Aws::Client::ClientConfiguration>(TAG);
    config->scheme = Aws::Http::Scheme::HTTPS;
    config->requestTimeoutMs = 10000; // 10 seconds
    config->connectTimeoutMs = 3000; // 3 seconds
    config->retryStrategy = Aws::MakeShared<Aws::Client::DefaultRetryStrategy>(TAG, 5, 10);
    config->proxyScheme = Aws::Http::Scheme::HTTP;
    config->proxyHost = "mysanitizedproxy.net";
    config->proxyPort = 8080;

    //auto credentialsProvider = Aws::MakeShared<Aws::Auth::AnonymousAWSCredentialsProvider>(TAG);
    //auto client = Aws::MakeShared<Aws::mySanitizedApi::mySanitizedApiClient>(TAG, credentialsProvider, *config);
    auto client = Aws::MakeShared<Aws::mySanitizedApi::mySanitizedApiClient>(TAG, *config);
    auto request = Aws::MakeShared<Aws::mySanitizedApi::Model::GetCredentialsRequest>(TAG);
    request->SetSanitizedRequestVariable(_sanitizedValue);
    auto outcome = client->GetCredentials(*request);
    Assert::IsTrue(outcome.IsSuccess());
}

[singku]

If you are generating customized client code using our code generator, the default signer should be the default AuthV4Signer. What have you done to change the signer name to mySanitizedApiAuthorizer?

If you did change the signer, you should implement a customized signer provider to feed your customized client constructor, such that the core can find the correct signer implementation based on the signer name.

[mattman71]

That is pretty much the mystery. I didn't do anything to change the signer name to mySanitizedApiAuthorizer. The aws apigateway get-export is doing that when it generates the c2j. That authorizer is the name of the custom authorizer that my apigateway uses to call a lambda function (which, of course, needs a custom authorizer to do the temp credentials). I didn't want to muddy the waters here by mentioning the C# or direct http (with browser) too much, but it seems obvious from those other working mechanisms, that the mySanitizedApiAuthorizer is not needed for the client side -- those other mechanisms don't know anything about it. Something is happening with the apigateway get-export that is leading me on a wild goose chase, and I'm not sure why it's doing that.

I feel like there's a missing link, but I'm not finding a walk-through, so it's all trial and error -- I don't really know what I'm doing, and I can't figure out what I'm missing or doing wrong.

The api gateway was supposed to be basically an anonymous pass-through -- the client passes some critical information to be validated, then the api gateway turns around and uses a lambda with some other mojo to get proper temp credentials for the resource my client will be accessing next (s3, obviously, from my list of libraries, haha).

[mattman71]

Update: through reverse engineering what I think the sdk code generator is doing (I have no actually traced into the source code to confirm this, I'm only working with the outputs), I have managed to hack together something that sort of works but requires several hands-on manual hacks.

There are 2 problems that I see with the custom api code generator mechanism:

1. the aws apigateway get-export --export-type c2j ... output includes the api-2.json file that I need to drop into code-generation/api-descriptions. That's fine, however, in the case of my custom api gateway, I need (and have) a custom authorizer. For some unknown reason, the code-generation mechanism is taking the "authtype" and "authorizer" fields from my custom api gateway operation method and is trying to use that as a signer. I have no idea why, but this is not the right thing to do for mine (is it ever the right thing to do for anybody?). The C# sdk does not do this. This is later causing a null reference exception at runtime when it tries to resolve the signer. Even if I correctly instantiate my client with AnonymousAWSCredentialsProvider. I would consider this a bug. To get around this problem, I am manually editing the api-2.json c2j output file to change "authtype" to "none" and remove the "authorizer" because the client doesn't need to know about that at all.
2. the uri generation that happens as part of the code-generation is wrong. It correctly includes my endpoint + region + amazonaws.com, but it is not including the ".execute-api." part that is supposed to come between endpoint and region. If I manually edit this api-2.json output from c2j, and hack my "endpointPrefix" to be my endpoint + ".execute-api" appended to the string, then the uri comes out correct. This appears to be a bug.

With those two problems highlighted, now is there anybody that could help with a fix? I'm still willing to believe I'm doing something wrong, but I've looked everywhere and I can't find anything on my side that I could do to make these behaviors with the sdk code generation change; therefore, they appear to me to be bugs.

Thanks!

[wps132230]

Hi, for the first question, if you specify authtype and authorizer for your operation, sdk will assume that you will use custom signer for that operation. One of the solutions is to implement the custom signer like AWSNullSigner:
https://github.com/aws/aws-sdk-cpp/blob/3d6ea5ed0ce06e0cebe675e00ca41e8c5f8d2306/aws-cpp-sdk-core/include/aws/core/auth/AWSAuthSigner.h#L252

For the second question, it's a bug on our side, we will fix that as soon as possible, and thank you for reporting it to us.

[mattman71]

Hi, thanks for the reply!

Regarding problem two, I will anxiously await a fix for that part.

Back on the first question... the authtype and authorizer are being generated by the aws cli.

aws apigateway get-export --rest-api-id mySanitizedRestApiId --stage-name mySanitizedStageName --region us-west-2 --export-type c2j ./c2j.zip

Within that c2j.zip is an api-2.json which I then rename to mySanitizedApi.yyyy-mm-dd.normal.json and place into code-generation/api-descriptions.

I am practicing CD (continuous deployment), so build automation is important. If I have to manually edit this file, it breaks automation. I'd rather not have to come up with custom tools or scripts to poke dynamically generated content to get it to work. I simply do not understand why the c2j export process is putting my custom authorizer in there (it is used on the server-side only -- the client doesn't need to know anything about it). And I'm not sure if that is the problem, or that the code generator is using that to imply anything about a signer, which seems completely unrelated.

My custom authorizer has some required query string parameters, which the code generator is correctly handling in the Request class. No problem there. The problem all centers around whatever is going on with this signer assumptions. Using the default v4 signer should work fine, as far as I know, it's just not using it and I don't know why.

As for your reference to the AWSNullSigner -- that doesn't do me any good. The generated class gives me no opportunity to inject a signer. The signer stuff is all handled in the embedded generator code and is not exposed to the api constructor or anything.

       /**
        * Initializes client to use DefaultCredentialProviderChain, with default http client factory, and optional client config. If client config
        * is not specified, it will be initialized to default values.
        */
        MySanitizedApiClient(const Aws::Client::ClientConfiguration& clientConfiguration = Aws::Client::ClientConfiguration());

       /**
        * Initializes client to use SimpleAWSCredentialsProvider, with default http client factory, and optional client config. If client config
        * is not specified, it will be initialized to default values.
        */
        MySanitizedApiClient(const Aws::Auth::AWSCredentials& credentials, const Aws::Client::ClientConfiguration& clientConfiguration = Aws::Client::ClientConfiguration());

       /**
        * Initializes client to use specified credentials provider with specified client config. If http client factory is not supplied,
        * the default http client factory will be used
        */
        MySanitizedApiClient(const std::shared_ptr<Aws::Auth::AWSCredentialsProvider>& credentialsProvider,
            const Aws::Client::ClientConfiguration& clientConfiguration = Aws::Client::ClientConfiguration());

Using the last constructor, as I mentioned, I can pass the AnonymousAWSCredentialsProvider, but that is not the problem -- the problem is the signer stuff that is in the embedded generated code (which I certainly hope I don't have to manually edit). When doing this with the C# SDK stuff, it "just works" -- it doesn't know anything about the custom authorizer on the client side, and works without any problem.

Something is wrong here, but I'm not savvy enough to try to figure out which piece is causing the problem. I am trying to help all I can to fix this (for me, and for anybody else who stumbles into this and gives up), but we need more specifics on interaction to pinpoint it -- and make very few assumptions that I know what I'm doing. 😆

[wps132230]

1) We will add ".execute-api." to the endpoint.
2) We will add support for customers to add any singer (including NULLSIGNER) to their custom client construction.

Does that sounds good to you?

[mattman71]

Sounds like that should work. Thanks!

[wps132230]

Updated: Code are under review.

[mattman71]

How are things coming along with this? I wish I had some extra bandwidth to help. :/

[wps132230]

So sorry about the delay, we added more tests(including some sample code about how to add null signer to the client).
We will merge these code in one week.

[mattman71]

Thanks for the update! No complaints from me for having to wait for good test coverage and updated samples -- just wanted to make sure you didn't forget about me. :)

[jkoutavas]

Hi mattman71.

You wrote: "I used the aws apigateway get-export method to pull my c2j [...]" -- how did you do this exactly?

I've tried generating the java custom SDK for my API Gateway from the AWS console, both for maven and for gradle, but there's no .c2j file to be found.

A search of the web has found very little in the way of example .c2j files (I'd be pleased just going through the process of generating an example custom C++ API SDK.)

Help, please? Thanks.

[mattman71]

If you look above at my comment on May 17, the aws apigateway command is what does that part. Just fill in the appropriate parameters for your rest api id, the name of the stage where you've published your api gateway, the region you're hosting in, etc.

After all of that, you may still get stuck because of the above 2 bugs, but I think they're very close to delivering fixes for those.

You may not be affected by the signer problem if you're not using a custom authorizer in your api gateway. But you'll likely be affected by the endpoint bug -- the generator leaves out the .execute-api. part of the URL for the endpoint. If you're not concerned with build automation, you can fix that with a quick manual edit in the json file you get from your c2j.zip and place into api-descriptions. There will be an endpoint key/value in there, and you can just append .execute-api to it -- you'll figure out what I mean when you see it; it's basically constructing the url with string concatenation, and you just have to add the missing piece to the unique encoded part of the beginning of your url, so that when it appends the rest of the suffix, it'll all work together. Clear as mud, right? 🤣

[jkoutavas]

Okay, sorry, I missed that May 17th comment somehow. Thank you so much for kindly pointing it out.

[mattman71]

No worries, it was just a tiny one line that scrolled, and I tend to throw up walls of text; easy to get lost in the noise. 😆

[wps132230]

Hi @mattman71 sorry for letting you wait for such a long time.
We just pushed this commit: https://github.com/aws/aws-sdk-cpp/commit/ebcc3ded77ea8a9b39e18676b2d54a91712b009f, in which

1. Fixed the URI bug, appending .execute-api to the endpoint.
2. Added support to define custom signer including null signer in the client constructor.
3. Added a test to show an example of how to define custom singer and make use of it
4. Added a build script to show how to generate, build and make use of a custom client: https://github.com/aws/aws-sdk-cpp/blob/master/scripts/build_custom_client.sh. We will run it with every future release to make sure it's always compatible with aws-cpp-sdk-core. You can find the api description file here, which is a very basic example provided by API Gateway.

And more details about how to "generate custom client", after this commit, we will support two approaches:

1. With cmake flag -DADD_CUSTOM_CLIENT, I think that's how you generate client right now. In this approach, we treat custom client as any other AWS service, and build aws-cpp-sdk-core together with custom client.
2. With python script specified in the build script. In this approach, we take custom client as a separate package, which means we have to build and install aws-cpp-sdk-core as a dependency. And that's expected to be the approach we use in the future if we integrate custom client code generation in API Gateway console, like Java, Ruby. You can download the source code(including the cmake file) from API Gateway console and build custom client on the machine with aws-cpp-sdk-core library built and installed.

Please let us know if that meets your requirements or you have more questions.

[mattman71]

Acknowledged. I'm digging in to see if all the pieces fit together now, and will respond back with results...

[mattman71]

Preliminary results are looking good. It's awkward that I am required to make a custom signer provider class that provides the null signer, rather than just override the signer name with nullsigner. But it does work once I set all that up, so I'm no longer blocked.

I've tested the method 1 you mentioned, with the -DADD_CUSTOM_CLIENT. You were correct in assuming that's the one I use. I can confirm that method is working with the two promised fixes.

Thanks!

[jkoutavas]

In our case we have "signatureVersion" : "v4" in our {custom-service-name}.normal.json file. When doing the cmake to generate the custom service, we're seeing the generated code for our custom client doing MakeRequest(uri, request, HttpMethod::HTTP_GET, Aws::Auth::NULL_SIGNER); instead of the expected MakeRequest(uri, request, HttpMethod::HTTP_GET, Aws::Auth::SIGV4_SIGNER);

We do our cmake on macOS like this:

cmake -DBUILD_ONLY="core;cognito-identity;cognito-idp" -DADD_CUSTOM_CLIENTS="serviceName={custom-service-name}, version=2018-07-17" -DMINIMIZE_SIZE=ON -DBUILD_SHARED_LIBS=OFF -DSIMPLE_INSTALL=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET="10.11" -DCPP_STANDARD=17 .

Are we expected to pass a v4 signer into the constructor of the client, or is this a bug in code-gen?

[mattman71]

Check the authtype in your json. That was the problem I ran into -- I actually wanted the nullsigner, but authtype was set to my custom authorizer (which isn't a signer) and it had a corresponding authorizer section in the json. You can look at the authtype from some of the other json files in the api-descriptions folder to get an idea of which value you probably want to use for the default v4 signer.

[mattman71]

@wps132230 I've been able to get my custom client working using the updates (I'm running 1.4.89). But I have a new problem where my custom client is returning custom credentials (access key id, secret, and session token), but when I plug those into my s3client -- I'm getting 403 errors (access denied).

I'm guessing I should submit another issue for that? (NOTE: using those same temp credentials, by poking them in my .aws/credentials file, and using the aws s3 CLI works fine, so the custom client does not seem to be the problem since my api is now working as intended)

[jkoutavas]

@mattman71 wrote: Thank you. I added the following to my swagger.yaml and all came out great.

securityDefinitions:
  sigv4:
    type: "apiKey"
    name: "Authorization"
    in: "header"
    x-amazon-apigateway-authtype: "awsSigv4"

[mattman71]

@wps132230, nevermind about my previous comment -- I think I'm missing something in the policy definition for my temp credential.

This bug can be closed. It would be easier to pass one of the predefined signername strings to the custom client constructor rather than having to make a custom signerprovider just to wrap one of the predefined ones, but perhaps there are others who would prefer to make their own signerprovider that's not simply a wrapper for one of the defaults.