Sendgrid-csharp: Unable to verify signature from event webhook

[Jericho]

Issue Summary

I am trying to validate the feature that was recently added to the sendgrid-csharp client which validates signed event webhook. Unfortunately, it doesn't seem to work when SendGrid posts a sample JSON payload to my webhook listener.

Here is the payload POSTed to me by SendGrid:

POST /StrongGrid HTTP/1.1
Host: localhost:56736
User-Agent: Go-http-client/1.1
Content-Length: 3698
Accept-Encoding: gzip
Content-Type: application/json
X-Forwarded-For: 167.89.117.41
X-Original-Host: 09676049fd42.ngrok.io
X-Twilio-Email-Event-Webhook-Signature: MEUCIQCeMcxBRAVPe2524K99rZ6TNnOAy4felQISmx87NMNSQgIgaLrcM7gYi4xBF0UtSDh9rtgDB9ncw8kp7hQOKddR0ik=
X-Twilio-Email-Event-Webhook-Timestamp: 1592925630

[{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"processed","category":"cat facts","sg_event_id":"A__ghUTvJW9BMlaFhDxcdQ==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"deferred","category":"cat facts","sg_event_id":"YMKAt-phqCnNMkjScL0Qog==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","response":"400 try again later","attempt":"5"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"delivered","category":"cat facts","sg_event_id":"u_7gFLJfVlGwDm45B2nUPw==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","response":"250 OK"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"open","category":"cat facts","sg_event_id":"tP_AofoLLLp8uWmwHprL9A==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","useragent":"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","ip":"255.255.255.255"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"click","category":"cat facts","sg_event_id":"9kva_hyZcZren2AJ2bnsEA==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","useragent":"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","ip":"255.255.255.255","url":"http://www.sendgrid.com/"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"bounce","category":"cat facts","sg_event_id":"KWBrG2Ym8q1rxToRm9vhdw==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","reason":"500 unknown recipient","status":"5.0.0"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"dropped","category":"cat facts","sg_event_id":"c9SbbZS8Sm6pMHXjYnJZfw==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","reason":"Bounced Address","status":"5.0.0"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"spamreport","category":"cat facts","sg_event_id":"KKvsgacB8I8AlYg5pfNIyg==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"unsubscribe","category":"cat facts","sg_event_id":"aaz74YADki-ktyHMmrlYxw==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0"},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"group_unsubscribe","category":"cat facts","sg_event_id":"T0sX5h0JA7m2YbED-jolbQ==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","useragent":"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","ip":"255.255.255.255","url":"http://www.sendgrid.com/","asm_group_id":10},{"email":"[email protected]","timestamp":1592925630,"smtp-id":"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e","event":"group_resubscribe","category":"cat facts","sg_event_id":"bFCbttkX4OSxDj3xKPM4Yw==","sg_message_id":"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0","useragent":"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","ip":"255.255.255.255","url":"http://www.sendgrid.com/","asm_group_id":10}]

and here are the headers:

{
  "Accept-Encoding": [ "gzip" ],
  "Content-Length": [ "3698" ],
  "Content-Type": [ "application/json" ],
  "Host": [ "localhost:56736" ],
  "User-Agent": [ "Go-http-client/1.1" ],
  "X-Forwarded-For": [ "167.89.117.41" ],
  "X-Original-Host": [ "09676049fd42.ngrok.io" ],
  "X-Twilio-Email-Event-Webhook-Signature": [  "MEUCIQCeMcxBRAVPe2524K99rZ6TNnOAy4felQISmx87NMNSQgIgaLrcM7gYi4xBF0UtSDh9rtgDB9ncw8kp7hQOKddR0ik=" ],
  "X-Twilio-Email-Event-Webhook-Timestamp": [ "1592925630" ]
}

and finally, my public key is MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2is1eViXeZ9NwNbYKD/b51+WBZQVf+mLT0QCLiD6+HgWlNkrldvci/3m/o72GgCr3ilINxo9FpHElSHNnlYA7A==.

Code Snippet

Given this information, I wrote the following unit test (similar to the TestVerifySignature unit test):

[Fact]
public void TestSamplePayload()
{
  var publicKey = "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2is1eViXeZ9NwNbYKD/b51+WBZQVf+mLT0QCLiD6+HgWlNkrldvci/3m/o72GgCr3ilINxo9FpHElSHNnlYA7A==";
  var samplePayload = "[{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"processed\",\"category\":\"cat facts\",\"sg_event_id\":\"-w3n3K8nBCtORXZVD9wxWQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"deferred\",\"category\":\"cat facts\",\"sg_event_id\":\"27_-LKWAeSzRe_JdEA8N8g==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"response\":\"400 try again later\",\"attempt\":\"5\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"delivered\",\"category\":\"cat facts\",\"sg_event_id\":\"Vx4gXYiwyWNyGDRtSn-RWQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"response\":\"250 OK\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"open\",\"category\":\"cat facts\",\"sg_event_id\":\"hdB0Dxh27lTQDmSsN9-3zg==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"click\",\"category\":\"cat facts\",\"sg_event_id\":\"eQbgQ_Aff12HIhnwf3aN_A==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"bounce\",\"category\":\"cat facts\",\"sg_event_id\":\"mPsgXgr8qq1cWHFirZCoSA==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"reason\":\"500 unknown recipient\",\"status\":\"5.0.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"dropped\",\"category\":\"cat facts\",\"sg_event_id\":\"K1pmJN0j0TrvQ8VJza5mxg==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"reason\":\"Bounced Address\",\"status\":\"5.0.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"spamreport\",\"category\":\"cat facts\",\"sg_event_id\":\"k0y94nBNOaWkc0XTiTB53g==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"unsubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"LgPxePMfafBIVUv_HLF0ZQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"group_unsubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"GPB7vO-DgotTjSxE7Odpew==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\",\"asm_group_id\":10},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"group_resubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"W5Bm2KOpz0eNjj1qX7Yl6A==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\",\"asm_group_id\":10}]";
  var signature = "MEUCIQCkVB8ZeiGaWA6o3/PGnqNQgdqzOCERs6w999YTquAiDQIgdTAvHUk6+HMzLI//7NHWfIROPg//P+ZAo17QISy8Y1U=";
  var timestamp = "1592925285";

  var isValidSignature = Verify(
    publicKey,
    samplePayload,
    signature,
    timestamp
  );

  Assert.True(isValidSignature);
}

Unfortunately, my unit test does not succeed.

[thinkingserious]

Hello @Jericho!

Can you please try without the surrounding brackets on samplePayload?

Thanks!

With best regards,

Elmer

[Jericho]

Same result without the surrounding brackets.

[Jericho]

By the way, have you been able to run the unit test I included in my original comment? Do you observe the same problem?

[thinkingserious]

I have not been able to get the test you provided to pass. It looks like there was a similar issue here and a resolution was obtained. I will continue to investigate. Thank you for your patience.

[Mitchies1970]

Hi. I have been battling with this too and finally have it working.
As noted in the issue referred to above, the raw body (payload) is expected to have a carriage return at the end. When I finally added the carriage return manually, the validation of the signature passed. See below for my code in C#. Hope this helps.
Oh and please note I am using the SendGrid package.

string signatureInHeader = "MEQCIDwDTmSGOeTC0QEnba7kgPA......";

string timestampInHeader = "1595290381";

//The Key provided on SG Admin pages Mail Settings -> Signed Event Webhook Requests
string publicKeyFromSGWebhookAdminPage = "MFkwEwYHKoZ............";

//Raw body request.... not JSON formatted. Very important that the carriage return is at the end of the body (closing ] )
string payLoad = File.ReadAllText("BodyRaw.txt");

SendGrid.Helpers.EventWebhook.RequestValidator requestValidator
= new SendGrid.Helpers.EventWebhook.RequestValidator();

//Convert
var publicKey = requestValidator.ConvertPublicKeyToECDSA(publicKeyFromSGWebhookAdminPage);

bool isValid = requestValidator.VerifySignature(publicKey, payLoad, signatureInHeader, timestampInHeader);

Console.WriteLine($"Signature is valid: {isValid}");

[Jericho]

@Mitchies1970 thanks for sharing your experience. Unfortunately, adding carriage return and line feed at the end of the payload does not fix my scenario. I modified my unit test like so:

[Fact]
public void TestSamplePayload()
{
    var publicKey = "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2is1eViXeZ9NwNbYKD/b51+WBZQVf+mLT0QCLiD6+HgWlNkrldvci/3m/o72GgCr3ilINxo9FpHElSHNnlYA7A==";
    var samplePayload = "[{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"processed\",\"category\":\"cat facts\",\"sg_event_id\":\"-w3n3K8nBCtORXZVD9wxWQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"deferred\",\"category\":\"cat facts\",\"sg_event_id\":\"27_-LKWAeSzRe_JdEA8N8g==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"response\":\"400 try again later\",\"attempt\":\"5\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"delivered\",\"category\":\"cat facts\",\"sg_event_id\":\"Vx4gXYiwyWNyGDRtSn-RWQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"response\":\"250 OK\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"open\",\"category\":\"cat facts\",\"sg_event_id\":\"hdB0Dxh27lTQDmSsN9-3zg==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"click\",\"category\":\"cat facts\",\"sg_event_id\":\"eQbgQ_Aff12HIhnwf3aN_A==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"bounce\",\"category\":\"cat facts\",\"sg_event_id\":\"mPsgXgr8qq1cWHFirZCoSA==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"reason\":\"500 unknown recipient\",\"status\":\"5.0.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"dropped\",\"category\":\"cat facts\",\"sg_event_id\":\"K1pmJN0j0TrvQ8VJza5mxg==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"reason\":\"Bounced Address\",\"status\":\"5.0.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"spamreport\",\"category\":\"cat facts\",\"sg_event_id\":\"k0y94nBNOaWkc0XTiTB53g==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"unsubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"LgPxePMfafBIVUv_HLF0ZQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"group_unsubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"GPB7vO-DgotTjSxE7Odpew==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\",\"asm_group_id\":10},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"group_resubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"W5Bm2KOpz0eNjj1qX7Yl6A==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\",\"asm_group_id\":10}]";
    var signature = "MEUCIQCkVB8ZeiGaWA6o3/PGnqNQgdqzOCERs6w999YTquAiDQIgdTAvHUk6+HMzLI//7NHWfIROPg//P+ZAo17QISy8Y1U=";
    var timestamp = "1592925285";

    samplePayload += "\r\n";

    var isValidSignature = Verify(
      publicKey,
      samplePayload,
      signature,
      timestamp
    );

    Assert.True(isValidSignature);
}

I tried both \r\n and \r but neither solved the problem.

Also, quick observation: the payload in the unit test written by @childish-sambino does not have a trailing \r\n which proves that carriage return and line feed are not always necessary at the end of the payload.

[eshanholtz]

I've been able to reproduce this bug. This issue has been added to our internal backlog to be prioritized. Pull requests and +1s on the issue summary will help it move up the backlog.

[sdion-cmq]

Any updates on this ? I am facing the same issue with the sendgrid-java library. Adding a trailing carriage return at the end of the payload did solve the issue as mentioned by @Mitchies1970.

But like @Jericho said, the provided example doesn't have a carriage return at the end. I guess the payload used for the example was just a string that was then signed with the private key instead of an actual event being sent to sendgrid and passing through their whole flow where something does add a carriage return before signing it but doesn't return it as part of the actual payload

I can go ahead and add the carriage return and fix my issue, but when the issue is fixed on your side, will this introduce a breaking change where the added carriage return will break signature validation ?

Thanks

[thinkingserious]

Hello @sdion-cmq, thanks for reaching out with further detail!

We don't have a fix just yet.

I tried adding the carriage return on Jericho's example, also in our Python and Java helper libraries with no luck. It looks like there is also some other issue that we are still trying to track down.

I'm not sure about if the fix will be a breaking change.

Any further updates will be posted here.

[sdion-cmq]

Hi @thinkingserious
Thanks for the quick update. Don't hesitate if you need more info from my side to reproduce this with the Java helper library.
And sorry for hijacking this thread in the C# repo. I figured it's probably the same root cause, regardless of the client library used to validate the signature. If that's not the case I can open an issue in the Java repo too.

I'll also add that although the carriage return fixes my validation, I too cannot successfully validate Jericho's example with it. Could it be related to using the test endpoint that sends a sample payload vs getting a real payload from a real sendgrid event ? (which is what I am doing for my test)

Thanks !

[thinkingserious]

@Jericho, @sdion-cmq,

Can you both verify how you are reading the payload that is POSTed by Twilio SendGrid?

It is expected that the payload is read as raw bytes. Thanks!

[sdion-cmq]

Hi @thinkingserious
I am not sure I understand. I am reading the payload as a String since the Java helper expects the payload as a String. The C# helper does as well.

It seems the helper libs actually convert the string back to bytes before validating here : https://github.com/sendgrid/sendgrid-java/blob/e71c7b09936715c4a9e39eb10c620b5a41385b11/src/main/java/com/sendgrid/helpers/eventwebhook/EventWebhook.java#L60

We tried reading the payload as bytes and then converting it to String to use the helper, but we end up with the same payload that doesn't have a carriage return and thus doesn't validate with the signature.

Thanks !

[thinkingserious]

Thank you for the additional information @sdion-cmq!

We are still investigating and your additional insight is helpful, thanks again for taking the time to elaborate.

[thinkingserious]

@Jericho,

To get your unit test to pass, please generate the base64 encoding of the payload one time and use that string literal, signature, and timestamp for the request in your unit test.

Here is a dotnetcore mvc controller method that we used to verify the SDK verification is working when reading the body as-is:

public IActionResult Index()
{
    string body;
    using (var reader = new StreamReader(Request.Body))
    {
        body = reader.ReadToEndAsync().Result;
    }

    var data = System.Text.ASCIIEncoding.ASCII.GetBytes(body);
    var b64 = System.Convert.ToBase64String(data);
    Console.WriteLine(b64);

    //var b = System.Convert.FromBase64String("BASE64_STRING_LITERAL_OF_THE_PAYLOAD_GOES_HERE");
    //body = System.Text.ASCIIEncoding.ASCII.GetString(b);

    RequestValidator validator = new RequestValidator();

    string signature = Request.Headers[RequestValidator.SIGNATURE_HEADER];
    string timestamp = Request.Headers[RequestValidator.TIMESTAMP_HEADER];
    Console.WriteLine(signature);
    Console.WriteLine(timestamp);

    var publicKey = validator.ConvertPublicKeyToECDSA("YOUR_PUBLIC_KEY_GOES_HERE");
    Console.WriteLine(validator.VerifySignature(publicKey, body, signature, timestamp));


    return View();
}

Please let us know if that works for you. Thanks!

@sdion-cmq,

Do you mind creating a new issue at [sendgrid-java](https://github.com/sendgrid/sendgrid-java/issues) with sample code used to generate the payload?

[thinkingserious]

Hello Everyone!

A few updates:

• Updated Getting Started Guide Docs
• Updated Event Webhook Integration Tester

[Jericho]

@thinkingserious I have modified my unit test according to your instruction but it doesn't seem to solve the problem:

[Fact]
public void TestSamplePayload()
{
  var publicKey = "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2is1eViXeZ9NwNbYKD/b51+WBZQVf+mLT0QCLiD6+HgWlNkrldvci/3m/o72GgCr3ilINxo9FpHElSHNnlYA7A==";
  var samplePayload = "[{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"processed\",\"category\":\"cat facts\",\"sg_event_id\":\"-w3n3K8nBCtORXZVD9wxWQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"deferred\",\"category\":\"cat facts\",\"sg_event_id\":\"27_-LKWAeSzRe_JdEA8N8g==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"response\":\"400 try again later\",\"attempt\":\"5\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"delivered\",\"category\":\"cat facts\",\"sg_event_id\":\"Vx4gXYiwyWNyGDRtSn-RWQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"response\":\"250 OK\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"open\",\"category\":\"cat facts\",\"sg_event_id\":\"hdB0Dxh27lTQDmSsN9-3zg==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"click\",\"category\":\"cat facts\",\"sg_event_id\":\"eQbgQ_Aff12HIhnwf3aN_A==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"bounce\",\"category\":\"cat facts\",\"sg_event_id\":\"mPsgXgr8qq1cWHFirZCoSA==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"reason\":\"500 unknown recipient\",\"status\":\"5.0.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"dropped\",\"category\":\"cat facts\",\"sg_event_id\":\"K1pmJN0j0TrvQ8VJza5mxg==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"reason\":\"Bounced Address\",\"status\":\"5.0.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"spamreport\",\"category\":\"cat facts\",\"sg_event_id\":\"k0y94nBNOaWkc0XTiTB53g==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"unsubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"LgPxePMfafBIVUv_HLF0ZQ==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\"},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"group_unsubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"GPB7vO-DgotTjSxE7Odpew==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\",\"asm_group_id\":10},{\"email\":\"[email protected]\",\"timestamp\":1592925285,\"smtp-id\":\"\u003c14c5d75ce93.dfd.64b469@ismtpd-555\u003e\",\"event\":\"group_resubscribe\",\"category\":\"cat facts\",\"sg_event_id\":\"W5Bm2KOpz0eNjj1qX7Yl6A==\",\"sg_message_id\":\"14c5d75ce93.dfd.64b469.filter0001.16648.5515E0B88.0\",\"useragent\":\"Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\",\"ip\":\"255.255.255.255\",\"url\":\"http://www.sendgrid.com/\",\"asm_group_id\":10}]";
  var signature = "MEUCIQCkVB8ZeiGaWA6o3/PGnqNQgdqzOCERs6w999YTquAiDQIgdTAvHUk6+HMzLI//7NHWfIROPg//P+ZAo17QISy8Y1U=";
  var timestamp = "1592925285";

  var data = System.Text.ASCIIEncoding.ASCII.GetBytes(samplePayload);
  var b64 = System.Convert.ToBase64String(data);
  Console.WriteLine(b64);

  var isValidSignature = Verify(
    publicKey,
    b64,
    signature,
    timestamp
  );

  Assert.True(isValidSignature);
}

By the way, I find it odd that I would have to base64-encode the payload while your own unit test don't seem to required base64-encoding their payload: https://github.com/sendgrid/sendgrid-csharp/blob/main/tests/SendGrid.Tests/Helpers/EventWebhook/RequestValidatorTests.cs

[eshanholtz]

I'm re-marking this issue as a bug. The validators should be updated to accept bytes for the payload, per the SendGrid documentation. The test suite should be updated based on a live example webhook, and should include the necessary carriage return that is included in live SendGrid webhook payloads.
This issue has been added to our internal backlog to be prioritized. Pull requests and +1s on the issue summary will help it move up the backlog.

@sdion-cmq, please create a new issue in sendgrid-java to properly track bugs related to the sendgrid-java request validator.

[childish-sambino]

Fixed by https://github.com/sendgrid/sendgrid-csharp/commit/bb5337f7f8ac63906178bea823cb781758f3a009

Strings are fine. I've included actual request data in the test (array of events with trailing\r\n, don't strip this out).

[brian-fuji]

I am still having an issue trying to get my WebHook to Authorise withe the following code. I am unsure what should be sent as the payload? Is what I have correct? I have tried manually setting the "sendgridkey" in this code as well, but still getting unauthorised?

public async Task<IHttpActionResult> EmailHook([FromBody] object jsonData)
        {
            var parser = new WebhookParser();
            try
            {
                IList<WebhookEventBase> events;

                var jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(jsonData) + "\r\n";

                var validator = new RequestValidator();

                var signature = string.Empty;
                var timestamp = string.Empty;

                if (Request.Headers.Contains(RequestValidator.SIGNATURE_HEADER) && Request.Headers.Contains(RequestValidator.TIMESTAMP_HEADER))
                {
                    signature = Request.Headers.FirstOrDefault(x => x.Key == RequestValidator.SIGNATURE_HEADER).Value?.First();
                    timestamp = Request.Headers.FirstOrDefault(x => x.Key == RequestValidator.TIMESTAMP_HEADER).Value?.First();

                    var sendgridKey = ConfigurationManager.AppSettings["SendgridWebhookKey"];

                    var publicKey = validator.ConvertPublicKeyToECDSA(sendgridKey);

                    if (!validator.VerifySignature(publicKey, jsonString, signature, timestamp))
                        return Unauthorized();
                }
                else
                {
                    return Unauthorized();
                }

                events = parser.ParseEvents(jsonString);
}

[belezi]

Hi guys I had the same issue with Sendgrid-java library. going through the documentation it states that sendgrid already sends the payload as a byte array so for me reading it as a byte array and not modifying or parsing the payload whatsoever worked. This is a snipped of my code I hope that it helps.

My resource:

public Response storeSendGridEvent(
            @HeaderParam("x-twilio-email-event-webhook-signature") @NotNull String signature,
            @HeaderParam("x-twilio-email-event-webhook-timestamp") @NotNull String timestamp,
            @ApiParam("Event Object") @NotNull byte[] events){
        Event event = new Event("email", signature, timestamp, events);
        NotificationEngine notificationEngine = new NotificationEngine(event);

        List<WebhookEventResponse> responses;

        try {
            responses = notificationEngine.processWebhookEvent();
        } catch (Exception e) {
            ArrayList<Object> errors = new ArrayList<>();
            errors.add(e.getMessage());
            return ResponseUtils.respond(Response.Status.BAD_REQUEST,null,null, errors);
        }
        return ResponseUtils.respond(Response.Status.OK, responses,null,null);
    }

Process method

@Override
    public List<WebhookEventResponse> processWebhookEvent(String signature, String timeStamp, byte[] payload) throws IOException, NoSuchAlgorithmException, SignatureException, NoSuchProviderException, InvalidKeySpecException, InvalidKeyException {
        try {
            Boolean authorization = verifySignature(signature, timeStamp, payload);
            System.out.println("Signature verification evaluates to: " + authorization);
        } catch (Exception e) {
            System.out.println(e.getMessage());
            e.printStackTrace();
        }

        List<WebhookEventResponse> webhookEventResponseList = new ArrayList<>();
        if (verifySignature(signature, timeStamp, payload)) {
            ObjectMapper objectMapper = new ObjectMapper();
            for (Object event: payload){
                JsonNode jsonNode = objectMapper.readTree(String.valueOf(event));

                String messageId = jsonNode.get("sg_message_id").asText();
                String timestamp = jsonNode.get("timestamp").asText();
                String eventName = jsonNode.get("event").asText();

                WebhookEventResponse webhookEventResponse = new WebhookEventResponse();
                webhookEventResponse.setMessage(messageId);
                webhookEventResponse.setStatus(eventName);
                webhookEventResponse.setErred(false);
                webhookEventResponseList.add(webhookEventResponse);
            }
        }
        return webhookEventResponseList;
    }

public Boolean verifySignature(String signature, String timeStamp, byte[] payload) throws IOException, NoSuchProviderException, NoSuchAlgorithmException, SignatureException, InvalidKeyException, InvalidKeySpecException {
        Security.addProvider(new BouncyCastleProvider());

        // prepend the payload with the timestamp
        final ByteArrayOutputStream payloadWithTimestamp = new ByteArrayOutputStream();
        payloadWithTimestamp.write(timeStamp.getBytes());
        payloadWithTimestamp.write((byte[]) payload);

        // create the signature object
        final Signature signatureObject = Signature.getInstance("SHA256withECDSA", "BC");

        signatureObject.initVerify(ConvertPublicKeyToECDSA(SendGridConfiguration.getInstance().getSignaturePublicKey()));
        signatureObject.update(payloadWithTimestamp.toByteArray());

        // decode the signature
        final byte[] signatureInBytes = Base64.getDecoder().decode(signature);

        // verify the signature
        return signatureObject.verify(signatureInBytes);
    }

    public java.security.interfaces.ECPublicKey ConvertPublicKeyToECDSA(String publicKey)
            throws NoSuchAlgorithmException, NoSuchProviderException, InvalidKeySpecException {
        byte[] publicKeyInBytes = Base64.getDecoder().decode(publicKey);
        KeyFactory factory = KeyFactory.getInstance("ECDSA", "BC");
        return (ECPublicKey) factory.generatePublic(new X509EncodedKeySpec(publicKeyInBytes));
    }