Grpc-gateway: Access to raw post body

[nscharfe]

I'm implementing a webhook handler and need to perform a checksum on the raw post body that is received but cannot figure out how to access it.

Sinatra Equivalent:

post '/my/webhook/url' do
    body = request.body.read
    ...
end

Django Equivalent:

def my_webhook_view(request):
  body = request.body
  ...

Node Equivalent:

app.use(require('body-parser').raw({type: '*/*'}));

app.post('/my/webhook/url', function(request, response) {
  body = request.body;
  ...
});

---

My original implementation was:

message WebhookRequest {
    string body = 1;
}

rpc HandleWebhook(WebhookRequest) returns (google.protobuf.Empty) {
    option (google.api.http) = {
        post: "/public/processWebhook"
        body: "body"
    };
}

This works as expected if the post body is explicitly a string. However, if the post body is a json object, it tries to unmarshal it into body, which is a string, and fails.

Ex post bodies:
"test" => is unmarshaled into WebhookRequest.body appropriately.
{"test": "val"} => "error": "json: cannot unmarshal object into Go value of type string"
"{\"test\": \"val\"}" => is unmarshaled into WebhookRequest.body

Since this is a webhook, I have no control over how the post body is constructed -- its not possible to encode it as a string before sending it over the wire.

I've tried updating WebhookRequest to hold a google.protobuf.Struct, google.protobuf.Any etc but none of them are able to preserve the post body in its raw form.

I'd imagine its possible to accomplish what I want by customizing message serialization but that feels like overkill.

Any help would be greatly appreciated. I'd imagine I'm overlooking something obvious.

Thanks in advance!

[achew22]

Unfortunately I think you're right that this is probably better handled by a custom marshaller. The proto API, as I understand it, isn't supposed to be able to form to every REST request, only to allow an RPC to be exported in a way that any browser can connect to.

If you make a custom marshaller you will have access to the full body message for hashing. I would be interested in adding that as an example marshaller if you would like to share that code.

PS: Why is there a message checksum in the web hook? Is this a common pattern that I've not yet run into?

[nscharfe]

Thanks for the help. Now that I'm looking a bit closer, it may make sense for me to simply attach the body to the context -- headers, metadata etc are already being attached so it may be a natural extension to add body as well for this specific purpose rather than try to shoe horn a new marshaling scheme. That seem reasonable?

As far as verifying signatures, checksums etc. in a webhook handler, I believe it is a common practice to ensure the webhook event was sent by the correct party. For example:
https://stripe.com/docs/webhooks/signatures
In this case, the official stripe lib handles the signing verification internally but expects to receive the raw post payload as input.

[achew22]

Okay, so it is more about signing than hashing. That makes sense for message integrity.

If I were going to implement this, I think I would implement it as a HTTP interceptor in the grpc-gateway, look for the Stripe-Signature header and if it has it, verify it is correct. If it fails that check, immediately return a 400 error (bad request) or a 412 (precondition failed) to the caller and not pass the request into the stack.

[hiromis]

I came across this issue while trying to create an endpoint that takes an arbitrary JSON, modifies one field, and forwards if to another RESTful service.

I considered creating a custom marshaller, but it seems like it's at MIME type level and not for a particular endpoint. All the other endpoints should use the default marshaller, but for one endpoint I would like an access to the body where I do not the structure of in advance. Any advice on how I might able to achieve that?

[rogchap]

Create a wrapper for your endpoint to set a custom MIME Content-Type then have a custom Marshaler for that MIME.

eg:

func customMimeWrapper(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if strings.Contains(r.URL.Path, "webhook") {
            r.Header.Set("Content-Type", "application/raw-webhook")
        }
        h.ServeHTTP(w, r)
    })
}

mux = runtime.NewServeMux(
    runtime.WithMarshalerOption("application/raw-webhook", & rawJSONPb{jsonpb}),
    runtime.WithMarshalerOption(runtime.MIMEWildcard, jsonpb),
    runtime.WithProtoErrorHandler(runtime.DefaultHTTPProtoErrorHandler),
)

restHandler = http.NewServeMux()
restHandler.Handle("/", customMimeWrapper(restServer))

[rogchap]

For those trying to do the same think this is what my marshaler consists of:

var typeOfBytes = reflect.TypeOf([]byte(nil))

type rawJSONPb struct {
    *gateway.JSONPb
}

func (*rawJSONPb) NewDecoder(r io.Reader) runtime.Decoder {
    return runtime.DecoderFunc(func(v interface{}) error {
        rawData, err := ioutil.ReadAll(r)
        if err != nil {
            return err
        }
        rv := reflect.ValueOf(v)

        if rv.Kind() != reflect.Ptr {
            return fmt.Errorf("%T is not a pointer", v)
        }

        rv = rv.Elem()
        if rv.Type() != typeOfBytes {
            return fmt.Errorf("Type must be []byte but got %T", v)
        }

        rv.Set(reflect.ValueOf(rawData))
        return nil
    })
}

expects a proto definition that uses bytes as the body:

service WebhookService {
  rpc HandleWebhook(WebhookRequest) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      post: "/myapi/v1/webhook"
      body: "rawData"
    };
  }
}
message WebhookRequest {
  bytes rawData = 1;
}

[hiromis]

@rogchap, it worked like a charm. Thank you!!

[johanbrandhorst]

Can we close this issue?

[nscharfe]

yep, feel free. thank you!

[arturgspb]

@rogchap

I got error with same code on compile:

kind: ERROR
message: "http: body field path \'rawData\' must be a non-repeated message."

Have you had such problems?