Grpc-gateway: Expose detailed metrics/traces for gateway internals

[activeshadow]

Curious if there's a way or a best practice to expose gateway metrics? I'm looking for things like context timeouts, cancellations, etc. Perhaps through interceptors?

[achew22]

Have you tried out our sister project, https://github.com/grpc-ecosystem/go-grpc-prometheus?

[activeshadow]

Thanks @achew22 was just looking at that. Didn't see it until after I posted this question, and was going to post an update here if it worked for me.

[activeshadow]

Is it possible to open this issue up again? I'm coming back around to using grpc-gateway, and I'm not seeing how best to implement metrics. I looked more into go-grpc-prometheus, but from what I can tell it's really only meant for usage server-side. Since the grpc-gateway is client-side from a gRPC perspective I'm not seeing how to make it work. I tried adding grpc_prometheus.UnaryInterceptor as a client-side interceptor via the DialOptions passed to the handlers in the grpc-gateway, but got errors.

I'd also like to see metrics on the normal HTTP server side of things a well.

Thoughts? Are there any examples of this use case that I'm just not seeing?

[aranw]

I've been looking at this myself today and been hoping to get metrics from the gateway for the HTTP handlers with metric tags similar to something like /test/:parameter and the method used. Has there been further discussion on supporting such a thing? Looking at the gateway code it seems everything that is needed is inside the generated code just doesn't seem to be accessible to any middleware

[johanbrandhorst]

There has not been any more discussion around this. The proposed solution, using a client-side interceptor should provide a lot of the value, but something more detailed around the gateway internals would be interesting too. I have a bit of interest in this space and I'd be interested in implementing something like OpenTelemetry metrics and traces into the gateway, but I don't have the time for it myself. I'd be happy to consider contributions in this space, though.

[maruina]

@aranw did you get it working? I have metrics for the gRPC calls, but I can't get any for the HTTP ones.

[johanbrandhorst]

This can be added in a http.Handler wrapping the runtime.ServeMux, like in the tracing example in the docs: https://grpc-ecosystem.github.io/grpc-gateway/docs/customizingyourgateway.html. More route-specific metrics are not auto generated at the moment, but could be considered.

[maruina]

Thanks, I will have a look!

[marcosnils]

@maruina @aranw just implemented HTTP metrics today. Particularly I've used https://github.com/slok/go-http-metrics in conjunction with this project to make it it work. Here's a code snippet on how to initialize things:

    grpcMux := grpc.NewServer(
        grpc.StreamInterceptor(grpc_prometheus.StreamServerInterceptor),
        grpc.UnaryInterceptor(grpc_prometheus.UnaryServerInterceptor),
    )

    if *enableGRPCReflection {
        reflection.Register(grpcMux)
    }

    gatewayMux := runtime.NewServeMux()

        // register your gRPC services here.

    mux := http.NewServeMux()
    mux.Handle("/metrics", promhttp.Handler())
    mux.Handle("/", gatewayMux)

    // Prometheus go-http-metrics middleware
    mdlw := middleware.New(middleware.Config{
        Recorder:           metrics.NewRecorder(metrics.Config{}),
        DisableMeasureSize: true,
    })
    h := std.Handler("", mdlw, mux)

    grpc_prometheus.Register(grpcMux)

    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // https://github.com/philips/grpc-gateway-example/blob/a269bcb5931ca92be0ceae6130ac27ae89582ecc/cmd/serve.go#L55
        if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
            grpcMux.ServeHTTP(w, r)
            return
        }
        h.ServeHTTP(w, r)
    })
    return &http.Server{
        Addr:    addr,
        Handler: h2c.NewHandler(handler, &http2.Server{}),
    }, nil

Once you have this ready, this tracks both gRPC and HTTP metrics.

[dimpavloff]

Hi,

Just to add to the above feature request from the PoV of metrics.

I don't think the suggested workaround of just wrapping the runtime.ServeMux is sufficient because this does not have access to the Pattern (or similar) for a given request. Therefore, either one has to ignore the HTTP Path or observe r.URL.Path in their metrics. The former makes metrics meaningless. The problem with the latter is that often times the path contains identities (e.g. /v1/user/<uuid>/) which can lead to a cardinality explosion for metrics, again making metrics useless (as the metrics server would be down :) ).

I don't think relying solely on gRPC Client Interceptors is sufficient for performance metrics because this omits the JSON <-> protobuf conversions which are a significant part of where the gateway spends its time.

I don't know what would be the best way to implement but it seems to me that it should be possible to set some information on the request context just before calling the handler ( https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/mux.go#L239 ) , allow for inserting middleware handlers after the match is performed, and then provide a method to extract the matched Pattern (or its String() representation). This is what gorilla/mux does afaict ( https://github.com/gorilla/mux/blob/v1.8.0/mux.go#L199 and https://github.com/gorilla/mux/blob/v1.8.0/mux.go#L439 )

What do you think?

[marcosnils]

The problem with the latter is that often times the path contains identities (e.g. /v1/user/<uuid>/) which can lead to a cardinality explosion for metrics, again making metrics useless (as the metrics server would be down :) ).

You can configure go-http-metrics to handle these cases. If you have something like /v1/user/<uuid> you can configure the middleware to track it as /v1/users/:id in prometheus without having cardinality explosion. It has been running like this in production for me and I haven't seen any issues so far.

[johanbrandhorst]

If you have both a middleware wrapper around the gateway mux _and_ a client-side grpc interceptor, you can see the time spent inside the gateway (if not exactly how much is spent on the JSON<->Protobuf marshalling), both with metrics and with tracing if you use that.

We're happy to consider proposals for exposing more information about the internals of the gateway for users, but I'm really weary of the gateway becoming more and more of a general purpose HTTP mux, which is not within the scope of this project. The win for the user has to be sufficiently high for the maintenance cost and user friendliness cost it adds. More configuration (something this project already explodes with) makes it more difficult for beginners to get started.

[marcosnils]

If you have both a middleware wrapper around the gateway mux _and_ a client-side grpc interceptor, you can see the time spent inside the gateway

Exactly.. the proposed solution I posted before provides this.

[dimpavloff]

Thanks for the swift responses, @marcosnils and @johanbrandhorst , appreciate them!

You can configure go-http-metrics to handle these cases. If you have something like /v1/user/ you can configure the middleware to track it as /v1/users/:id in prometheus without having cardinality explosion

Maybe I'm missing something but doesn't this require explicitly setting all the paths via calls to Handler from go-http-metrics or similar? Otherwise doesn't it default to the request's Path ( https://github.com/slok/go-http-metrics/blob/master/middleware/middleware.go#L74 ) ?
If it requires explicit registration calls for each endpoint, then this is duplicating information that has been defined within the proto annotations and therefore it is an error-prone maintenance burden that's better avoided. For large projects this is quite noticeable.
Alternatively, it requires importing the .proto srcs at runtime and using reflection on them and then duplicating some of the autogenerated code (what's in Register*HandlerFromEndpoint and Register*HandlerClient) -- this feels like a lot of work just to get to the Pattern string. Please do let me know if I'm missing a more obvious way.

If you have both a middleware wrapper around the gateway mux and a client-side grpc interceptor, you can see the time spent inside the gateway (if not exactly how much is spent on the JSON<->Protobuf marshalling), both with metrics and with tracing if you use that.

Agreed but my point is that it is difficult atm to have both unless the HTTP pattern is exposed in the metrics.

We're happy to consider proposals for exposing more information about the internals of the gateway for users, but I'm really weary of the gateway becoming more and more of a general purpose HTTP mux, which is not within the scope of this project. The win for the user has to be sufficiently high for the maintenance cost and user friendliness cost it adds. More configuration (something this project already explodes with) makes it more difficult for beginners to get started.

I definitely appreciate this point, +1 for simplicity. But IMO we're not talking about an edge case -- most services in production would benefit from such metrics and therefore would have to re-do one of the 2 options above. Are there alternative approaches that could avoid such concerns?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.