Moleculer: Metric enhancements

[icebob]

Branch: metrics2

Metric event payload changes

• [ ] add service as service name
• [ ] add type as event type. Values: call (current), custom - user generated event
• [ ] add error.data
• [ ] _add error.stack (not sure, it can generate big overhead)_

New Context method to create custom metric spans

• [ ] ctx.metric(name, params); - start a new metric span event

Example

{
    actions: {
        async update(ctx) {
            // ... Do something
            const span = ctx.metric("update-db", { id: 3 });

            await this.db.updateById(3, {});

            span.end();
            // ... Do something
        }
    }
}

[intech]

Hi, @icebob ! :hand:

Using the Elastic APM Agent, I got the following results:

:exclamation: The data collected through the metric events span and processed by the moleculer-elastic-apm.
See what I found problems with the categorization of data using the molecules molecular-web, molecular-bull and molecular-elasticsearch

:point_up_2: Look at the module built into the Elastic APM that defines the operations of ioredis, pg and elasticsearch packages. They look good.
Internal calls to microservices are not such nice types and names in my report.
This information will help to come up with an excellent solution.
I'll be glad to help! :wink:

[icebob]

Thank you!

Yes, it looks very pretty!

[icebob]

@intech I'm starting to rewrite tracing in Moleculer. What are your suggestions to be it better?

[intech]

@icebob I will describe the thoughts that I see. But I'm not so good at understanding all the internal processes of the moleculer and I can not say about the correctness of my decisions.
Think about the best solution based on your expert experience.

1. The most important thing I encountered is the division of transactions.
   https://github.com/elastic/apm-agent-nodejs/issues/376
   We need to build logical levels so that the moleculer-web participates in these measurements as a single transaction from the first http(s) call and until the last service response.
   The task is very much and I did not have time to study the problem in detail.
   According to my tests, you need to get more information from moleculer-web in the metric https://github.com/moleculerjs/moleculer-web/blob/master/src/index.js#L667 information about the query parameters before calling the broker. Also, the moleculer-web response must conform to the HTTP standard (200 OK) and contain the response body from the service. In this way, I can supplement the requests to the service in a few span with information about the request itself.

2. You must store the chain of the route node1 (v1.api.get) -> node4 (elasticsearch.search) -> node9 (redis.cache) in the metric to build the query tree.

3. How do you look at adding standard events besides tracing, which will show service calls by name with parameters. I've seen a nearby issue about hooks, but they only solve the problem sequentially and do not decide if to do a parallel task.

[icebob]

Hi @intech,
yesterday I've checked the Elastic APM docs & sources. I think it is not a good choice to monitoring & tracing microservices. The "only-one-active-transaction" concept can't trace microservices properly. NodeJS executes multiple requests in parallel, so you have a lot of unfinished transactions because a new startTransaction overwrites the previous active transaction.
On the other hand, you can't share transactions between processes. They can run only on the process where they've been started.
Finally, the spans have no parents (just the transaction as a root). So you can't build a trace tree from spans what Zipkin & Jaeger can do it. Moreover, you can't store this information in spans.

These issues are in the Elastic APM server & clients, what we can't solve it in a Moleculer tracing module.
Btw, I like that there is a free APM solution with a nice dashboard, but I think it should evolve to support microservices architecture tracing.

[intech]

@icebob you are right! Now I know which direction to continue developing apm elastic and moleculer dashboard.