Moleculer: [Tracing] Default API service with jaeger tracing causes mem buff overflow

Created on 31 Oct 2019  路  7Comments  路  Source: moleculerjs/moleculer

Current Behavior

Memory allocation error when adding jaeger tracing to API Gateway from base project.

Expected Behavior

Public folder contents are displays and memory allocation does not cause node crash.

Failure Information

api_1 | <--- Last few GCs ---> api_1 | nce start of marking 170 ms) (average mu = 0.599, current mu = 0.010) allocatio[56:0x55f4312e80c0] 411740 ms: Mark-sweep 1401.7 (1454.9) -> 1401.7 (1454.9) MB, 169.4 / 2.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 171 ms) (average mu = 0.411, current mu = 0.009) last reso[56:0x55f4312e80c0] 411908 ms: Mark-sweep 1401.7 (1454.9) -> 1401.7 (1454.9) MB, 168.7 / 1.9 ms (average mu = 0.253, current mu = 0.000) last resort GC in old space requested api_1 | api_1 | api_1 | <--- JS stacktrace ---> api_1 | api_1 | ==== JS stack trace ========================================= api_1 | api_1 | 0: ExitFrame [pc: 0x20c9e72dbe1d] api_1 | Security context: 0x24b600f9e6e9 <JSObject> api_1 | 1: flattenTags [0x162fdeb23051] [/app/node_modules/moleculer/src/tracing/exporters/base.js:~77] [pc=0x20c9e79c7bf4](this=0x08f8dc6e0e31 <BaseTraceExporter map = 0x1ad56bd26489>,/* anonymous */=0x0b822f5058a9 <Socket map = 0x1ad56bd36551>,/* anonymous */=0x1988eec029a1 <false>,/* anonymous */=0x09d15c67fd71 <Very long string[19463]>) api_1 | 2: flattenTags [0... api_1 | api_1 | FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

// moleculer config excerpt

tracing: {
enabled: true,
exporter: {
type: "Jaeger",
options: {
// HTTP Reporter endpoint. If set, HTTP Reporter will be used.
endpoint: null,
// UDP Sender host option.
host: "jaeger",
// UDP Sender port option.
port: 6832,
// Jaeger Sampler configuration.
sampler: {
// Sampler type. More info: https://www.jaegertracing.io/docs/1.14/sampling/#client-sampling-configuration
type: "Const",
// Sampler specific options.
options: {}
},
// Additional options for Jaeger.Tracer
tracerOptions: {},
// Default tags. They will be added into all span tags.
defaultTags: null
}
}
},

### Steps to Reproduce

1. Load the below code sandbox or add Jaeger tracing config to broker config
2. Add moleculer-web mixin with a public facing directory
3. Load the service and visit via the browser and it should never resolve

### Reproduce code snippet

Reproducible below, could be simply due to the host not being able to connect in the jaeger-client

https://codesandbox.io/s/moleculer-sample-4ir3h

### Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

* Moleculer version: 0.14
* NodeJS version: 10
* Operating System: node:10-alpine

### Failure Logs

```

Tracing

All 7 comments

@icebob saw the tag, I've got a repro link to codesandbox in the ticket. Is there anything else you'd like me to provide?

I believe a trace should be allowed from the ApiGateway. It's helpful to get a full trace from the say GraphQL resolver to the db and back. Hopefully I can find some time to dig into why it's happening in the source but figured I'd file a bug to track it.

@derekbar90 no, it's good but i'm on holiday and i can't check it right now. So the tag is for me :)

Enjoy the holiday and thanks for all the hard work and dedication.

I've tested here and I was able to repro the error. It seems that it's caused by the params: {} that has req and res objects.
image

Then the flattenTags function tries to recursively flat these objects:
https://github.com/moleculerjs/moleculer/blob/de6c167a62ec019506dfbb383742e292dc457806/src/tracing/exporters/base.js#L77-L92

I'm going to dig further and try to fix it.

After further inspection I believe that this happens only with v0.8.5
@derekbar90 can you try to install v0.9 npm i --save moleculer-web@next and test it?

Ok, I found the issue. v0.8.5 was released long before the v0.14 which has built-in tracing features. Because of this [email protected] does not have tracing params disabled:

    actions: {
        // REST request handler
        rest: {
            visibility: "private",
                        // MISSING CONFIG in v0.8.5
            tracing: {
                tags: {
                    params: false
                }
            },
            handler(ctx) {
                const req = ctx.params.req;
                const res = ctx.params.res;

Just update to v0.9 and the issue will be gone

My current dep config is at:

"moleculer-web": "^0.9.0-beta6",

That's strange :confused: I can only reproduce the issue with v0.8.5
Can you please create a repro repo with v0.9? The one that you've provided (https://codesandbox.io/s/moleculer-sample-4ir3h) uses v0.8.5

Was this page helpful?
0 / 5 - 0 ratings