Opentelemetry-js: Create the transport layer and refactor exporters so that they can be used both in node and web

[obecny]

Now the jaeger and zipkin depends on http node module. We could create either in base or core a transport layer that is platform specific and then refactor exporters so that they can be used in node and browser.

[OlivierAlbertini]

just a note. Jaeger exporter doesn't use http node module.

[obecny]

Yes you right jaeger uses the udp sender but still you won't be able to use it in browser. I used some shortcut but my idea is to be able to use all exporters in web and then simply decide what to use as a transporter.

[mayurkale22]

IMO the ideal way is to have the browser clients send/write to the OpenTelemetry Collector via HTTP/JSON and have the collector send to one or more tracing backends.

The best reference architecture would be Browser > OpenTelemetry Agent > OpenTelemetry Collector > Destination(s). WDYT?

[obecny]

Lets take the real life example - exactly what I'm doing now.
As a developer I'm trying to add a new plugin (Document Load) in this moment. I want to be able to easily debug what I'm doing inside the browser and I want to see the results and spans easily - can be in console as a start and then maybe I would like those spans to sent to jaeger or zipkin (locally) and then compare if it looks as I think it should. Once I do this I will have a better understanding how all elements are working together and where to add what or what could be improved too. For this I had to add a simple new package ConsoleExporter so that I can see the spans in console - Maybe there was a simpler way but I wasn't able to find it / understand that easily.

My current updated example for web tracer looks like this

import { WebTracer } from '@opentelemetry/web';
import { ConsoleExporter } from '@opentelemetry/exporter-console';
import { DocumentLoad } from '../../packages/opentelemetry-plugin-document-load/src/index';

const webTraxcer = new WebTracer({
  exporter: new ConsoleExporter(),
  plugins: [
    new DocumentLoad()
  ]
});

And I can easily see those span in console.
But the last step which I would like to try is to simply somehow send those spans to zipkin and see how they look like (similar to what is done for nodejs).
And now I wonder if I could simply use the zipkin and just replace the http with XMLHttpRequest, or at least try to send the spans from ConsoleExporter just using the correct transform or any other easy way.

[OlivierAlbertini]

@obecny
As a developer, I would prefer to use the same way than I did on node side:

tracer.addSpanProcessor(new SimpleSpanProcessor(exporter));

SimpleSpanProcessor will send directly
BatchSpanProcessor will use batch mechanism
another will use web worker

WDYT ?

[draffensperger]

Can you use a platform section in the Zipkin exporter to isolate its use of the http module, similar to how we do it for performance.now for the browser vs. process.hrTime for Node? It could use XMLHttpRequest for the browser to send the spans to zipkin.

In general, I like the idea of exporting spans via the OpenTelemetry Collector, or actually even writing mini-collector type code that people could embed in a server that would speak the same protocol as the collector so that they don't need to run the collector process if they just have a simple deployment setup of e.g. a Node server and a web frontend. See this issue for OpenCensus Web where someone was asking about this. The advantage of these approaches is that it could allow us to have some type of rate limiting or other security measure in place on the server side to help prevent overload of spans to an unauthenticated trace backend - for example, an application developer could only ingest spans from users logged into their app, etc..

That said, I totally agree that the ability to export to Zipkin, at a minimum for local development purposes, would be super useful - and to drive adoption we should give that option. Someone may even choose to use it in production and just live with the risk of overloaded Zipkin servers.

[mayurkale22]

Looks like this is mainly useful for local development purposes, does it makes sense to modify Zipkin Exporter only and keep Jaeger Exporter as it is or vice a versa ?

[obecny]

As a summary: there is already ConsoleSpanExporter which I extended a bit. @OlivierAlbertini the tracer is calling addSpanProcessor.
So the only thing to consider now is to have a new task to refactor zipkin only so that it can be used in browser too, using the platform.

[mayurkale22]

As discussed in the SIG, will add/implement agent exporter (similar to https://github.com/census-instrumentation/opencensus-web/tree/master/packages/opencensus-web-exporter-ocagent) to send data to OpenTelemetry Collector. At this moment, will not update the Zipkin/Jaeger exporter to support web use-case.

I think we should reconsider this in the futurre. Please comment if you disagree.

[obecny]

After investigation here what I found out during working on new @opentelemetry/exporter-web-collector (https://github.com/open-telemetry/opentelemetry-js/issues/499)
If I understand all correctly then so far there is no dedicated format for opentelemetry-collector which was my initial thought.
So the opentelemetry-collector supports these format: Jaeger, OpenCensus, Prometheus and Zipkin. I have also tried to simply integrate the @opentelemetry/exporter-web-collector with for example the LightStep collector which supports also the zipkin format. My conclusion is that the @opentelemetry/exporter-web-collector will be in fact nothing more then a Zipkin exporter that should simply use the navigator.sendBeacon or XMLHttpRequest as a fallback, when sending spans to any backend, opentelemetry-collector, LightStep collector etc.
This leads me to my initial conclusion to refactor the zipkin plugin using already the mentioned mechanism platform. Otherwise the @opentelemetry/exporter-web-collector will have most copy paste data from Zipkin or I will have to export most of the stuff from zipkin and use it in new @opentelemetry/exporter-web-collector package.

[mayurkale22]

AFAIK opentelemetry-collector expects data in the OpenCensus proto format, that's how it's been done in OpenCensus Node and Web. I think we should reuse the most of the code from OpenCensus web agent-exporter. OpenTelemetry format is still under flux but I don't expect too many changes from current proto (or OpenCensus proto).

[obecny]

Hmm, I thought this is just for internal, and not exposed yet from opentelemetry-collector
So should the request looks like
ExportTraceServiceRequest
where spans are are ReadableSpan converted into
https://github.com/census-instrumentation/opencensus-web/blob/master/packages/opencensus-web-exporter-ocagent/src/api-types.ts#L148 ?
And then this example can be treated as the way to go ? https://sourcegraph.com/github.com/census-instrumentation/opencensus-web@68aa988732e03f2de49ebc0213c3b2525aa5b68b/-/blob/packages/opencensus-web-exporter-ocagent/test/test-ocagent.ts#L112

[mayurkale22]

Yes, right and as mentioned earlier opentelemetry-collector proto might change in the future, but won't be drastic.

[draffensperger]

One other helpful link is the Swagger/Open API definitions that correspond to the OpenCensus agent proto format. These were automatically generated by a grpc-gateway protoc compiler plugin.