Skywalking: [Core] Add new concept, EVENT

[wu-sheng]

Hi SkyWalking team

I have been thinking about this concept for a long time, I am going to officially propose this concept now.
Right now, we have tracing and metrics fully supported, logging is on the way due to the browser provided and satellite is going to do more in this field. We have the typical 3 concepts, logging/tracing/metrics, in the observability.
But in the real world, it is more complex than those 3. EVENT is one of the most typical things.

System performance could be changed due to different variables, such as upgrading, rebooting, chaos testing, hosted VM overload. Some of them are detectable in some tools, such as rebooting in k8s, upgrading in CI/CD pipeline. But at the same time, they are only accessible through agent tech. We call this the event.

Event should include owner(affected instance/service/etc.) with time range(start/end). This would be easy to implement in OAP as actually, it is a new kind of record only. The major difference is, one event would include 2 records, typically, one for the start, the other for ending, linked by an UUID. Because an event could take a long time, 2 records could make the event started shows up ASAP.

• gRPC/HTTP interface should be provided to report events. CLI should follow this to give a command to report events too.
• GraphQL interface should be provided for UI to visualize the event.

As a special request to UI, @Fine0830

We need to add a timeline based on the selected time range, showing the event dot(if even hasn't stopped), or a bar(covering start to end timestamp). When the mouse clicks the dot or bar, all charts(if in the dashboard), should show a vertical line(or range) to highlight the metrics values at this special time points.

[kezhenxu94]

Who will be the owner of this feature? I’d like to undertake this if no one is working on it yet.

[wu-sheng]

We don't have yet.

[kezhenxu94]

We don't have yet.

OK. Let me kick it off by some further discussions (later)

[wu-sheng]

@BFergerson You pinged me about the similar thing, if you like, join the discussion here.

[kezhenxu94]

@wu-sheng @BFergerson I just drafted a simple doc about the event concept, please take a look at https://github.com/apache/skywalking/pull/6183

[kezhenxu94]

6183 implements the concept in terms of protocol, storage and analysis, https://github.com/apache/skywalking-cli/pull/88 implements the reporting CLI, my next plan would be building an event exporter targeting Kubernetes. As for the (Java) agent toolkit, I don't see much sense it makes so I'll just push it into the backlog until someone asks for that or is willing to contribute

[wu-sheng]

my next plan would be building an event exporter targeting Kubernetes

This is a good one for sure. I want to add another interesting things. We could report an booting/shutdown event in the javaagent code, by using service#boot and JVM close hook. What do you think?

As for the (Java) agent toolkit, I don't see much sense it makes so I'll just push it into the backlog until someone asks for that or is willing to contribute

Agree. Don't have idea who will need that at the codes level.

About more use cases, we could consider an event called JVM arguments changed, such as GC, Xmx. We could make CLI + shell + local file cache to determine the change, or use the instance name to find the previous arguments at the OAP side.

[kezhenxu94]

I want to add another interesting things. We could report an booting/shutdown event in the javaagent code, by using service#boot and JVM close hook. What do you think?

Great idea!

About more use cases, we could consider an event called JVM arguments changed, such as GC, Xmx. We could make CLI + shell + local file cache to determine the change, or use the instance name to find the previous arguments at the OAP side.

I think both have their limitations, using CLI requires the users to bind the CLI tool to their (CD) workflows manually. If we do this in OAP side, the instance names may be randomly generated, making every instance independent.

[wu-sheng]

I think both have their limitations

All event detection will have limitation. Such as JVM close hook may not be executed, if user doesn't shut down the service elegantly.

If we do this in OAP side, the instance names may be randomly generated, making every instance independent.

Definitely yes. My point is once we detected this, we generate an event, if this is a new instance, we don't need this. The idea behind this proposal is, we need an OAP core-level API for change detection for contributors to enhance.

[kezhenxu94]

To summarize, to todo list includes:

• [x] UI implementations https://github.com/apache/skywalking-rocketbot-ui/pull/452
• [x] Agent booting / shutdown events
• [x] Kubernetes event exporter (tracking this task here http://github.com/apache/skywalking-kubernetes-event-exporter)
• [x] OAP internal report mechanism
• [ ] Agent start up JAVA_OPTS changes events detection

(Feel free to add more to the list if you have any)

[wu-sheng]

OAP internal report mechanism

@kezhenxu94 Do we have this? We have internal event API?

[kezhenxu94]

OAP internal report mechanism

@kezhenxu94 Do we have this? We have internal event API?

We have shared event-analyzer module, so I think moduleManager.find(EventAnalyzerModule.NAME).provider().getService(EventAnalyzerService.class).analyze(Event.newBuilder().setXXX().build()) is enough for internal use, do you think we should provide dedicated APIs for this?

[wu-sheng]

I am not sure whether this is enough, let's wait and see. This API will be used in the alarm module.