Kafkajs: Improving stable release process

[Nevon]

Caveat

At this point, I am only seeking discussion regarding this. It is not decided that we will move forward with this. These are just some of my personal thoughts on how to solve a problem, and I have not run them past other maintainers or community members before now. If there are concerns, please let us know in the calm and respectful manner that I would expect from this community.

Problem

Making stable releases is difficult right now. In the past, @tulios and I were working with Kafka on a regular basis, and we both had access to dozens of teams that were all using KafkaJS in various ways. When we wanted to make a release, we could simply move one or a few of our services to using our beta versions (or branches, before we had beta releases) for some time and then feel quite confident about making a stable release.

Since then, myself and @JaapRood have switched jobs to companies where we don't currently use KafkaJS, and @tulios has switched positions to where he is no longer able to devote as much time to coding. In other words, we no longer have the same easy access to running beta builds in production or production-like environments.

In an ideal world, any build that comes out of the CI pipeline green should be good enough for a stable release - and in _most_ cases it is - but experience has shown us that sometimes there are more subtle issues that only spring up while under load or under sustained use. Testing for those kinds of issues require long-running infrastructure that we can't afford ourselves.

Possible solution

Members of our community are already running the beta releases because they want early access to fixes and features. However, the feedback from their use doesn't actually reach back to us. If there's a catastrophic bug, we might hear about it through an issue being created, but most of the time we don't hear anything, and we can't know if that's because there's nothing wrong, because no one is running a particular beta release, or because people just haven't contacted us.

A possible solution to this would be to introduce telemetry from the beta builds.

🔬 Telemetry

If we could get some insight into how our beta releases are performing in the wild, we could use that data to find issues early and to gain confidence in making a release, allowing us to make releases more frequently and with better stability.

What I think would be most useful to us are error reports. If we could get access to error reports from our users, that would be incredible helpful. Sentry has an open-source sponsorship program that could probably make this feasible from an economical standpoint.

However, there are definitely some challenges:

🤫 Privacy

KafkaJS handles a lot of sensitive data, and no company would want their data to be sent to a third-party that they don't control - nor do we actually want that data.

To mitigate this, we could do a few things:

1. Make the telemetry opt-in. Add an environment variable like KAFKAJS_ERROR_REPORTING that needs to be set to enable the error reporting. This would drastically reduce the amount of data we'd receive, of course, since most people would probably not opt-in, but this seems like a no-brainer to me regardless.
2. Never send buffers. While it would be super useful to get for example fetch requests or responses for certain kinds of errors, the risk of including sensitive data is too big, so we should never send those. Just knowing that _there is_ some parsing error in a certain request version, even if we don't know exactly in which scenario, is better than nothing.
3. Be conservative in what errors to send. Introspect the error stack to be sure that the error is actually coming from our code, and not from something inside the user's eachMessage function or similar. This would also help us reduce false-positives, so it's a win-win, even if it requires some work.

🔨 Build system

While we _could_ condition the error reporting at runtime, same as with the opt-in flag, I would feel better if the code wasn't even there for stable releases. This requires us to have some kind of build system, which I've been happy not to have until now.

We don't _necessarily_ need a whole mess of Webpack or Parcel or whatever - it could be as simple as a bash script that seds out a require and rms a directory, but we would need _something_. Another drawback of this is that our beta release and stable release would no longer be _the exact same_, so the data we got from testing the beta release is no longer 100% valid for the stable release.

😰 Engineering effort and maintenance

While this would help us, it also has a cost. If we suddenly get an influx of error reports, who's going to go through those? If it's a small number, then perhaps it's manageable, but if it's a large number the issues might get lost in noise, rendering the whole thing meaningless (turning it from an asset into a liability).

Making this a reality is a non-trivial amount of work, and we're already spread thin as it is, so I would only want to go through with this is if actually ended up helping us in the end.

😡 Prior art and controversy

Last but not least, I don't know how people would react to this. Even if we make it opt-in and try to make sure that we don't leak any information from the applications, I haven't found any prior art from this, so I don't know how people would react. In end-user applications it's very common, and for SaaS no one seems to bat an eye that they are tracking anything that can be tracked, but for open-source libraries the rules can be different.

Next steps

I would like to get input on and discussion about:

1. Are there examples of similar open-source libraries doing this? By that I mean kind of low-level components like database drivers, networking libraries, etc.
2. If you are currently running the beta releases, would you consider enabling error reporting? If no, would there be some concerns that we could alleviate, or are they intrinsic to the error reporting itself?
3. Do you agree with my assessment that this would help us have confidence in our releases? (Assuming we did roll this out and got some non-negligible number of developers using it) If not, why not?
4. Do you have other ideas regarding how we could solve the original problem?

[JaapRood]

Thanks for getting the discussion started on this @Nevon, you obviously put a bunch of time and thought behind this.

At my new job we're starting to use _some_ KafkaJS, but nothing has made it into production yet. That might change, after which it could become easier for me to run things. Would have to discuss with my team, of course!

The idea of gathering some kind of telemetry from KafkaJS deployments I think is a pretty good idea, and not something I would be against, given enough confidence and control over what I'm sharing. What helps, of course, is knowing about the goal for gathering these analytics, knowing who has / had access to them, etc. I think the answer to all of those is basically the most minimal thing that could be done.

What I'm wondering is whether _error reporting_ is the type of telemetry that makes for a good signal, and fits the privacy constraints we'd be after. As you've noted, even just the stacks themselves can be full of sensitive information. Even if we'd manage to sanitise those, the question becomes how we can interpret them in a meaningful way: when is the error inherent to Kafka, when is it because of user-error? I can imagine betas being adopted by those for whom the stable versions caused an issue, which makes me wonder about the stability of that in the first place.

Rather, I wonder if usage metrics are actually the thing we're after: aggregated stats on key metrics a healthy KafkaJS would exhibit. E.g. successful / failed connections, successful / failed requests, type of error rates, throughput rates, etc. If per project we'd had some anonymous identifier, we can compare numbers between versions. The relative comparison between these could hopefully give us some kind of useful signal.

Since these aggregated metrics shouldn't really be privacy sensitive, the question of who has access to it and how is it used could be answered by making it all public, whether that's the raw data being available, or even something as fancy as a web UI. Might even be a good source for considering whether it's safe to run a beta-build in the first place.

Obviously, the issue with that instead of just leveraging something like Sentry is that it would require the building of a backend. For gathering the metrics, I'm familiar with deploying T-digest as way to create histograms / distributions in a memory-fixed way. A service + cluster running KakaJS would seem like _the_ way to accept the results of this, which would also instantly give a first target to run the telemetry enabled and perhaps even make for a good example project. Where and from what resources to run that, is another matter, though. Also helps with the full transparency of where the telemetry is going.

[terebentina]

I'm working with kafka and kafkajs and we're running a handful of services all revolving around real time streaming and etl/alarms/alerts based on the data from kakfa. We'd probably benefit the most from a kafka streams nodejs solution (hey Jaap :) ).
I am willing to help if we can figure out a good way. Just let me know. Alternatively I am sure it'd be ok on our side to get one of you involved in a small project where you get access to our kafka and we get access to your expertise.

Regarding using beta releases in general (not related to just kafkajs), my rule of thumb is that it's marked as beta for a reason: the maintainers have reasons for not calling it production ready so I tend to stay away from them and just use the latest stable. The only exception I can make from this rule is when starting a new service from scratch and nothing/nobody is relying on it yet. Then we can play with it, gather insights, scrap it and start from scratch, etc.
Luckily we're at a phase where we build such new services quite often so tell me what you need.

Regarding the automated telemetry I'm against sending actual data to you because the data we're processing doesn't belong to us but to our customers.
But I have no problem with sending logs, statistics, kpis or stack traces.
One thing that you must consider about this, though, is that not all kafkajs-based services will have internet access. Ours certainly don't.
So, in our case, if I wanted to send you some logs, they'd have to be written to a file, taken out of the service and then delivered to you somehow.
Which leads me to the next idea: would using the logger and a "special" log level for this be feasible? Could I just set the log level to logLevel.TELEMETRY and, maybe, logDestination: '/path/to/file.txt'? While not as simple as automated data submission, it would let me see what I am sending to you and I could control what/if/when to send. Certainly not for everybody though.
Maybe the cherry on top would be to listen to a unix signal and attempt sending the file then.

Apart from these, a few quick random thoughts:

• I think there should be no difference between stable/beta so whatever telemetry options you choose, let them be there in stables too.
• Automated telemetry raise another concern: what if that code is buggy? What if the server it tries sending it to is down. Or what if it's up but it takes a long time to answer - will it slow my service down?
• Proper telemetry is a job in itself. You might make it too verbose or not enough. Send it too often or not enough. That's why manual submission seems to be the safest bet to start with.

Sorry if I got too carried away...must've been something in this grape juice ;)

[goriunov]

@Nevon personally i am against of telemetry. Also it would be really hard to manage as there many things which could go wrong and confuse you (as those issues could be in users app and not kafka js library itself)

IMHO it is better to implement constantly running environment with some dummy data going through and probably automated chaos logic with different conditions which will test the builds and could report all issues. You could set up continues deployments of every beta release in that env and monitor it for few days/weeks before going to stable. This will make sure that all issue will be isolated to kafka js and not users code.

[Nevon]

@goriunov:

IMHO it is better to implement constantly running environment with some dummy data going through and probably automated chaos logic with different conditions which will test the builds and could report all issues.

I am actually looking into this in parallel. This is something we've wanted to set up for a long, long time, but the issue has always been that Kafka clusters are freaking expensive to run, even rather modest ones. The original intention was to set one up on DO using the credits we get as part of our sponsorship, but the amount of credits we received is not enough to actually cover the costs, especially if we also include the fact that we need to run the actual test application as well. Additionally, as we've all become busier, time is now also at a premium, so I really don't want to have to maintain a Kafka cluster if it can be avoided.

Nowadays, there is the option of managed Kafka clusters, so this is an avenue I'm pursuing at the moment, but the viability of it depends on whether or not any providers of these clusters are willing to sponsor us with one. I hope to have some news regarding this over the next few days or weeks.

If we managed to have this, I would consider user-submitted telemetry to be much less important. There are still some advantages to it, since it represents a much greater variety in usage and environments, but since it comes with rather hefty downsides as well, I would say it becomes much less worth it.

@JaapRood:

Rather, I wonder if usage metrics are actually the thing we're after: aggregated stats on key metrics a healthy KafkaJS would exhibit.

I thought of this as well, and while I'm not 100% sure, my current line of thinking is that the kind of aggregate metrics that you would use for monitoring a Kafka application are not that useful for finding regressions in KafkaJS itself. The reason being that metrics tend to tell you the _overall_ health of a system, so while it could be useful for detecting for example performance regressions, it wouldn't be very useful for detecting a regression where empty transactions committed against MSK clusters fail because of a TypeError, because that would be such a small part of the overall data that it wouldn't appreciably affect the metrics.

If I'm monitoring my system, I'm absolutely more interested in aggregate metrics - who cares about an occasional TypeError if I'm still consuming at the rate I need to - but for catching rare outliers I think I'm more interested in exception tracking.

Ideally I would have both, of course, but if I have to choose, I think the latter is more beneficial for this particular use-case.

@terebentina:

Regarding the automated telemetry I'm against sending actual data to you because the data we're processing doesn't belong to us but to our customers.

Of course. This would never be an option, nor really anything we would be interested in.

One thing that you must consider about this, though, is that not all kafkajs-based services will > have internet access. Ours certainly don't.
So, in our case, if I wanted to send you some logs, they'd have to be written to a file, taken out of the service and then delivered to you somehow.

I would say we basically already have this as an option. It would just be the regular errors logs, which you can already send to us by opening up a Github issue.

Automated telemetry raise another concern: what if that code is buggy? What if the server it tries sending it to is down. Or what if it's up but it takes a long time to answer - will it slow my service down?

To be fair, all the code we write _could_ be buggy, so automated telemetry or not, you already run that risk. 😄Of course if we were to implement something like this, it would have to go through the same level of review and testing as any other code we write, to make sure that it doesn't impact your service even in failure scenarios.

[JaapRood]

The reason being that metrics tend to tell you the overall health of a system

I guess we're talking about two different things then. In my mind, whether to publish a beta version to _latest_ has to do with overall health and stability of the system, not whether a bug has been fixed or not. Some of the bugs we tried to fix in PRs for any particular version might still be present despite efforts to fix them, but if there are any other enhancements and the system is at least as stable as it was before, it's an improvement and worth shipping.

That said, to hunt down a particular bug, I could imagine adding very specific metrics, measuring the occurrence of very special conditions, similarly to how one would use console.assert.

If what you're talking is basically distributed regression testing, I think we'll bite off more than we can chew.

[Nevon]

I guess we're talking about two different things then.

I think so. There's definitely merit to the argument that if the new release is working well enough for most people that there's no significant impact on the aggregate metrics, it's good enough to be released.

However, there's another way you can look at it as well, which is to not look at the aggregate but rather at the impact for the outliers (similar to how you measure latency not by the average, but by the worst X percentiles).

That said, I'm open to the idea if you think it would be more useful to us. I _would_ love to have a nice dashboard telling me how KafkaJS behaves in the wild. I'm just worried that the amount of data would be unmanageable and hard to take action on.

If what you're talking is basically distributed regression testing, I think we'll bite off more than we can chew.

Yes, I think that's more what I'm talking about. Not catching regressions to known issues, as those we tend to cover with tests and I'm not too worried about them reappearing without us noticing, but rather introducing new ones, such as the unhandled promise rejection we introduced in certain situations with the new fetch mechanism. Now, you might say that such an issue would affect the metrics, and so we would know that the beta release is not good for general release, but I fail to see how we would be able to take action on it when we don't know the cause.

What makes you say that we would be biting off more than we can chew? I certainly see that there are challenges, but they don't seem insurmountable to me (without having tried yet, so I'm just speculating here). Filter out anything that doesn't originate within KafkaJS, filter out handled errors (any KafkaJS* error, except the Retry error, which would need some special handling), anonymize the stack trace, ship it to Sentry tagged with the current release version. Errors will already be grouped so that we can see which ones are appearing frequently, along with the ability to silence recurring false positives that our filtering missed.

Now, we could get a lot of this benefit more easily by running our own Kafka consumer/producer, which is still plan A in my book. We wouldn't get the diversity we get from leveraging our community, but the complexity is also much lower.

[JaapRood]

What makes you say that we would be biting off more than we can chew?

There's two bits two part to this:

1. In my new job we're dealing with capturing similar information from customer deployments, and doing so that really strips all sensitive meaning, while still keeping anything that is actionable is hard.
2. Even when implementing 1 correctly, getting people to build enough to trust it to actually run it in a place where we would get high quality captures of data that we're after. People are generally fine running it locally in development environments, but that's exactly where you get very low signal-to-noise.

I found it takes the user being able to inspect generated logs that can then manually forwarded is how you get this kind of data, evidenced by @terebentina's response.

Now, we could get a lot of this benefit more easily by running our own Kafka consumer/producer, which is still plan A in my book. We wouldn't get the diversity we get from leveraging our community, but the complexity is also much lower.

[...] I'm just worried that the amount of data would be unmanageable and hard to take action on.

Fair enough, too! I reckon constructs like t-digest can definitely help in aggregating heaps of data in a relatively fixed amount of space, and report rates should give reasonable control as well. That said, we couldn't accept endless amount of data, but probably also don't have to.

Now, we could get a lot of this benefit more easily by running our own Kafka consumer/producer, which is still plan A in my book. We wouldn't get the diversity we get from leveraging our community, but the complexity is also much lower.

That's where I think metrics aggregation is a nice compromise.

• Generating useful health stats is useful to any consumer anyway, whether it's forwarded to some global aggregator or not
• Aggregating these stats makes for a nice task for the Kafka consumer / producer, and a nice open-source example of something serious done with KafkaJS.
• It itself becomes a source of the telemetry, so even if very few others opt-in, there'd be something to work of. (Could generate some funky behaviours? At least a strong signal if something's not right).

[ankon]

From our point:

1. We do run beta releases (not just of KafkaJS) of dependencies in production, _iff_ we feel that the benefits greatly outweigh the risks. One of the reasons why we feel relatively comfortable with KafkaJS in that regard is that we feel like we could handle fixing issues ourselves if needed. We definitely _prefer_ to run official releases though.
2. In terms of collecting metrics: This might be feasible for our test environment(s), but I do feel icky about that for the production environments -- not just for reasons of accidentally getting data sent to the wrong place, but also for the pure practicality that @terebentina mentioned: Outbound access to the internet is not something I want for my services. Having some metrics stored in a (configurable!) known place that I can collect and provide on demand for an issue would be fine though I think.

Are there examples of similar open-source libraries doing this? By that I mean kind of low-level components like database drivers, networking libraries, etc.

The single one on top of my head: Vaadin, a collection of components for frontend development. See https://github.com/vaadin/vaadin-usage-statistics. And, FWIW and as a disclaimer: We're using these components, but have disabled the collection of statistics in our production builds as well.

[terebentina]

Upon more reflection, I have some more ideas to bounce off you:

• telemetry from kafkajs users might not be the right thing to use to help you progress a beta/rc to stable. The infinite number of use cases and high variability of environments might be more of a burden than a help. You have no control over anything and you will get error reports for reasons that don't depend on kafkajs. More even, once you fix something, it'll take time for users to upgrade to newer betas which will result in an upgrade lag and reports for already fixed issues (granted, a tool like sentry can help you organise these reports).
• so, maybe a kafka cluster loaded with a lot of real-world data would be better? A persistent kafka and enough data for tests to crunch it in hours/days. This would keep the environment conditions fixed which would make it easier for you to find regressions/mem leaks/slowdowns/etc.
• in which case the "problem" becomes just one of paying for a long running infra so...money essentially. Could you set up github donations (or patreon/whatever) for the project? (no idea what this requires so not sure if it's feasible or not). You could use this to pay for a cloud kafka instance.
• once you "officially" advertise kafkajs as "looking for sponsorship" you might even find cloud kafka providers willing to offer this for free.
• I've spoken with the management at my company and they're not opposed to sponsoring kafkajs (hence the GH donations idea).

Thoughts?

[Nevon]

• so, maybe a kafka cluster loaded with a lot of real-world data would be better? A persistent kafka and enough data for tests to crunch it in hours/days. This would keep the environment conditions fixed which would make it easier for you to find regressions/mem leaks/slowdowns/etc.
• in which case the "problem" becomes just one of paying for a long running infra so...money essentially. Could you set up github donations (or patreon/whatever) for the project? (no idea what this requires so not sure if it's feasible or not). You could use this to pay for a cloud kafka instance.
• once you "officially" advertise kafkajs as "looking for sponsorship" you might even find cloud kafka providers willing to offer this for free.
• I've spoken with the management at my company and they're not opposed to sponsoring kafkajs (hence the GH donations idea).

Thank you very much for the consideration. We haven't talked about having actual sponsorship via OpenCollective or the likes, but my personal thoughts are not that positive. However, if they were to go specifically to running costs like this, I am theoretically open to it as a possibility, although I foresee overhead that I would prefer not to have to deal with (reporting, taxes, etc.). Cloud Kafka instances also aren't that cheap, so it also depends on how much companies are willing to chip in.

However, regarding direct sponsorship from cloud Kafka providers, I am currently in talks about that. I agree with you that having a real-world running application would give us the best bang for our buck.

[WaleedAshraf]

We are about to start using KafkaJS in production. I don't think we'll be able to provide logs, but providing metrics are a possibility for development and staging environments. That also exported on-demand or time to time.

I'm thinking from a perspective that how can we make it easier for users to report errors here. How can we make it that if a user sees some issue, the first thing that comes to their mind is to open an issue on Github or Slack instead of adding some patches for their use case or just ignoring it all.

Do we know of any case, where we think that an issue was very clear but no one ever reported it?

Maybe we just need more user base and this problem will go away. "More the users, more testing". It sounds odd, but yeah, it will increase the chance of issues being reported. That means we can look for some marketing/tech talks/conferences/blog posts, etc.

[Nevon]

I have some great news! Thanks to Ale Murray at Confluent, we now have a sponsorship that will allow us to run a small testing cluster for one year on Confluent Cloud!

So now it's time to get to work actually building something to use it! 😄

[goriunov]

That is really great ! i think it would be really good to have separate apps with different conditions.

One of those could be an app which will be running consumer and producer with script to block kafka connection like firewall block on specific port, also could do dns resolution block, important to keep them at different rates, which could be really good for chaos testing and system recovery checks

The other app could be usage of all functions/features such as consume, produce, admin probably good to have latency metrics on those to see if we have any degradation

We can consistently output debug logs to files and keep them for a week/few days or so, and only report actual unexpected errors, lost of connections, no messages etc to emails

If above things run consistently we can easily catch most of the issues before they go to production.

[Nevon]

@tulios and I just finished setting up the pipeline for a Node app that gets automatically deployed with the latest beta version of KafkaJS whenever it becomes available. Tomorrow I'm going to add more instrumentation to it, so that we can have alarms to tell us if something is wrong with a new version. At the moment it's just alerting on basic things like CPU and memory:

If anyone is curious, the project is here: https://github.com/kafkajs/canary-app

At the moment the cost of this is covered by Amazon (AWS credits) and Confluent (Confluent Cloud credits). Hopefully they'll keep sponsoring us, since it's on my account 😅

With this in place, I think we'll have a much easier time releasing KafkaJS, although we will still rely on the community to run the beta releases and to let us know if anything is wrong, since we all have different environments and use KafkaJS differently.

---

Regarding 1.13.0 - we're aiming to release it tomorrow, once we've had a chance to run it for 24 hours or so.