Kafkajs: List of companies using KafkaJS

[Nevon]

Hey KafkaJS users!

We sometimes get questions from people on whether or not KafkaJS is used in production. As the maintainers, we hear from people every now and then, but it's not documented anywhere. Knowing how KafkaJS is used and by whom helps us make decisions and also helps users that feel unsure if they can trust this library or not.

✋ If you or your company is using KafkaJS, please let us know in this thread! ✋

Once we have a couple of users listed, I will turn this into a PR adding the users section to the website.

Known users:

• Klarna - European fintech using KafkaJS across a few dozen services. @tulios & @Nevon

[kpala]

Nightingale Health - Finnish health tech company using KafkaJS in a laboratory information management system. @kpala

[pimpelsang]

Thinking about using in production pipedrive Tallinn, so far only in tests :)

[paambaati]

Indix uses KafkaJS to move massive amounts of data across various systems every day! We also open-sourced the kafkajs-lz4 module that we use in-house.

Indix is a data-as-a-service company that aims to be the Google Maps of product data.

[vykimo]

IBM France Watson
We use it to handle the pipeline of a cognitive platform. 🙂

[drojas]

Azlo is using KafkaJS to deliver data between micro-services and from micro-services to data lake

[julienvincent]

JourneyApps

We are now using KafkaJS in three areas:

• To power communication and data-replication between our many micro-services.
• To power GraphQL subscriptions for real-time communication between client interfaces
• To power real-time co-editing in our platform IDE

[StevenLangbroek]

We (Quandoo) are writing our first service with it now.

[yarncraft]

As a data engineer at Charp, I am using Kafka.js with Confluent to build an IoT streaming analytical layer.

[StevenLangbroek]

Oh wow totally forgot to update; yeah we've been running this in production for about 4-5 months now :).

Broadly speaking, we consume 3 topics into storage (Postgres & Redis) and allow users to read their data through a GraphQL API.

[ankon]

Good one, also should mention this here:

We (NEXT Technologies) are using KafkaJS to power our platform of about 40 microservices written in TypeScript (event sourcing). We use Kafka hosted in AWS (MSK).

[WaleedAshraf]

Hi everyone,

We at relayr are also planning to switch from Blizzard/node-rdkafka to kafka-node OR KafkaJS. We are looking for something which is production-ready. Blizzard/node-rdkafka has many issues with the consumer getting disconnected and weird error codes and no proper way to disconnect and shutdown the producer/consumer and much more.
We have 20+ Nodejs services that produce and consume messages. We also produce and consume in batch. We are looking for something which also provides some kind of back-pressure or at least the ability to implement it.

If anyone has already done research on pros/cons OR how KafkaJS is a better option, OR if you have faced any issues with kafka-node please share. That would be really helpful.

Thanks,

[kpala]

If anyone has already done research on pros/cons OR how KafkaJS is a better option, OR if you have faced any issues with kafka-node please share. That would be really helpful.

We had kafka-node in use for maybe a year or two since it was the recommended library for Node back then. Lots of connectivity issues and ultimately the only way we could handle these was restarting our services.

Moreover, kafka-node provided multiple options for creating producers and consumers and these options just kept piling up over time without a proper insight which of these we were supposed to be using (we started from HighLevelConsumer and ended up to ConsumerGroupStreams). Now and then I had to go read the source code to understand which options were available since this was not properly documented. This also made me realize the codebase was a mess.

When we were finally fed up with the connectivity issues, we started to migrate to node-rdkafka with equally disappointing results and since we didn't invest much in that migration, we decided to look for yet another library.

With kafkajs our system has been surprisingly stable and we've been happy so far.

However, it's also worth mentioning that during this time we've been improving our infrastructure and Kafka itself has improved a lot as well. Thus, it's not so black-and-white but I wouldn't go back to kafka-node nevertheless.

[WaleedAshraf]

That's really helpful @kpala. I also found some more info, like these benchmarks.
Once I'm done with research and comparison, I'll also share my findings here. 👍

[ThisIsMissEm]

Just to further on @StevenLangbroek's comment, at Quandoo we've also written a bit of a layer on top of kafkajs to help with managing subscriptions to topics, as we've internal standards for our kafka data.

There are still things that we wish were better documented or easier to understand (e.g., a recipes section, like, how do I calculate consumer lag, how do I know when my consumer is "ready" (kappa), and there's a few issues we had around consumer group names getting reused on different topics which leads to only one topic being subscribed)

But overall, kafkajs has been a pleasure to work with & the support from developers outstanding.

[WaleedAshraf]

@ThisIsMissEm for

issues we had around consumer group names getting reused on different topics which leads to only one topic being subscribed

Can you explain a bit more about the issue and how you fixed it?
In our case, we have one consumer-group per service which consumes from different topics. So, we may also run into the same problem if it only consumes from one topic.

[ThisIsMissEm]

@WaleedAshraf it was because we had multiple distinct kafkajs consumers, so called subscribe() multiple times with the same groupId, but different topics.

The solution was to use a unique group per topic as a stop-gap, and then write a consumer manager layer, so we could have different parts of code each connect into the kafka instance, but only one actually subscription.

Writing the I have just realised I could probably instantiate my consumer classes separately, and then pass them into a single "kafka subscriber" method or something (It's a slight inversion of control). I'll try writing it tomorrow maybe.

Basically if you have multiple kafka.consumer's then don't share the groupId; also, you'll notice it not working pretty quickly because you'll see it in the logs.

Which reminds me, I should open-source our small function to reformat kafkajs logs for pino.

[WaleedAshraf]

Thanks for sharing this. 👍👍 I'll test it out tomorrow. Probably we'll have same issue because we use 1 groupId for multiple topics when we subscribe to consumer.

I'm not sure if there's an issue/pr for this to fix within kafkaJS or why there is this limitation.

[ThisIsMissEm]

It's not really a kafkajs issue; it was me misusing the library.

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
})

const consumer1 = kafka.consumer({ groupId: 'test-group' })

await consumer1.connect()
await consumer1.subscribe({ topic: 'test-topic', fromBeginning: true })

const consumer2 = kafka.consumer({ groupId: 'test-group' })

await consumer2.connect()
await consumer2.subscribe({ topic: 'another-topic', fromBeginning: true })

Two kafka.consumers, both sharing the same groupId, but different subscriptions

[tulios]

@WaleedAshraf this is not a KafkaJS limitation, consumer groups can subscribe to multiple topics, but all consumers within the group have to subscribe to the same topics. The concept of a consumer group falls short if every consumer in the group has a different subscription.

If you need help with other questions or need help in general, please, create a specific issue or join out slack channel, let's not flood unrelated issues will all sorts of questions.

Thanks.

[DevNM21]

StudyMonk now uses KafkaJs to track various events and actions performed by the students in our platform

[cwnga2]

We (Oriente) are writing our first service with it now.
this library helps us reduce using callback effort :)
nice~~~