Druid: Druid-Kafka-Indexing-Service Running Time

[bahadircolak44]

Hello I want to ask you a quick question about druid-kafka. I am using kafka-indexing-service and I think it works too slow. I am always sending data to kafka(using logstash) but druid takes that data every 10-12 minutes. It doesn't matter how big data is. For example, I sent a 1.5 mb file to kafka and it took 11 mins. Then I shut it down and started using logstash and sent data continuously to kafka. After 10-12 mins, druid took all data. After 10-12 mins, it took new data. I mean every 10-12 mins, it is looking for new data. I am not sure if I explained it clearly. So I am wondering:

• Is it normal?
• How can I make it faster?
• Is it possible to make it real-time? (If there is a data in Kafka, take it immediately)*
  I look forward to hearing from you. Thank you for your time.

[RestfulBlue]

you doing something completely wrong.
Do you send queries to broker or historical?
Is there something in the logs?
Which version of druid do you use?

[bahadircolak44]

Neither. I am using coordinator. Like:

curl -XPOST -H'Content-Type: application/json' -d @quickstart/schema.json http://localhost:8081/druid/indexer/v1/supervisor
But I am using docker this is why I sent 8081. Otherwise, I am using port 8090. But both of them work like this I mean really slow

I checked logs I saw nothing different.
My druid version is 0.12.1

[xbaran]

What are u looking for is there http://druid.io/docs/latest/development/extensions-core/kafka-ingestion

intermediatePersistPeriod | ISO8601 Period | The period that determines the rate at which intermediate persists occur. | no (default == PT10M)

Change it to PT10S and u will see ur date updated more often. Every 10 sec ;-)

[stale[bot]]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[stale[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.