We have released Kafka support!!
This is just the beginning of the journey and we’re looking forward to working with the community to build this out.
8 Likes
Thank you so much. Am having some trouble when I tried to set this up.
I installed the required version of Kafka, set up the config as mentioned. But the collector is not starting. When I try to open the consumer to see the collector payload, I see only empty terminal.
Below is the last few lines of the collector debug messages of producer starting. Can you please help?
15:27:09.326 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bufferpool-wait-time
15:27:09.327 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name buffer-exhausted-records
15:27:09.327 [main] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 1 to Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: null)], partitions = [])
15:27:09.327 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name connections-closed:
15:27:09.327 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name connections-created:
15:27:09.328 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bytes-sent-received:
15:27:09.328 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bytes-sent:
15:27:09.328 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bytes-received:
15:27:09.329 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name select-time:
15:27:09.329 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name io-time:
15:27:09.329 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name batch-size
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name compression-rate
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name queue-time
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name request-time
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name produce-throttle-time
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name records-per-request
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name record-retries
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name errors
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name record-size-max
15:27:09.332 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka version : 0.10.1.0
15:27:09.332 [kafka-producer-network-thread | producer-2] DEBUG o.a.k.c.producer.internals.Sender - Starting Kafka producer I/O thread.
15:27:09.332 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka commitId : 3402a74efb23d1d4
15:27:09.332 [main] DEBUG o.a.k.clients.producer.KafkaProducer - Kafka producer started
[DEBUG] [09/26/2017 15:27:09.940] [scala-stream-collector-akka.actor.default-dispatcher-5] [akka://scala-stream-collector/user/IO-HTTP/listener-0] Binding to /0.0.0.0:8083
[DEBUG] [09/26/2017 15:27:09.972] [scala-stream-collector-akka.actor.default-dispatcher-2] [akka://scala-stream-collector/system/IO-TCP/selectors/$a/0] Successfully bound to /0:0:0:0:0:0:0:0:8083
[INFO] [09/26/2017 15:27:09.975] [scala-stream-collector-akka.actor.default-dispatcher-2] [akka://scala-stream-collector/user/IO-HTTP/listener-0] Bound to /0.0.0.0:8083
Hi @manju,
For us to be able to help you here we are going to need a lot more context than just the output from the collector (which appears to be operating correctly and without errors).
Please provide:
- The collector configuration file
- How you setup Kafka and the associated topics?
- How you are actually checking for events downstream of the collector?
- What things you have tried to resolve the issue?
Cheers!
Josh