Run Kinesis and Kafka at the same time in collector?

For engineers Kafka real-time pipeline
1

Hi all

We are facing a problem when we have configured a collector/enrich form the batch pipeline, but figure out if we can use the same collector to start a real-time pipeline: that’s mean we need to send the events to Kinesis (batch) and to Kafka (real-time).

Any have done these?

Can we use in the config file this syntax??

sink {
…
enabled = “kinesis,kafka”

kinesis {
  ...
}

kafka {
  ...
}

}

Best to all

2

Hi @spatialy ,

Looking at the collector code, it seems that’s not possible: you can either use the Kinesis or the Kafka sinks.

In any case, having the collector writing to two different sinks could lead to having data discrepancies down the line as one of the message brokers could be down. It would be much safer to have the collector to write just to one sink and then have a small app that reads from Kinesis and writes to Kafka (or the other way around).

Hope that helps!

1 Like
3

Hi @danisola you are rigth, in the code only one option are allowed.

Let me try and go back after and left the feedback here.

To try:

  • Implement the RT collector and move data between RT Kafka sink and BATCH Kinesis sink. ( @danisola recommendation)
  • Implement dual tracker in the web. (to see performance of feeding two independent pipelines)