Cannot send data to Kafka sink from collector

For engineers Collectors
1

I’m trying to send the collector data to a Kafka topic, here is my sink config for Kafka, I have tested the connection and it works on that port, but when sending data to my collector(docker image is: snowplow/scala-stream-collector-kafka:2.10.0) I get the following error:
ERROR com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KafkaSink - Sending event failed: Topic good_sink not present in metadata after 60000 ms.

# Enable the chosen sink by uncommenting the appropriate configuration
    sink {
      # Choose between kinesis, google-pub-sub, kafka, nsq, or stdout.
      # To use stdout, comment or remove everything in the "collector.streams.sink" section except
      # "enabled" which should be set to "stdout".
      enabled = kafka
  

      brokers = "b-1.poc2.prmupj.c2.kafka.eu-west-1.amazonaws.com:9098"
      ## Number of retries to perform before giving up on sending a record
      retries = 0
      # The kafka producer has a variety of possible configuration options defined at
      # https://kafka.apache.org/documentation/#producerconfigs
      # Some values are set to other values from this config by default:
      #"bootstrap.servers" -> brokers
      #retries             -> retries
      #"buffer.memory"     -> buffer.byteLimit
      #"linger.ms"         -> buffer.timeLimit
      producerConf {
        acks = all
      #  "key.serializer"     = "org.apache.kafka.common.serialization.StringSerializer"
      #  "value.serializer"   = "org.apache.kafka.common.serialization.ByteArraySerializer"
      }
    }

Do I need to create the topic on the kafka cluster myself or will the collector automatically do it ?

Thanks for the help!

2

Hi @Alexandre5602 ,

The collector expects the topics to be created beforehand, so you’ll have to create them.