Set all config variables into enrironment

1

I wanted to bring up snowplow/scala-stream-collector-kafka:2.4.1 container first (without passing config file) and then set all things from config file into environment variable.

Is that possible?
if yes, what are equivalent environment variable needs to set in order to in corporate config file set up.

2

Hi @pramod.niralakeri can you explain any differently what you mean by “bringing up the container first”? We already discussed in this thread how it’s possible to configure the kafka collector using environment variables.

The answer I gave before demonstrated the minimum set of required environment variables, but it is also possible to set every other configuration option using the same method.

Please can you give an example of what you are trying to do that is not covered by that previous answer?

3

I want to setup every config variables using config file or environment variable. except kafka broker list, which I’ll pass during docker run command.

So essentially I want to setup all variables in 2 different way.

  1. while building image all config file variables except kafka broker.
  2. while running image send only kafka broker
4

Hi @pramod.niralakeri, OK I understand now you want to build your own version of the docker image that has most of the paramters pre-configured in the image. Generally we recommend instead using the standard docker image, and passing all configuration at runtime. Nevertheless, it is possible to do what you’re asking.

You could prepare a file config.hocon, in which you set every required parameter, except you use special syntax for the brokers so it can be set from an environment variable:

collector {
  interface = "0.0.0.0"
  port = 80
  
  streams {
    good = "my-good-stream"
    bad = "my-bad-stream"

    sink {
      brokers = ${KAFKA_BROKERS}
    }
  }
}

Then bake it into a docker image using a dockerfile like this

FROM snowplow/scala-stream-collector-kafka:2.4.1
COPY config.hocon /snowplow/config.hocon
CMD ["--config", "/snowplow/config.hocon"]

And then build it and run it like this, setting your environment variable:

docker build -t my-custom-collector .
docker run -e KAFKA_BROKERS=localhost:9092 my-custom-collector
2 Likes
5

fantastic, this is exactly what I was looking for.

thank you so much.