Stream collector docker image customise config file

1

I’ve a product with 5 markets(different location) and each market specific configurations is stored in different config files. Like a_market…hocon, b_market…hocon and so forth.
I want to deploy Stream collector snowplow/scala-stream-collector-kafka:2.4.0-rc1 image. and want to pass each market specific config file as a

option 1 - pass as parameter
option 2 - Embed all the config values from config to inside the docker image and then deploy it.

could anyone please help

2

Hey @pramod.niralakeri the easiest way would be to mount a volume with the config you want for each container - you could create custom Docker containers for each application but its a lot of overhead just to pass in a config.

Some relevant documentation if you are doing this with Kubernetes: Configure a Pod to Use a ConfigMap | Kubernetes

3

Hi @pramod.niralakeri,

It if helps, the latest collector release lets you also pass parameters using java system properties. So you can do something like:

docker run \
  snowplow/scala-stream-collector-kafka:2.4.1 \
  -Dcollector.interface=0.0.0.0 \
  -Dcollector.port=80 \
  -Dcollector.streams.good=goodstream \
  -Dcollector.streams.bad=badstream \
  -Dcollector.streams.sink.brokers=localhost:9092

You can even use a mixture of hocon files and system properties. System properties take precedence over the hocon. This new feature of using system properties was only introduced in version 2.4.1.

So you might be able to have a shared common hocon file, and then override specific values for your different markets.

1 Like
4

I’ve couple of questions

  1. System properties is ENV variables I believe right?
  2. if ENV variables, what are env variables I need to set, where do I get?
5

Hi @pramod.niralakeri, by system properties I meant the using the -D command line flag. This is a standard flag for setting what in java applications are called system properties.

So in my example in my previous comment, -Dcollector.port=80 means set the collector.port system property to 80.

Alternatively, there is a way of using environment variables instead of system properties:

docker run \
  -e "CONFIG_FORCE_collector_interface=0.0.0.0" \
  -e "CONFIG_FORCE_collector_port=80" \
  -e "CONFIG_FORCE_collector_streams_good=goodstream" \
  -e "CONFIG_FORCE_collector_streams_bad=badstream" \
  -e "CONFIG_FORCE_collector_streams_sink_brokers=localhost:9092" \
  snowplow/scala-stream-collector-kafka:2.4.1 \
  -Dconfig.override_with_env_vars=true

The syntax is slightly awkward, but it is standard behaviour described in the readme of this config-parsing library. Note it still requires the -Dconfig.override_with_env_vars=true to make it work. Personally I prefer the syntax in my earlier post using just system properties.

1 Like