Scala Stream Collectors - Config File Example

For engineers
1

Hi,

I’m new to Snowplow. I’m current trying to implement the Scala Stream Collectors on AWS to connect to Kinesis. I currently just follow the instruction here: https://github.com/snowplow/snowplow/wiki/Configure-the-Scala-Stream-Collector

Just wondering, is there an example with dummy actual value version of the config.hocon.sample file for Kinesis implementation? I think the instruction/comment on the config.hocon.sample is very brief.

I was searching for this topic in this discourse page, but couldn’t find one. But if there is already one, please feel free to close this topic, and redirect me to it.

Sorry for this newbie post. I’m very interested in implementing Snowplow for Web Analytics solution. Thank you in advanced.

Regards,

R

2

Have you tried https://github.com/snowplow/snowplow/blob/master/2-collectors/scala-stream-collector/examples/config.hocon.sample ?

1 Like
3

Yes, that is the example I’m using. My mistake, I forgot to replace some of the {{}} values. It’s up and running now. Thank you.

4

Hello,

I am also looking into Snowplow streaming however I’m stuck on this command:

docker run --rm
-v $PWD/config.hocon:/snowplow/config.hocon
-p 8080:8080
snowplow/scala-stream-collector-kinesis:2.6.1 --config /snowplow/config.hocon

I get errors such as
Key not found: ‘interface’
Key not found: ‘port’
Key not found: ‘good’
Key not found: ‘bad’

I believe these are required fields within the config.hocon however I’m unsure how to find this file and update it which I think might sort these errors.

Any help much appreciated.

5

Hi @shamsul910 ,

We have a couple of example config files in the collector github repository:

Our docs site also has a full list of every field in the config file

6

Thank you for your reply. I guess my quetion is how can I use the examples you’ve provided within that command so that it looks for that example stream-collector/config.kinesis.minimal.hocon at master · snowplow/stream-collector · GitHub

Would I adjust the command to
docker run --rm
-v $PWD/config.hocon:/snowplow/config.kinesis.minimal.hocon
-p 8080:8080
snowplow/scala-stream-collector-kinesis:2.6.1 --config stream-collector/config.kinesis.minimal.hocon at master · snowplow/stream-collector · GitHub

7

Oh I understand the question now! There are a few steps to getting it working:

First, download the example config file, and make any edits if you need to. Save it somewhere like $HOME/collector.hocon.

Now, you need to “mount” the config file into the docker container when you run it, so the config file is visible on the filesystem within the container. This is what the -v argument does in the snippet you posted. There is much more information in this docker documentation about mounting files into a container. The syntax is -v <path-on-host-machine>:<path-on-container>.

Next, you need to tell the collector where to look for the file inside the container using the --config argument.

So a full command might look something like this, assuming your local file is saved at $HOME/collector.hocon:

docker run \
  -v $HOME/collector.hocon:/snowplow/collector.hocon \
  -p 8080:8080 \
  snowplow/scala-stream-collector-kinesis:2.6.1 --config /snowplow/collector.hocon