Configure stream-collector, stream-enrich and iglu-server to use JSON logging

For engineers
1

Hi,

We’re using these Docker images to run snowplow and using DataDog for our application logging.

How can I configure JSON logging in each of these components?

Thanks

Andrew

2

Hi @andrew ,

Welcome to Snowplow community!

I’m not sure exactly what you mean by JSON logging, so I’ll assume that you’re talking about structured logging where each line of log would be a JSON of key value pairs, like here.

For that to be possible, the apps would need to log using StructuredArguments, which is not the case at the moment.

Why do you need to log the lines as JSONs ?

3

Hi @BenB

We’re using DataDog to visualise our application logs. Their documentation recommends to use JSON logging (or structured as you say). https://docs.datadoghq.com/logs/log_collection/java/?tab=log4j

To alleviate this complexity, configure your logging library to produce your logs in JSON format

4

If structured logging isn’t possible it looks like we can use grok parsing instead: https://docs.datadoghq.com/logs/log_configuration/parsing/?tab=matchers

UPDATE: I used grok parsing with the following rule

JavaRule \[%{notSpace:thread}\] %{notSpace:level} %{notSpace:class} - %{data:msg}