Debug collector and enrichment

For engineers
1

Is it possible to enable some setting or flag in order for the collector /enrichment to log events for what it’s doing? I’m experiencing some sporadic behavior and it’d be helpful to know what the collector and enrichment processes are doing.

I know for performance reasons this is not a good idea, but for learning / troubleshooting in non prod it’d be helpful.

I have a script to continuously poll kinesis streams but thought maybe there’s an easier way.

2

In a non-prod environment the easiest way is to probably use a configuration that uses stdout as the sink e.g., stream-collector-stdout. These images should exist for both collector and enrichment and will just write the events to stdout rather than PubSub / Kinesis / Kafka.

Also - depending on what you are trying to debug then Micro (collection + schema validation) or Mini (collection + schema validation + enrichment) might be helpful too.

2 Likes
3

Circling back on my issue. My sporadic issue was quite a bit of time was going by without any events going into the Enricher process. Turns out my collector config had high buffer limits for timeLimit and recordLimit. I lowered those for testing and now see my events.

For reference:

      byteLimit = 256000000 # ~ 244Mb
      recordLimit = 10  # Not supported by Kafka; will be ignored
      timeLimit = 5000 # 5 seconds

Thanks for the help!