Enrich app stopped processing events

Hi! Events stopped coming in our snowflake database, and I discovered that enriched kinesis stream have stopped receiving data. raw stream keeps receiving events, so I suppose the problem must be in the enrich app. the last message in the enrich app log is this:

INFO snowplow.enrich.stream.sinks.KinesisSink - Successfully wrote 35 out of 35 records
INFO snowplow.enrich.stream.sources.KinesisSource - Checkpointing shard shardId-000000000029
lib.worker.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: ip-172-31-15-27.eu-west-1.compute.internal:dad813a8-1cc6-43cc-bd9b-f3ea2d31128e
stream.sources.KinesisSource - Processing 54 records from shardId-000000000028
Processing 32 records from shardId-000000000029

the log suddenly ends without any errors, but problem here is that elected leader does not exist anymore, but it is specified in the dynamo tables as leaseOwner. my guess that the app was restarted by autoscaling, so the id of the instance has changed.

terminating enrich app instance and starting a new one does not change anything, dynamo still points to this worker that does not exist and enrich app does not print any new logs in cloudwatch.

could you guide me please what else should I check, because it seems like something is wrong with the assignment of new enrich instances to kinesis streams.

P.S I also do not see any java processes running on an instance when checking via ps aux | grep java on newly launched instances and no snowplow files: sudo find / -type f -name "*snowplow*"


Hi @pavvell ,

Sorry for the delay.

It seems that your instance got stuck in the middle of processing some events.

I would expect KCL to elect a new leader.

Could you try with the log level set to DEBUG please ?

Please note that the app that you are using is deprecated, in favor of enrich-kinesis. You can find the instructions to set it up on this page and the configuration reference here.