Hi! Events stopped coming in our snowflake database, and I discovered that enriched kinesis stream have stopped receiving data. raw stream keeps receiving events, so I suppose the problem must be in the enrich app. the last message in the enrich app log is this:
INFO snowplow.enrich.stream.sinks.KinesisSink - Successfully wrote 35 out of 35 records
INFO snowplow.enrich.stream.sources.KinesisSource - Checkpointing shard shardId-000000000029
lib.worker.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: ip-172-31-15-27.eu-west-1.compute.internal:dad813a8-1cc6-43cc-bd9b-f3ea2d31128e
stream.sources.KinesisSource - Processing 54 records from shardId-000000000028
Processing 32 records from shardId-000000000029
the log suddenly ends without any errors, but problem here is that elected leader does not exist anymore, but it is specified in the dynamo tables as leaseOwner. my guess that the app was restarted by autoscaling, so the id of the instance has changed.
terminating enrich app instance and starting a new one does not change anything, dynamo still points to this worker that does not exist and enrich app does not print any new logs in cloudwatch.
could you guide me please what else should I check, because it seems like something is wrong with the assignment of new enrich instances to kinesis streams.
P.S I also do not see any java processes running on an instance when checking via ps aux | grep java on newly launched instances and no snowplow files: sudo find / -type f -name "*snowplow*"
thanks!