We ran into issues with Snowbridge reporting the following error:
Failed to pull next Kinesis record from Kinsumer client: shard error (shardId-n) in getShardIterator: ResourceNotFoundException: Shard shardId-n in stream snowplow-analytics-enriched-good under account 123456 does not exist
I think this was caused by the updates to the DynamoDB tables failing and potentially kinesis shard scaling occurring.
The solution for this was to delete the shards in the metadata table that no longer exist.
Is there a cleaner way to do this? Shouldn’t service check for shards as part of startup?
I have only encountered this error during testing when I’ve made some configuration mistake - for example when I’ve changed the stream I’m reading from, but forgot to purge the DDB table - then I’ll see this error.
The client does check for shards - the way it works is that there is one ‘leader’, which polls the stream for shards at an interval, and updates that table. The other clients (which will be one per pod/snowbridge instance) read from DDB and operate from there.
So I guess you can reach this state if the leader wasn’t able to update DDB correctly, and another client booted up before it did so, attempted to read an old shard ID, and that shard didn’t exist.
But what I’m confused about is how a shard ID already in there returned this error - I don’t know enough about the internals of kinesis but I would have expected to be able to call a method on a shard ID even after that shard ID has been merged to another in a scaling action.
Is it possible that things weren’t operational for long enough for all of the data in the shard to have expired completely?