Exception initializing

For engineers Troubleshooting
1

Hi, when we try to start our s3 bad loader service in the log file we started receiving the following error. We believe this started when we changed our Kinesis streams from provisioned to ondemand mode.

[RecordProcessor-0000] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitializeTask - Caught exception:
[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisDataFetcher - Initializing shard shardId-000000000000 with 49616304284255447076828137858273813305795307353669632002

How can we resolve this error?

2

Hi @pt-mike ,

Welcome to Snowplow community !

At first I thought that the issue could be that AWS performed some resharding following the change, making what’s in DynamoDB table for KCL invalid, but from AWS documentation :

Switching between capacity modes of a data stream does not cause any disruptions to your applications that use this data stream.

And also :

When you switch from provisioned to on-demand capacity mode, your data stream initially retains whatever shard count it had before the transition, and from this point on, Kinesis Data Streams monitors your data traffic and scales the shard count of this on-demand data stream depending on your write throughput.

Too bad that the log message is not more explicit. Would it be possible to set the log level to DEBUG to hopefully spot more useful information ?

A possible solution would be to delete the DynamoDB table holding KCL information, and update this line in S3 loader to the timestamp (UTC) when it started to fail:

"position": {
  "AT_TIMESTAMP": {
    "timestamp": "2022-04-05T10:00:00Z"
  }
}
3

Hi @BenB, thanks for the reply! I actually went ahead and tested deleting the KCL table in DyanmoDB and restarted the s3 bad loader service. After about a minute the checkpoints in the table were updating and eventually we had s3 bad data loading again. Since this was broken for about 20+ days with our Kinesis retention of only 2 days, we lost most of the bad data.

If we have the issue again I’ll keep your recommendations in mind!

Mike