Hello! Is there a way to monitor the progress of the S3 Loader reading event data from Kinesis? I’d like to determine when to scale up the number of S3 Loader instances running when the stream has multiple shards, but am not sure how to tell if it’s falling behind on the stream or not.
Thanks for any insight!
You can look up Kinesis metrics in Cloudwatch. GetRecords.IteratorAgeMilliseconds is a per stream metric which reflects how much your consumers are lagging behind. From the AWS docs:
The age of the last record in all GetRecords calls made against an Kinesis stream, measured over the specified time period. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The Minimum and Maximum statistics can be used to track the progress of Kinesis consumer applications. A value of zero indicates that the records being read are completely caught up with the stream.