Lag in the elasticsearch

For engineers Enrichment
1

Hi,

We are seeing around 24 hours of lag in the elasticsearch whereas the data processed to redshift is fine. This issue is in prod environment only. In all other environments it is working fine. I am also getting the error in enrichment logs as below:

ERROR com.snowplowanalytics.snowplow.enrich.kinesis.sinks.KinesisSink - 250 records failed with error code ProvisionedThroughputExceededException. Example error message: Rate exceeded for shard shardId-000000000002 in stream prod_enriched under account XXXXXXXXX.

I am suspecting that this is due to large data volume in Prod environment. Please help on this.

Thanks,
Vikram.

2

There are a number of AWS limits imposed on a Kinesis stream that typically scale (for read and write operations) to the number of shards you are running. If you’re generating ProvisionedThroughputExceededException exceptions the easiest solution is to increase the number of shards in your Kinesis stream.

1 Like
3

Thanks Mike for the solution. I have increased the number of shards but that too didn’t resolve the issue and still I am getting the same error. Is there any other option to reduce the lag (24 hours).