Do we need to recreate the old DynamoDB associated or update them? I tried dropping the dynamodb table and while it got recreated it did not get populated with items, leading into error “LeaseTable is empty”.
Hey, I figured that the reason why data did not arrive in the loader buckets was because enrich was configured on the “good” stream from collector, however our correct data ends up in the bad stream due to it’s >1MB size.
On the configuration page I did not see any specific property with which I could increase this limit of 1 MB.
Is it possible in any other way or should we compress the data that we send? Now I set the bad stream as input for the enrich and thus data flows though and ends up in our data warehouse, however this does not feel right
This limit is non-configurable as it’s a built in protection to prevent any single event from exceeding 1 MB as that’s the limit of what Kinesis allows for a single message - so this is a Kinesis limit rather than a Snowplow one.
This is possible to overcome by moving to a different messaging service (Kafka / PubSub) but 1 MB is really large (assuming ~1 bytes per character that’s ~1 million characters per event). I’m curious as to what your use case as there may be a better solution depending on what you are trying to achieve (it’s possible to to compress events but not ideal as you’ll need something to decompress them again).