In many configs and thread I see dynamoDB is mentioned. But I see no configurations regarding it.
Is it mandatory to have dynamoDB? if so, any alternative if we don’t have dynamoDB?
Hi @pramod.niralakeri - DynamoDB is primarily used as the backend store for the Kinesis Consumer Library which sits behind our stream consumer applications for AWS.
To use Stream Enrich for example you need to have a DynamoDB table which records the state of the processing of the Kinesis stream to allow for horizontal scaling - so each worker can “steal” the lease on a shard in the Kinesis stream and to ensure other workers know that this shard is taken.
There is no direct reference to DynamoDB in those configs however the name of the application in the config does refer to the name of the table that will be used (terraform-aws-enrich-kinesis-ec2/config.hocon.tmpl at main · snowplow-devops/terraform-aws-enrich-kinesis-ec2 · GitHub). If you have not created the table ahead-of-time the application should take care of creating the table directly (assuming you have granted sufficient IAM permissions to allow that to work).
Hope this helps!