Scala Enrich using DynamoDB insted of files

For engineers Enrichment
1

Hello Support,

I just have checked that i am passing directory path in snowplow-stream-enrich 0.10.0 version but it is still connecting to dynamodb but i am using file for resolver and enrichments.

Bellow is the command i am running and error i am getting

Command : java -jar -Dorg.slf4j.simpleLogger.defaultLogLevel=debug snowplow-stream-enrich-0.10.0 --config enrich.conf --resolver file:/root/resolver.json --enrichments file:/root/enrichments/

Error:

[main] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection [id: 1][route: {s}->https://dynamodb.region.amazonaws.com] can be kept alive for 60000 MILLISECONDS

[main] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection released: [id: 1][route: {s}->https://dynamodb.region.amazonaws.com][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]

[main] DEBUG com.amazonaws.request - Received error response: com.amazonaws.AmazonServiceException: User: arn:aws:iam::564569418690:user/kinesis is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:region:564569418690:table/{{enrichStreamsAppName}} (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6A15UD9SAE81QIRNQRQ2BUT7RJVV4KQNSO5AETVJF66Q9AYUAAJG)
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator

2

Hi @geetanshjindal - regardless of the configuration file options you use, Stream Enrich will use DynamoDB with Kinesis.

This is because Stream Enrich uses the Kinesis Client Library (KCL):

http://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

And the KCL uses DynamoDB to keep track of its current progress (“checkpoints”) processing each of a Kinesis stream’s constituent shards.

1 Like
3

Okay, Thanks for the Information