[scala] [enrich] exception while sync'ing Kinesis shards and leases

For engineers AWS real-time pipeline
1

Hi,

I am dealing with an exception trying to run enricher and it is failing with this logs:

[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Using workerId: snowplow-staging-data-snowplow-enrich-5f9cdfddd4-tk8nw:6f05d1af-cdd0-44ee-bde6-de196d825075
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Running: data-snowplow.
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Processing raw input stream: snowplow-good
[main] INFO com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for both region name as eu-west-1, and Amazon Kinesis endpoint as https://kinesis.eu-west-1.amazonaws.com. Amazon Kinesis endpoint will overwrite region name.
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 1
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases

The policy included to the enricher has kinesis full access:

{
    "Sid": "VisualEditor1",
    "Effect": "Allow",
    "Action": [
        "kinesis:*"
    ],
    "Resource": "*"
}

I was trying to catch what is happening but i am a bit loss, anyone could lead to what can be misconfigured?

2

Finally it was related with a fatal mess of AWS policies and roles. Easy to see with debug activated -Dorg.slf4j.simpleLogger.defaultLogLevel=debug

3

What was the error exactly in the policies?

4

Can you share the solution? The debug didn’t really reveal anything for us

5

Not sure if was the same problem as OP but if you’re running into difficulties you should make sure that the policy for stream enrich can access both Kinesis and DynamoDB (Dynamo is required for checkpointing).

6

Hey,

I haven’t found the solution yet. Still getting the same error

7

Hey,

I ran into the same issue.

It seems to be related to the dynamoDB permissions.
I didn’t have time to check which permission, I allowed everything and it works.

Good luck.

8

I also ran into this issue. The stream enrich component needs permissions to the following resources:

  • Read permission to input kinesis stream (collector good)
  • List permission to kinesis streams
  • Write permission to output kinesis streams (good & bad)
  • Read/write/create permission to Dynamodb state table (table name is the “appName” value in application.conf)
  • PutMetricData to Cloudwatch

I’ve written a more detailed outline/template of an IAM policy for the Stream Enrich component at https://andrew.hawker.io/writings/2020/02/12/snowplow-fargate-permissions/.

2 Likes
9

Hey @ahawker, I just found your reply after getting this same error. I tried attaching your policy to my StreamEnrich instance, but it’s still not working for me. Do you have any additional policy for your enrich instance?

Edit: figured it out: I had manually created the DynamoDB table. After deleting it, StreamEnrich created the table at startup and it works now.

Edit2: Thanks again @ahawker, the IAM policy blog post is super useful!

1 Like