Setting aws credentials chain to default with new stream enrich configuration file format

Hello,

Was wondering if it is possible to set AWS credential chain to default with the new configuration file format (enrich/config.kinesis.extended.hocon at master · snowplow/enrich · GitHub) as it previously was. Asking as we need to configure Snowplow running in EKS to use IRSA rather than node IAM role.

Hi @Aurimas_Griciunas ,

The default provider chain is used by default.

You might be interested in following this issue.

Thank you for the link. I’ll still post some thoughts here for others to follow in the future.

The way AWS EKS IRSA works is that on assigning a service account to the pod a web identity token is created as a file in the pod and two environment variables are set:

AWS_ROLE_ARN - the AWS IAM role arn of the role to be assumed by the pod.
AWS_WEB_IDENTITY_TOKEN_FILE - points to the web identity token to be used for authentication.

Default chain should take these env variables as the top priority if I am not mistaken.

For stream enricher it falls back to assuming node IAM role for some reason.

facing the same problem :confused:

IRSA works for the kinesis client but fails on dynamoDb client :confused:

i’m trying to understand the code but i have no experience on scala :cry: but i’m presuming that the problem comes from how the dynamodb aws client is initialized.

seems related to lease function that uses version 2 of aws sdk
software.amazon.kinesis.leases.exceptions.DependencyException: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: User: arn:aws:sts::xxxx:assumed-role/xxxx/i-xxx is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:eu-west-3:xxxxx:table/snowplow-enrich-kinesis because no identity-based policy allows the dynamodb:DescribeTable action (Service: DynamoDb, Status Code: 400, Request ID: xxxxx)

Hi @Aurimas_Griciunas and @hanfi ,

Sorry for taking so long to get back to you.

We have released enrich-kinesis 3.4.1-rc2 that adds irsa support.

Please stay tuned on Discourse for the final release (hopefully before the end of the week).