Pass config and resolver as base64 strings to rdb-loader

[mgloel]

Hi,

we are trying to pass the resolver.json and config.hocon as base64 strings to the rdb-loader. Unfortunately we cannot pass them as variables because the command expects a string. We did not find another way then to explicitly add the base64 string of the respective files.
We tried to assign the base64 to a variable but the command evaluates the variable as a string

#!/bin/bash

export RESOLVERB64="\"$(base64 resolver.json)\""
export CONFIGB64="\"$(base64 config.hocon)\""

/home/snowplow/bin/snowplow-rdb-loader --iglu-config $RESOLVERB64 --config $CONFIGB64

Is there another way than passing the explicit base64 string?

[anton]

I think you need something like:

snowplow-rdb-loader-0.19.0.jar \
    --iglu-config $(cat /path/to/iglu_resolver.json | base64 -w 0) \
    --config $(cat /path/to/config.hocon | base64 -w 0)

[Oleksandr_Shcherbak]

Hi @anton
I’m trying to deploy an image onto k8s pod. I got a similar error and I tried to fix it with your suggestion, but it doesn’t solve my problem. Seems like app can’t decode strings. What is the problem could be? Thanks

[enes_aldemir]

Hey @Oleksandr_Shcherbak,

I don’t think pod arguments would be able to execute bash command therefore this way is unlikely to work with Kubernetes.

You might try the way described here to pass config to container.

[anton]

Hi @Oleksandr_Shcherbak,

I think @enes_aldemir is right about passing config parameters as Java System Properties, but unfortunately only parameters from --config can be passed like that. --iglu-config however accepts only base64-encoded strings. I’m not very familiar with K8S, but I believe you can get base64-encoded parameters from ConfigMap. For example, this SO question looks relevant: Kubernetes/Helm: any examples with ConfigMap and "binaryData:"? - Stack Overflow

[Oleksandr_Shcherbak]

Thanks for responses @enes_aldemir @anton !

I set args via ENV vars successfully, but I got error with DynamoDB. It looks like it can’t resolve reference to file from base64 encoded string and go to dynamoDB table. Did I miss something from docs?

Caused by: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: User: arn:aws:sts::908485410358:assumed-role/dwh-self-snowplow-oidc/aws-sdk-java-1649861410240 is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:eu-central-1:908485410358:table/snowplow-transformer

[anton]

I think your configs are fine. The problem is with permissions. Your user/role cannot create a DynamoDB table (which is necessary for any application using Kinesis Client Library - the app stores its progress there) and it is typically a first thing the Transformer does when it starts.

[Oleksandr_Shcherbak]

Yes, I solved it adding permissions, thanks!

[EddieM]

Hi @Oleksandr_Shcherbak
Thanks for the update - really helpful to other users.
Cheers,
Eddie