What DynamoDB tables are required for snowplow-stream-enrich to work correctly?

For engineers
1

Hello Dear Developers and Snowplow Support

There was a question in the course of component configuration.
I am using Snowplow Stream Enrich version 3.3.1.
For Snowplow Stream Enrich, I currently have two DynamoDB tables created: “snowplow-enrich-kinesis”, “snowplow-enrich-kinesis-config”. Earlier in terraform configurations I found that these tables are required.
In testing, I see that the “snowplow-enrich-kinesis” table is actually being used. But the second one is not used in any way. What is the “snowplow-enrich-kinesis-config” table for? Can I remove it and not break anything? Or in rare cases it is used and I don’t see it?

And I would also like to understand how snowplow stream enrich understands that it needs a table called “snowplow-enrich-kinesis”. The configuration does not explicitly specify this name anywhere. Also appname is not set.

Please clarify and help me figure it out. Sincerely, Dmitry Medkov.

2

Hi @Dmitry_Medkov so the appName is what is used as the table name for tracking state of progress. If the appName is unset it defaults to “snowplow-enrich-kinesis” → enrich/config.kinesis.extended.hocon at master · snowplow/enrich · GitHub

The other table is intended to hold the configuration for Enrichment JSONs and Iglu Resolver. It gets passed to the application like this, the values look like this and the contents get seeded like this.

The config table is only read when the application boots so you might not see a lot of activity on it but it does indeed get used!

2 Likes
3

Thanks for your reply.

But what if I start a service in a container like this ?
I form configurations in files and they are not stored in DynamoDB.

#!/bin/sh

exec java "${ENRICH_KINESIS_DIR}"/snowplow-enrich-kinesis-"${ENRICH_KINESIS_VERSION}".jar --enrichments "${ENRICH_KINESIS_DIR}"/enrichments --iglu-config "${ENRICH_KINESIS_DIR}"/iglu_resolver.json --config "${ENRICH_KINESIS_DIR}"/enrich_kinesis.hocon >> "${LOG_DIR}"/snowplow-enrich-kinesis.log 2>&1

In that case, the “snowplow-enrich-kinesis-config” table is not used and I can delete it, right?

4

Hey @Dmitry_Medkov indeed if you are not referencing the DynamoDB table anywhere then yes it will not be used and can be deleted.

5

Thank you @josh