Enrich Server Cannot Convert Configuration/Enrichment (No Longer Validating Events)

Starting on May 6th, 2022 our enrich-server began to throw the following error:

[pool-1-thread-2] ERROR com.snowplowanalytics.snowplow.enrich.common.fs2.Run - CLI arguments valid but some of the configuration is not correct. Error: Cannot convert configuration to a io.circe.Json. Failures are:
(/snowplow/config/enrich.hocon: 51) Unable to parse the configuration: Expecting close brace } or a comma, got ':' (if you intended ':' to be part of a key or string value, try enclosing the key or value in double quotes).

This seems to happen when the server is attempting to pull enrichments from DynamoDB. Thinking that this might be an issue with our custom enrichments, I removed all enabled enrichments and deployed once again via terraform:

module "enrich_kinesis" {
  source = "snowplow-devops/enrich-kinesis-ec2/aws"

  name                 = "enrich-server-${var.aws_region}"
  vpc_id               = var.vpc_id
  subnet_ids           = var.subnet_ids_private
  in_stream_name       = module.good_stream.name
  enriched_stream_name = module.enriched_stream.name
  bad_stream_name      = module.bad_stream.name

  ssh_key_name         = "snowplow-ssh-local"
  ssh_ip_allowlist     = ["${var.vpc_cidr}"]

  custom_s3_hosted_assets_bucket_name = "${var.aws_bucket_name}-${var.aws_region}"
  telemetry_enabled = var.telemetry_enabled
  associate_public_ip_address = false

  # Linking in the custom Iglu Server here
  custom_iglu_resolvers = [
      name            = "Custom Iglu Server"
      priority        = 0
      uri             = "http://${module.iglu_lb.dns_name}/api"
      api_key         = var.iglu_super_api_key

I am still getting the same errors. I can see all of the following enrichments in the enrich-server-us-east-1-config table:


I’ve also attempted deleting out these default configs and restarting the enrichment server. While the error seems to go away, the server still refuses to pull records form kinesis and validate them. We have made no significant changes to our pipeline since this date, so I’m truly at a loss for why the enrich-server’s behavior suddenly changed.

Hi @Katie62442 this error seems to be with the config file for enrich not for an enrichment itself.

If you can SSH into the server and share the Enrich config file here we might be able to spot what has gone wrong!

So, I’ve actually been struggling to ssh into the EC2 instance to retrieve the config. This is the current steps I’ve taken to access the enrich server:

1. Generate .pem file using aws-cli
aws ec2 create-key-pair --key-name snowplow-test-key --query 'KeyMaterial' --output text > snowplow-test-key.pem

2. Replace ssh key name in the terraform file and redeploy
 ssh_key_name         = "snowplow-test-key"

3. Make sure that the Security Group has an open port for my IP

4.  ssh -i "snowplow-test-key.pem" ec2-user@<ip_addr>

Assuming that my networking configurations are correct, does that look like a valid method for getting the appropriate ssh key? I am getting a connection refused error.

I assume that the enrich config is generated on launch? Could we simply alter the launch config to output this file?

Hi @Katie62442 the launch configuration template should actually have what you need - can you share that here?

Yeah, here’s the user data:

set -e -x
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------

readonly CONFIG_DIR=/opt/snowplow/config

function install_docker_ce() {
  sudo yum install -y docker-20.10.4-1.amzn2
  sudo systemctl enable docker
  sudo systemctl start docker
  sudo usermod -a -G docker ec2-user

function get_instance_id() {
  curl --silent --location "http://<ip>/latest/meta-data/instance-id/"


sudo mkdir -p ${CONFIG_DIR}
sudo cat << EOF > ${CONFIG_DIR}/enrich.hocon
# Config Hash: <hash>
  "input": {
    "type": "Kinesis"
    "appName": "enrich-server-us-east-1"
    "streamName": "good-stream"
    "region": "us-east-1"
    "initialPosition": {
      "type": "TRIM_HORIZON"
  "output": {
    "good": {
      "type": "Kinesis"
      "streamName": "enriched-stream"
      "region": "us-east-1"
      "maxBufferedTime": 500 millis
      "collection": {
        "maxCount": 500
        "maxSize": 1000000
    "bad": {
      "type": "Kinesis"
      "streamName": "bad-stream"
      "region": "us-east-1"
      "maxBufferedTime": 500 millis
      "collection": {
        "maxCount": 500
        "maxSize": 1000000

  "assetsUpdatePeriod": "7 days"

  "monitoring": {
    "metrics": {
      "stdout": {
        "period": "60 seconds"
      "cloudwatch": false

  "telemetry": {
    "disable": true
    "interval": 15 minutes
    "method": POST
    "collectorPort": 443
    "secure": true
    "userProvidedId": ""
    "moduleName": enrich-kinesis-ec2
    "moduleVersion": 0.3.0

  "featureFlags" : {
    "acceptInvalid": false


sudo docker run \
  -d \
  --name enrich \
  --restart always \
  --network host \
  --log-driver awslogs \
  --log-opt awslogs-group=/aws/ec2/enrich-server-us-east-1 \
  --log-opt awslogs-stream=$(get_instance_id) \
  -v ${CONFIG_DIR}:/snowplow/config \
  -e 'JAVA_OPTS=-Dorg.slf4j.simpleLogger.defaultLogLevel=info' \
  snowplow/snowplow-enrich-kinesis:3.0.3 \
  --config /snowplow/config/enrich.hocon \
  --iglu-config dynamodb:us-east-1/enrich-server-us-east-1-config/snowplow_resolver \
  --enrichments dynamodb:us-east-1/enrich-server-us-east-1-config/snowplow_enrichment_

Hi @Katie62442 thanks for that! Looks like there is an issue when Telemetry is disabled the config is not formed correctly anymore. For the meantime you can ping back to and older version of this module (v0.2.1).

We will update here once the module is fixed! Apologies for the inconvenience.

1 Like

@Katie62442 just sharing that the bug fix is now here (Release/0.3.1 by jbeemster · Pull Request #19 · snowplow-devops/terraform-aws-enrich-kinesis-ec2 · GitHub) and as soon as we merge it in you should be able to bump up to latest again.

Hi @Katie62442

We’ve published v0.3.1 - if you can update your terraform to use this version of snowplow-devops/enrich-kinesis-ec2/aws then you should be good.

1 Like