Enrich Server Cannot Convert Configuration/Enrichment (No Longer Validating Events)

For engineers Troubleshooting
1

Starting on May 6th, 2022 our enrich-server began to throw the following error:

[pool-1-thread-2] ERROR com.snowplowanalytics.snowplow.enrich.common.fs2.Run - CLI arguments valid but some of the configuration is not correct. Error: Cannot convert configuration to a io.circe.Json. Failures are:
(/snowplow/config/enrich.hocon: 51) Unable to parse the configuration: Expecting close brace } or a comma, got ':' (if you intended ':' to be part of a key or string value, try enclosing the key or value in double quotes).

This seems to happen when the server is attempting to pull enrichments from DynamoDB. Thinking that this might be an issue with our custom enrichments, I removed all enabled enrichments and deployed once again via terraform:

module "enrich_kinesis" {
  source = "snowplow-devops/enrich-kinesis-ec2/aws"

  name                 = "enrich-server-${var.aws_region}"
  vpc_id               = var.vpc_id
  subnet_ids           = var.subnet_ids_private
  in_stream_name       = module.good_stream.name
  enriched_stream_name = module.enriched_stream.name
  bad_stream_name      = module.bad_stream.name

  ssh_key_name         = "snowplow-ssh-local"
  ssh_ip_allowlist     = ["${var.vpc_cidr}"]

  custom_s3_hosted_assets_bucket_name = "${var.aws_bucket_name}-${var.aws_region}"
  telemetry_enabled = var.telemetry_enabled
  associate_public_ip_address = false

  # Linking in the custom Iglu Server here
  custom_iglu_resolvers = [
    {
      name            = "Custom Iglu Server"
      priority        = 0
      uri             = "http://${module.iglu_lb.dns_name}/api"
      api_key         = var.iglu_super_api_key
    }
  ]
}

I am still getting the same errors. I can see all of the following enrichments in the enrich-server-us-east-1-config table:

snowplow_enrichment_campaign_attribution
snowplow_resolver
snowplow_enrichment_event_fingerprint_config
snowplow_enrichment_ua_parser_config
snowplow_enrichment_referer_parser
snowplow_enrichment_yauaa_enrichment_config

I’ve also attempted deleting out these default configs and restarting the enrichment server. While the error seems to go away, the server still refuses to pull records form kinesis and validate them. We have made no significant changes to our pipeline since this date, so I’m truly at a loss for why the enrich-server’s behavior suddenly changed.

2

Hi @Katie62442 this error seems to be with the config file for enrich not for an enrichment itself.

If you can SSH into the server and share the Enrich config file here we might be able to spot what has gone wrong!

3

So, I’ve actually been struggling to ssh into the EC2 instance to retrieve the config. This is the current steps I’ve taken to access the enrich server:

1. Generate .pem file using aws-cli
aws ec2 create-key-pair --key-name snowplow-test-key --query 'KeyMaterial' --output text > snowplow-test-key.pem

2. Replace ssh key name in the terraform file and redeploy
 ssh_key_name         = "snowplow-test-key"

3. Make sure that the Security Group has an open port for my IP

4.  ssh -i "snowplow-test-key.pem" ec2-user@<ip_addr>

Assuming that my networking configurations are correct, does that look like a valid method for getting the appropriate ssh key? I am getting a connection refused error.

I assume that the enrich config is generated on launch? Could we simply alter the launch config to output this file?

4

Hi @Katie62442 the launch configuration template should actually have what you need - can you share that here?

5

Yeah, here’s the user data:

#!/bin/bash
set -e -x
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# -----------------------------------------------------------------------------
#  BASE INSTALL
# -----------------------------------------------------------------------------

readonly CONFIG_DIR=/opt/snowplow/config

function install_docker_ce() {
  sudo yum install -y docker-20.10.4-1.amzn2
  sudo systemctl enable docker
  sudo systemctl start docker
  sudo usermod -a -G docker ec2-user
}

function get_instance_id() {
  curl --silent --location "http://<ip>/latest/meta-data/instance-id/"
}

install_docker_ce

sudo mkdir -p ${CONFIG_DIR}
sudo cat << EOF > ${CONFIG_DIR}/enrich.hocon
# Config Hash: <hash>
{
  "input": {
    "type": "Kinesis"
    "appName": "enrich-server-us-east-1"
    "streamName": "good-stream"
    "region": "us-east-1"
    "initialPosition": {
      "type": "TRIM_HORIZON"
    }
  }
  "output": {
    "good": {
      "type": "Kinesis"
      "streamName": "enriched-stream"
      "region": "us-east-1"
      "maxBufferedTime": 500 millis
      "collection": {
        "maxCount": 500
        "maxSize": 1000000
      }
    }
    "bad": {
      "type": "Kinesis"
      "streamName": "bad-stream"
      "region": "us-east-1"
      "maxBufferedTime": 500 millis
      "collection": {
        "maxCount": 500
        "maxSize": 1000000
      }
    }
  }

  "assetsUpdatePeriod": "7 days"

  "monitoring": {
    "metrics": {
      "stdout": {
        "period": "60 seconds"
      }
      "cloudwatch": false
    }
  }

  "telemetry": {
    "disable": true
    "interval": 15 minutes
    "method": POST
    "collectorUri": 
    "collectorPort": 443
    "secure": true
    "userProvidedId": ""
    "autoGeneratedId": 
    "moduleName": enrich-kinesis-ec2
    "moduleVersion": 0.3.0
  }

  "featureFlags" : {
    "acceptInvalid": false
  }
}

EOF

sudo docker run \
  -d \
  --name enrich \
  --restart always \
  --network host \
  --log-driver awslogs \
  --log-opt awslogs-group=/aws/ec2/enrich-server-us-east-1 \
  --log-opt awslogs-stream=$(get_instance_id) \
  -v ${CONFIG_DIR}:/snowplow/config \
  -e 'JAVA_OPTS=-Dorg.slf4j.simpleLogger.defaultLogLevel=info' \
  snowplow/snowplow-enrich-kinesis:3.0.3 \
  --config /snowplow/config/enrich.hocon \
  --iglu-config dynamodb:us-east-1/enrich-server-us-east-1-config/snowplow_resolver \
  --enrichments dynamodb:us-east-1/enrich-server-us-east-1-config/snowplow_enrichment_
6

Hi @Katie62442 thanks for that! Looks like there is an issue when Telemetry is disabled the config is not formed correctly anymore. For the meantime you can ping back to and older version of this module (v0.2.1).

We will update here once the module is fixed! Apologies for the inconvenience.

1 Like
7

@Katie62442 just sharing that the bug fix is now here (Release/0.3.1 by jbeemster · Pull Request #19 · snowplow-devops/terraform-aws-enrich-kinesis-ec2 · GitHub) and as soon as we merge it in you should be able to bump up to latest again.

8

Hi @Katie62442

We’ve published v0.3.1 - if you can update your terraform to use this version of snowplow-devops/enrich-kinesis-ec2/aws then you should be good.

1 Like