Snowplow Open Source add Enrichment

For engineers Enrichment
1

Hello

I set up Snowplow using the Quickstart Guide in GCP.
I now want to add the enrichments IP Lookup & IP Anon.

I have extended the config.tf file as follows:


  iglu_config = templatefile("${path.module}/templates/iglu_config.json.tmpl", { resolvers = jsonencode(local.resolvers) })

  campaign_attribution     = var.enrichment_campaign_attribution == "" ? file("${path.module}/templates/enrichments/campaign_attribution.json") : var.enrichment_campaign_attribution
  event_fingerprint_config = var.enrichment_event_fingerprint_config == "" ? file("${path.module}/templates/enrichments/event_fingerprint_config.json") : var.enrichment_event_fingerprint_config
  referer_parser           = var.enrichment_referer_parser == "" ? file("${path.module}/templates/enrichments/referer_parser.json") : var.enrichment_referer_parser
  ua_parser_config         = var.enrichment_ua_parser_config == "" ? file("${path.module}/templates/enrichments/ua_parser_config.json") : var.enrichment_ua_parser_config
  yauaa_enrichment_config  = var.enrichment_yauaa_enrichment_config == "" ? file("${path.module}/templates/enrichments/yauaa_enrichment_config.json") : var.enrichment_yauaa_enrichment_config
 ip_lookups        = var.enrichment_ip_lookups == "" ? file("${path.module}/templates/enrichments/ip_lookups.json") : var.enrichment_ip_lookups
 anon_ip        = var.enrichment_anon_ip == "" ? file("${path.module}/templates/enrichments/anon_ip.json") : var.enrichment_anon_ip

  enrichments_list = compact([
    local.campaign_attribution,
    local.event_fingerprint_config,
    local.referer_parser,
    local.ua_parser_config,
    local.yauaa_enrichment_config,
    local.ip_lookups,
    local.anon_ip,    
    var.enrichment_api_request_enrichment_config,
    var.enrichment_cookie_extractor_config,
    var.enrichment_currency_conversion_config,
    var.enrichment_http_header_extractor_config,
    var.enrichment_iab_spiders_and_bots_enrichment,
    var.enrichment_javascript_script_config,
    var.enrichment_pii_enrichment_config,
    var.enrichment_sql_query_enrichment_config,
    var.enrichment_weather_enrichment_config,
  ])

So exactly like the 5 default enabled enrichtments.
The apply runs without errors, but I don’t see any change or these enrichments are not performed.
Is my chosen path not right? Does the JSON have to go directly into the main.tf file?

I would be grateful for help.

Best regards,

2

Hey @franco, just checking — do your individual JSON config files have enabled set to true inside them?

1 Like
3

Hi @stanch, thank you. Enabled ist set to “true”

LookUP:
{
	"schema": "iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-0",
	"data": {
		"name": "ip_lookups",
		"vendor": "com.snowplowanalytics.snowplow",
		"enabled": true,
		"parameters": {
            "geo": {
                "database": "GeoLite2-City.mmdb",
                "uri": "{PLACEHOLDER_BUCKET_URL}"
            }
        }
	}
}

IP:ANON
{
	"schema": "iglu:com.snowplowanalytics.snowplow/anon_ip/jsonschema/1-0-1",
	"data": {
		"name": "anon_ip",
		"vendor": "com.snowplowanalytics.snowplow",
		"enabled": true,
		"parameters": {
			"anonOctets": 1,
			"anonSegments": 1
		}
	}
}
4

Hi @franco is there a reason you opted to add this in there rather than using the already available variables in the module?

Like here: terraform-aws-enrich-kinesis-ec2/variables.tf at main · snowplow-devops/terraform-aws-enrich-kinesis-ec2 · GitHub

Documentation here: GitHub - snowplow-devops/terraform-aws-enrich-kinesis-ec2

5

Hi @josh,

I tried to carry out the configuration exactly as they are stored in the pre-configured enrichments. So i thought i could add the enrichments also this way.
I’m somehow more sympathetic to an “extra config” json file. But as I said, only for me personally. If it’s not possible, I’ll follow the documentation in this step.
Thank you very much!

Best regards,
Franco

6

Hi @franco I am afraid unless you follow the tested and exposed module API it is going to be very difficult to help you debug what is going wrong here!

From what I can see it should work as expected - to debug I would be looking at the startup-script that is being applied to the server instance as well as the logs to see if there are any errors in there.