Using custom javascript enrichment

For engineers Enrichment
1

Hi Experts,

We are trying to implement custom javascript enrichment to the enrichment process hosted in a GCP pipeline.
Here is a verbal eg for what we are trying to achieve.

We would like to modify user_ipaddress field in 2 ways based on the value of another field of the event.

  1. Similar to ip_annon enrichment, we want to mask the last n octets of the user_ipaddress.
  2. Mask the entire user_ipaddress .

Is it possible to achieve this by modifying the user_ipaddress field in place like we have for ip_annon without having to add a custom schema for the updated value in bigquery?

2

Hey @ashish_george. Yes, it’s totally doable. You would need to use the setter to amend the canonical field, namely setUser_ipaddress(). That is, depending on the value of user_ipaddress, which you can obtain with the getter getUser_ipaddress(), you would set the new value like this:

event.setUser_ipaddress(new String(some_modified_value));

where some_modified_value is your new value you want to assign to user_ipaddress depending on its original value.

Note that you do not need to return anything in process(event) function as you do not want any derived context. Using setters overrides the original values in event’s TSV. Thus, be extra careful as you won’t be able to restore the original values.

1 Like
3

@ihor Thank you for your quick response, really appreciate it.
We will try to implement this, and will share the how it went.

4

Hi @ihor ,

In our testing environment, i added the cx js enrichment with just a small piece of code to replace all user_ipaddress to xxx.xxx.xxx.xxx

I sent some events but the user_ipaddress is still the same. the enrichment is not updating the user_ipaddress .
Below is the 2 line of code.

function process(event)
{
var obfuscated_ip = “xxx.xxx.xxx.xxx”;
event.setUser_ipaddress(new String(obfuscated_ip));
}

5

Hi @ashish_george , that should work. Most likely you need to restart enrichment to clear cache (assuming you made the updates in the right place).

6

yes @ihor I added the below enrichment, restarted the enricher component as well.

{
“schema”: “iglu:com.snowplowanalytics.snowplow/javascript_script_config/jsonschema/1-0-0”,

“data”: {

  "vendor": "com.snowplowanalytics.snowplow",
  "name": "javascript_script_config",
  "enabled": true,
  "parameters": {
  	"script": "ZnVuY3Rpb24gcHJvY2VzcyhldmVudCkKeyAgIAogICAgdmFyIG9iZnVzY2F0ZWRfaXAgPSAieHh4Lnh4eC54eHgueHh4IjsKICAgIGV2ZW50LnNldFVzZXJfaXBhZGRyZXNzKG5ldyBTdHJpbmcob2JmdXNjYXRlZF9pcCkpOwp9"
  }

}
}

These are the enricher logs, which shows that the javascript_script_config is loaded as well.

[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.pubsub.Environment - Parsed following enrichments: iab_spiders_and_robots_enrichment, event_fingerprint_config, ua_parser_config, anon_ip, ip_lookups, javascript_script_config

Not sure what’s wrong.

7

Hi @ashish_george , there are 2 issues that I could spot. I can see wrong quotes in your configuration, which could be simply copy/paste problem in your comment than the actual content of the configuration file.

For example, you are using “curly” quotes instead of straight quotes around schema and data properties. For better visibility, here’s the image that demonstrates it better

image
image1378×454 82.5 KB

Secondly, your base64 encoded string needs to end in one or two equal sign (it least that was an issue in the past to my knowledge). That is, if you add a new line to your unencoded function, the encoded version will get it with an equal sign (or two). Something like this

ZnVuY3Rpb24gcHJvY2VzcyhldmVudCkKewogICAgdmFyIG9iZnVzY2F0ZWRfaXAgPSAieHh4Lnh4eC54eHgueHh4IjsKICAgIGV2ZW50LnNldFVzZXJfaXBhZGRyZXNzKG5ldyBTdHJpbmcob2JmdXNjYXRlZF9pcCkpOwp9Cg==

Once updated, do restart enrichment to apply the changes.

8

@ihor Awesome, i got it working, thanks a lot for your pointers, worked like a charm.

9

@ihor I was trying out a different way to hash the IP.
Was not able to get the below code working, is it because of the md5 usage?

function process(event) {
    var ipAddress = event.getUser_ipaddress();
    let saltStr = '.125.078';
    var saltedIp = md5(ipAddress + saltStr);
    event.setUser_ipaddress(new String(saltedIp));
}
10

Hi @ashish_george,

We are working on a new guide for writing and testing JavaScript enrichments. It’s still in review, but as a quick way of unblocking you, I think this section may help: Writing your enrichment | Snowplow Documentation. (You can use md5 hashing from the Apache library.)

11

@stanch

Thank you Nick for the quick response.
I tried with the below code as per the documentation, events are not flowing in.

function process(event) {
    const salt = 'pepper';
    const hashedIp = org.apache.commons.codec.digest.DigestUtils.md5Hex(event.getUser_ipaddress() + salt);
    event.setUser_ipaddress(new String(hashedIp));
}
12

What version of Enrich are you using? The only thing I can think of is if your Enrich is below 3.0.0, you wouldn’t get ES6 syntax. You can try changing const to var though.

13

@stanch Enrich version was 2.0.5, i made the changes as you said and it worked. Thanks a lot Nick. really appreciate the quick response and resolution.