Best way to test/debug javascript enrichment scripts?

For engineers Enrichment
1

When using this feature - https://github.com/snowplow/snowplow/wiki/JavaScript-script-enrichment - there is a link to http://snowplowanalytics.com/blog/2013/10/21/scripting-hadoop-part-1-adventures-with-scala-rhino-and-javascript/ which mentions the scalding example for running Scala, but I’m not sure that would apply to javascript as well.

The feedback loop on testing changes to EmrEtlRunner processes is kind of long, when it takes at least 8 minutes to start an EMR cluster up. Is there a way that others have used for testing javascript script enrichments without having to run the whole process every time? Any tips would probably save a lot of time!

Thanks,
Darren

2

Hi @darren - I have seen a couple of options used:

  1. Bring up a Scala REPL, import Scala Common Enrich and invoke the enrichment directly
  2. Build a shim and run the code directly in node.js

I prefer the first option because you are much closer to the true runtime (versus node.js which is testing the JavaScript but not much else).

3

Hi @alex,

Thanks so much for responding to my question. I will give option 1. a shot most likely. I appreciate it!

Best,
Darren

4

@darren have you had any luck with option #1? I haven’t been able to invoke the enrichment directly. There are no public methods on the JavascriptScriptEnrichment object.

I have been trying to execute parts of https://github.com/snowplow/snowplow/blob/master/3-enrich/scala-common-enrich/src/test/scala/com.snowplowanalytics.snowplow.enrich.common/enrichments/registry/JavascriptScriptEnrichmentSpec.scala inside the REPL and here’s what I have done so far:

cd ~/Projects/github/snowplow/snowplow/
git checkout r94-hill-of-tara
cd ./3-enrich/scala-common-enrich/
sbt console

scala> import com.snowplowanalytics.snowplow.enrich.common.enrichments.registry.JavascriptScriptEnrichment
import com.snowplowanalytics.snowplow.enrich.common.enrichments.registry.JavascriptScriptEnrichment

scala> val PreparedEnrichment = {
     |     val script =
     |       s"""|function process(event) {
     |           |  var platform = event.getPlatform(),
     |           |      appId    = event.getApp_id();
     |           |
     |           |  if (platform == "server" && appId != "secret") {
     |           |    throw "Server-side event has invalid app_id: " + appId;
     |           |  }
     |           |
     |           |  // Use new String() because http://nelsonwells.net/2012/02/json-stringify-with-mapped-variables/
     |           |  var appIdUpper = new String(appId.toUpperCase());
     |           |  return [ { schema: "iglu:com.acme/foo/jsonschema/1-0-0",
     |           |             data:   { appIdUpper: appIdUpper }
     |           |           } ];
     |           |}
     |           |""".stripMargin
     |
     |     val compiled = JavascriptScriptEnrichment.compile(script)
     |     JavascriptScriptEnrichment(compiled.toOption.get)
     | }
<console>:30: error: method compile in object JavascriptScriptEnrichment cannot be accessed in object com.snowplowanalytics.snowplow.enrich.common.enrichments.registry.JavascriptScriptEnrichment
    val compiled = JavascriptScriptEnrichment.compile(script)