Enricher not working

For engineers Enrichment
1

I am using following command
./snowplow-stream-enrich-0.9.0 --config en.conf --resolver file:/root/res.json --enrichments file:/root/ip_lookups.json
and i am getting following error:

Exception in thread “main” java.lang.NullPointerException
at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:112)
at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.filter(TraversableLike.scala:262)
at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:108)
at com.snowplowanalytics.snowplow.enrich.kinesis.KinesisEnrichApp$.extractEnrichmentConfig(KinesisEnrichApp.scala:234)
at com.snowplowanalytics.snowplow.enrich.kinesis.KinesisEnrichApp$delayedInit$body.apply(KinesisEnrichApp.scala:131)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.snowplowanalytics.snowplow.enrich.kinesis.KinesisEnrichApp$.main(KinesisEnrichApp.scala:71)
at com.snowplowanalytics.snowplow.enrich.kinesis.KinesisEnrichApp.main(KinesisEnrichApp.scala)
[INFO] [02/18/2017 16:22:04.901] [snowplow-scala-tracker-akka.actor.default-dispatcher-2] [akka://snowplow-scala-tracker/user/IO-HTTP/group-0] Message [spray.can.Http$Connect] from Actor[akka://snowplow-scala-tracker/user/IO-HTTP/host-connector-0/0#816606004] to Actor[akka://snowplow-scala-tracker/user/IO-HTTP/group-0#1639878845] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
Exception in thread “Thread-1” akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://snowplow-scala-tracker/user/IO-HTTP#-1768471552]] after [1000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)
#--------------------------------------------------------------------------------
en.conf
enrich {
source = "kinesis"
sink = “kinesis”

aws {
access-key: "key"
secret-key: “key”
}

streams {
in: {
raw: good

  maxRecords: 10000

  buffer: {
    byte-limit: 64000 #6KB
    record-limit: 100 
    time-limit: 30
  }
}

out: {
  enriched: enriched
  bad: bad

  backoffPolicy: {
    minBackoff: 20
    maxBackoff: 60
  }
}

app-name: "snowplow.kinesis.enrich"
initial-position = "TRIM_HORIZON"
region: "region-name"

}

akka {
loglevel = DEBUG # ‘OFF’ for no logging, ‘DEBUG’ for all logging.
loggers = [“akka.event.slf4j.Slf4jLogger”]
}

monitoring {
snowplow {
collector-uri: "ip address of collector server"
collector-port: 80810
app-id: "app-id"
method: “GET”
}
}
}
#--------------------------------------------------------------------------
res.json

{
“schema”: “iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-0”,
“data”: {
“cacheSize”: 500,
“repositories”: [
{
“name”: “Iglu Central”,
“priority”: 0,
“vendorPrefixes”: [ “com.snowplowanalytics” ],
“connection”: {
“http”: {
“uri”: “http://iglucentral.com”
}
}
}
]
}
}

#-------------------------------------------------------------------------
ip_lookup.json

{
“schema”: “iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/1-0-0”,

"data": {

    "name": "ip_lookups",
    "vendor": "com.snowplowanalytics.snowplow",
    "enabled": true,
    "parameters": {
        "geo": {
            "database": "GeoLiteCity.dat",
            "uri": "http://snowplow-hosted-assets.s3.amazonaws.com/third-party/maxmind"
        }
    }
}

}

2

Hello Support,
any help would be much appreciated. In Urgent need.

TIA

3

Hi @geetanshjindal,

IP port of your monitoring (in fact every IP port) is written in 16 bits, so max value is 65535. Error you get is HTTP tracker error here - OS doe no allow to exceed addressing.

4

Had the same error with the same stack trace but the root cause was different.

The enrichment directory provided in parameter wasn’t valid / existing.

If that can help someone :wink:

5

Hi,

You should setup path to directory with enrichments, not to particular enrichment JSON file. Enrichment lists the directory and analyses all the JSON files it founds inside.