Migrating from 0.8.1 to 0.19.1

For engineers AWS real-time pipeline
1

Hey there,

I’m trying to dockerize the latest version using your supplied containers.

I’ve updated the configuration with what I think it should look like, but I’m getting the following error when trying to run it.

I would love some pointers as to what I need to change.

The error I’m getting:

docker run -v ${PWD}/config:/snowplow/config enrich --config /snowplow/config/config.hocon --resolver file:/snowplow/config/iglu.json --enrichments file:/snowplow/config/enrichments/ --force-cached-files-download
Exception in thread "main" java.lang.NullPointerException
        at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:190)
        at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:186)
        at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:246)
        at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
        at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:186)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$8.apply(KinesisEnrich.scala:210)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$8.apply(KinesisEnrich.scala:207)
        at scala.Option.map(Option.scala:146)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.extractEnrichmentConfigs(KinesisEnrich.scala:207)
        at com.snowplowanalytics.snowplow.enrich.stream.Enrich$class.parseEnrichmentRegistry(Enrich.scala:175)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.parseEnrichmentRegistry(KinesisEnrich.scala:53)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2$$anonfun$apply$3.apply(KinesisEnrich.scala:67)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2$$anonfun$apply$3.apply(KinesisEnrich.scala:66)
        at scalaz.Validation$class.flatMap(Validation.scala:139)
        at scalaz.Success.flatMap(Validation.scala:345)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2.apply(KinesisEnrich.scala:66)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2.apply(KinesisEnrich.scala:62)
        at scalaz.Validation$class.flatMap(Validation.scala:139)
        at scalaz.Success.flatMap(Validation.scala:345)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2.apply(KinesisEnrich.scala:62)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2.apply(KinesisEnrich.scala:60)
        at scalaz.Validation$class.flatMap(Validation.scala:139)
        at scalaz.Success.flatMap(Validation.scala:345)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.main(KinesisEnrich.scala:60)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich.main(KinesisEnrich.scala)

My config:

enrich {

  streams {

    in {
      # Stream/topic where the raw events to be enriched are located
      raw = "rawdata"
      maxRecords =  10000
    }

    out {
      enriched = "enricheddata"
      bad = "rawdatabad"

      partitionKey = event_fingerprint
    }

    sourceSink {
      enabled =  kinesis

      region = "eu-west-1"

      aws {
        accessKey = iam
        secretKey = iam
      }

      maxRecords = 10000

      initialPosition = TRIM_HORIZON

      backoffPolicy {
        minBackoff = 1000
        maxBackoff = 60000
      }

      retries = 0

    }

    buffer {
      byteLimit = 4500000
      recordLimit = 500
      timeLimit = 60000
    }

    appName = "snowplow_enrich_progress"
  }

}
3

Hey @joewarrenmeeks IIRC some version needed the “optional” monitoring section, but it could be empty. Give it a go.

HTH

4

Hey Joe Warren Meeks,

Were you able to solve the issue. I am also blocked at same step either through jar run or docker run I’m getting Nullpointer exception.

If possible could you help me to resolve this

5

Hi @Sudhir_Batchu - would you mind posting the version of the app you are having the issue with, the command you are using the launch the container and the configuration you are using with the app?

6

Hi Josh,

Sorry for the delay. I’m able to run the enrichments. I had an issue with iglu central and schemas i resolved those then it started working.

Thanks for the response.

1 Like