Stream-enrich build - Invalid or corrupt jar file

For engineers Troubleshooting
1

Hi everyone

I’m having some trouble building the stream-enrich component from source. The build completes successfully but when running the binary I get:

Error: Invalid or corrupt jar file

I don’t think any of the normal explanations for this apply (mismatched zip versions, over 65k files). I get the same thing when running in the java -jar xxx com.snowplow.... format too so I don’t think the manifest is corrupt.

Has anyone else had the same issue? V grateful for any pointers!

2

Hey…
do you mind to provide more information how you build your snowplow stream enrich, like what branch, what java version, what sbt version etc ?
i just buid mine last night, it works fine with sbt 0.13 and it works fine,
here’s my step:

  • i set my java to java 1.7
  • clone the github.com/snowplow/snowplow on master branch
  • go to snowplow/3-enrich/stream-enrich
  • and run sbt assembly the jar file will be placed at
    target/scala-2.10/snowplow-stream-enrich-{{the snowplow version that you build}}

and just run it

1 Like
3

Strange - I am building master@278e99ce with a couple of tweaks, specifically upgrading the AWS SDK and KCL to the latest versions, and updating one import (com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason in kinesis/sources/KinesisSource.scala) to reflect a change in the new SDK version.

If I revert those changes the build works fine?!

I am building with java 1.7.0_80 and sbt 0.13.0.

@ChocoPowwwa could you make the same changes and see if you get the same results? I was using AWS SDK 1.11.73 and KCL 1.7.2.

5

UPDATE: Looks like it could be the file size limit after all - after a lot of trial and error, the total file for the stream-enrich jar jumps to 65561 with version 1.10.17 of the AWS SDK vs 65525 with version 1.10.16 (the ZIP limit is 65535).

If I unzip the jar and run java -cp extract_dir com.snowplowanalytics.....KinesisEnrichApp it works fine, which would corroborate the zip format theory.

That said, I’ve tried building with Java 8, which supposedly resolves the file limit issue, with no luck.

So at this point I’m a bit stumped. For now I am unzipping the jar in my Dockerfile and using the java -cp execution format. That’s hardly ideal as it results in a gargantuan image, but it’s all I got right now.

Any other ideas?!