Snowplow Java Tracker is not sending info to the S3

[JuanAndrade11]

Hi everyone, I’m trying to use the Snowplow Java tracker to send events from my backend to an s3, I’ve checked my collector’s URL and my schemas and they are ok (since I can send events with the Javascript tracker from the front-end). I’ve followed the Java Tracker documentation but nothing seems to work by now. My code is in Kotlin btw.

1. I created a subject like this

val subject: Subject = Subject.SubjectBuilder()
.userId(seeker.subjectId)
.build()

Where seeker.subjectId is a string.

2. This is the client I’m using.
   val client = OkHttpClient.Builder()
   .connectTimeout(5, TimeUnit.SECONDS)
   .readTimeout(5, TimeUnit.SECONDS)
   .writeTimeout(5, TimeUnit.SECONDS)
   .build()

3. The adapter
   val adapter: HttpClientAdapter = OkHttpClientAdapter.builder()
   .url(collector)
   .httpClient(client)
   .build()

where ‘collector’ is my collector’s URL. also string.

4. This is my emitter
   val emitter: Emitter = BatchEmitter.builder()
   .httpClientAdapter(adapter)
   .bufferSize(1)
   .requestCallback(callback)
   .build()

5. This is the tracker
   val tracker: Tracker = Tracker.TrackerBuilder(emitter, “AF003”, “cf”)
   .subject(subject)
   .base64(false)
   .platform(DevicePlatform.ServerSideApp)
   .build();

6. I’m trying to use an Unstructured event with this code

tracker.track(Unstructured.builder()
.eventData(context)
.build());

where context is a mapof(field: data) with the schema using the SelfDescribingJson method like this.
val context = SelfDescribingJson(schema, eventMap)

Also I’m not getting anything from the callback logs in the emitter.
I’m using buffersize = 1 just for testing purposes.

Am I missing something? Thank you in advance.

[PaulBoocock]

Nothing looks glaringly incorrect here. Could you perhaps try the SimpleEmitter rather than BatchEmitter just to rule anything out there?

Also what version of the tracker are you using?

[JuanAndrade11]

Hi Paul, thanks for your answer, I’ll try the SimpleEmitter
The version I’m using is 0.10.1

[JuanAndrade11]

It finally worked with the SimpleEmitter! thank you. but Why wasn’t it working with the batchEmitter? Do you have an idea?

[PaulBoocock]

If this is a simple console application, then the BatchEmitter works asynchronously, meaning the app might quit before the event ever sends. The SimpleEmitter blocks until the event sends which perhaps explains why it is working now.

One thing to try with the BatchEmitter is to close it when you’re done (Only do this once though, you shouldn’t do this everytime you send an event, it should be called on application shutdown).

emitter.close();

[JuanAndrade11]

Oh, It makes sense, Thank you so much for your help!

[JuanAndrade11]

Hello again, I really need to use the BatchEmitter, so I was going to try what you suggested but the method close() does not exist (?)

[mike]

It looks like it exists - are you getting an error when calling the method?

snowplow/snowplow-java-tracker/blob/master/src/main/java/com/snowplowanalytics/snowplow/tracker/emitter/BatchEmitter.java#L257

[JuanAndrade11]

Thanks Mike for the response! and…
Yes, I can’t call it. Could you provide me an example of how to use it?

[PaulBoocock]

There’s an example of using the Batch Emitter in the Java Trackers repository here: snowplow-java-tracker/Main.java at d60f3f75b9efbbca7310876f7e9f17d394d4540a · snowplow/snowplow-java-tracker · GitHub

[JuanAndrade11]

Thanks. I tried to use it that way but It’s still not sending events. the SimpleEmitter is ‘working’ but is increasing my CPU usage a lot. BatchEmitter is not working for me

[JuanAndrade11]

Hello again,

We did a lot of debugging today to understand why the BatchEmitter is not working for us, we got to this point where the code breaks:

This is the OkHttpClientAdapter.java. We are using Kotlin. What could be the issue? Is this Kotlin compatible? In our okhttp3 package, there is no create method with that input types. (string, media type) is always the other way around (media type, string).

This is our payload (We are using a BufferSize of 2 for testing purposes)

{“schema”:“iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4”,“data”:[{“e”:“ue”,“ue_px”:“eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS50b3JyZWxhYnMvbWF0Y2hfZGlzdHJpYnV0ZWQvanNvbnNjaGVtYS8yLTAtMCIsImRhdGEiOiJ7XCJtZXRob2RcIjpcImVtYWlsXCIsXCJwcm92aWRlclwiOlwicGF0aGZpbmRlclwiLFwibW9kZWxcIjpcInJlYWxpc3RpY1wiLFwib3Bwb3J0dW5pdHlfcmVmXCI6XCI1XCIsXCJvcHBvcnR1bml0eV9leHRlcm5hbFwiOnRydWUsXCJvcHBvcnR1bml0eV9tYXJrZXRcIjpcInJlbW90ZVwiLFwib3Bwb3J0dW5pdHlfcmVtb3RlXCI6dHJ1ZSxcIm9wcG9ydHVuaXR5X3R5cGVcIjpcImZ1bGwtdGltZS1lbXBsb3ltZW50XCIsXCJvcHBvcnR1bml0eV9jcmF3bGVkXCI6ZmFsc2UsXCJvcHBvcnR1bml0eV9sb2NhdGlvbnNcIjpcIm5vbmVcIixcInN1YmplY3RfaWRcIjpcIjFcIn0ifX0=”,“eid”:“37c4109b-268f-459c-bf79-95e88172d68c”,“dtm”:“1626824130619”,“p”:“srv”,“aid”:“PF”,“tna”:“MATCHES”,“tv”:“java-0.10.1”,“stm”:“1626824148009”},{“e”:“ue”,“ue_px”:“eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS50b3JyZWxhYnMvbWF0Y2hfZGlzdHJpYnV0ZWQvanNvbnNjaGVtYS8yLTAtMCIsImRhdGEiOiJ7XCJtZXRob2RcIjpcImVtYWlsXCIsXCJwcm92aWRlclwiOlwicGF0aGZpbmRlclwiLFwibW9kZWxcIjpcInJlYWxpc3RpY1wiLFwib3Bwb3J0dW5pdHlfcmVmXCI6XCI1XCIsXCJvcHBvcnR1bml0eV9leHRlcm5hbFwiOnRydWUsXCJvcHBvcnR1bml0eV9tYXJrZXRcIjpcInJlbW90ZVwiLFwib3Bwb3J0dW5pdHlfcmVtb3RlXCI6dHJ1ZSxcIm9wcG9ydHVuaXR5X3R5cGVcIjpcImZ1bGwtdGltZS1lbXBsb3ltZW50XCIsXCJvcHBvcnR1bml0eV9jcmF3bGVkXCI6ZmFsc2UsXCJvcHBvcnR1bml0eV9sb2NhdGlvbnNcIjpcIm5vbmVcIixcInN1YmplY3RfaWRcIjpcIjFcIn0ifX0=”,“eid”:“37c4109b-268f-459c-bf79-95e88172d68c”,“dtm”:“1626824130619”,“p”:“srv”,“aid”:“PF”,“tna”:“MATCHES”,“tv”:“java-0.10.1”,“stm”:“1626824148009”}]}

Do you have any ideas to make this work? We are using too much cpu with the SimpleEmitter but we really want to use Snowplow to track our events.

Please let me know if you need any additional context information. Have a nice day.

[PaulBoocock]

Thats rather interesting. This looks like a Kotlin incompatibility and that RequestBody.create is a deprecated method:

square/okhttp/blob/master/okhttp/src/main/kotlin/okhttp3/RequestBody.kt#L191-L199

Leave this with me and I’ll look to get a change in which moves away from this deprecated method.

[JuanAndrade11]

Thank you so much Paul.

[PaulBoocock]

Hey @JuanAndrade11

So digging a little deeper, it looks like we’re actually already using the new companion object method as the OkHttp docs suggest. The line which you’re seeing an error on is what OkHttp marks as the current API as far as I can see.

Could you comfirm the version of OkHttp you’re using? This switch of API was introduced in OkHttp3 v4, you’ll need to be on at least v4 but I assume it’d work with OkHttp3 v4.

[JuanAndrade11]

Thanks for your help Paul,
We updated our okhttp3 to version 4, that fixed the issue.
But, we still get an error in the okHttpClientAdapter.java
I’ve been trying to fix the import myself but nothing works, What could be the issue now?

[PaulBoocock]

I think you’ve found a missing dependency there. Older versions of OkHttp3 used to include Apache HttpComponents HttpClient which would have pulled in this dependency.

When we upgraded to v4 of OkHttp3, it looks like this has been missed. We have an optional Apache Http support so it doesn’t fail in our builds. In reality we don’t need this at all any more so I’ve opened an issue on our GitHub repo to address this.

I think to fix it, you should be able to pull in HttpComponents HttpCore v4: https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore/4.4.14

[JuanAndrade11]

Thank you, Paul, it finally worked! We’ll be running some experiments to verify the improvements in performance now, but at least we can use the BatchEmitter.