Error in collector

For engineers Collectors
1

We currently running snowplow-stream-collector-0.9.0. We are in the process of migrating to the new version. Data is being written to good KS and no bad records has been written. However, we are noticing an error message in the log file.

16:03:02.957 [pool-1-thread-1] ERROR c.s.s.c.s.sinks.KinesisSink - Record failed with error code [InternalFailure] and message [Internal service failure.]

We are running 10 threads. We are seeing this message in all the instances. The scala job seems to be running fine. The health of the system seems to be fine. Do we need to be concerned about this error?

2

How often are you seeing this error?

If this is occasional the Scala stream collector will retry sending failed records if errors are returned for a batch but if you’re consistently receiving this error on each retry then it would need further investigation.

3

We have been seeing this error consistently across all the nodes. Most of the transactions looks like it is flowing through. I am unable to confirm if the retry worked or not.

I will provide the other details once I have it.

Thanks
Sarathy