Snowplow Elastic Search Loader Error

Hi Team, I faced error in Elastic search loader where the process gave up on sending records to elastic search. I just wanted to confirm did we loose any data here or it still remains buffered in kinesis stream. Here the logs -

2021-11-05T13:55:56.495+05:30	[scala-execution-context-global-24] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Storage threw an unexpected exception, after 0 retries. Next attempt in 930730 nanoseconds

2021-11-05T13:55:56.495+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:55:56.495+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:55:56.495+05:30	<html>

2021-11-05T13:55:56.495+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:55:56.495+05:30	<body>

2021-11-05T13:55:56.495+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:55:56.495+05:30	</body>

2021-11-05T13:56:00.661+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:06.501+05:30	[scala-execution-context-global-24] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Storage threw an unexpected exception, after 1 retries. Next attempt in 1014670 nanoseconds

2021-11-05T13:56:06.501+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:56:06.501+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:56:06.501+05:30	<html>

2021-11-05T13:56:06.501+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:56:06.501+05:30	<body>

2021-11-05T13:56:06.501+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:56:06.501+05:30	</body>

2021-11-05T13:56:10.661+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:14.255+05:30	[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Current stream shard assignments: shardId-000000000000

2021-11-05T13:56:16.506+05:30	[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Sleeping ...

2021-11-05T13:56:16.506+05:30	[scala-execution-context-global-24] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Storage threw an unexpected exception, after 2 retries. Next attempt in 5291946 nanoseconds

2021-11-05T13:56:16.506+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:56:16.506+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:56:16.506+05:30	<html>

2021-11-05T13:56:16.506+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:56:16.506+05:30	<body>

2021-11-05T13:56:16.506+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:56:16.506+05:30	</body>

2021-11-05T13:56:20.661+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:26.519+05:30	[scala-execution-context-global-27] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Storage threw an unexpected exception, after 3 retries. Next attempt in 11543183 nanoseconds

2021-11-05T13:56:26.519+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:56:26.520+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:56:26.520+05:30	<html>

2021-11-05T13:56:26.520+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:56:26.520+05:30	<body>

2021-11-05T13:56:26.520+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:56:26.520+05:30	</body>

2021-11-05T13:56:30.661+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:36.524+05:30	[scala-execution-context-global-27] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Storage threw an unexpected exception, after 4 retries. Next attempt in 5181626 nanoseconds

2021-11-05T13:56:36.525+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:56:36.525+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:56:36.525+05:30	<html>

2021-11-05T13:56:36.525+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:56:36.525+05:30	<body>

2021-11-05T13:56:36.525+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:56:36.525+05:30	</body>

2021-11-05T13:56:40.661+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:46.529+05:30	[scala-execution-context-global-27] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Storage threw an unexpected exception. Giving up

2021-11-05T13:56:46.529+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:56:46.529+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:56:46.529+05:30	<html>

2021-11-05T13:56:46.529+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:56:46.530+05:30	<body>

2021-11-05T13:56:46.530+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:56:46.530+05:30	</body>

2021-11-05T13:56:46.530+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:46.531+05:30	[RecordProcessor-0000] ERROR com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - Shutting down application as unable to connect to Elasticsearch for over 10000 ms

2021-11-05T13:56:46.531+05:30	com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.lang.NullPointerException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:61) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:389) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.base/java.lang.Thread.run(Thread.java:829)

2021-11-05T13:56:46.531+05:30	Caused by: java.lang.NullPointerException at scala.io.Codec.decoder(Codec.scala:65) at scala.io.BufferedSource.reader(BufferedSource.scala:26) at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:27) at scala.io.BufferedSource.charReader$lzycompute(BufferedSource.scala:37) at scala.io.BufferedSource.charReader(BufferedSource.scala:35) at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:64) at scala.io.BufferedSource.mkString(BufferedSource.scala:93) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.$anonfun$fromResponse$1(ElasticsearchJavaRestClient.scala:44) at scala.Option.map(Option.scala:230) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient.fromResponse(ElasticsearchJavaRestClient.scala:35) at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:60) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:629) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:419) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381) ... 16 more Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://search-rr-snowplow-elk-dev-6uwjp2didil3hbsuamgoxs34g4.us-east-1.es.amazonaws.com:443], URI [/_bulk], status line [HTTP/1.1 504 Gateway Time-out]

2021-11-05T13:56:46.531+05:30	<html>

2021-11-05T13:56:46.531+05:30	<head><title>504 Gateway Time-out</title></head>

2021-11-05T13:56:46.531+05:30	<body>

2021-11-05T13:56:46.531+05:30	<center><h1>504 Gateway Time-out</h1></center>

2021-11-05T13:56:46.531+05:30	</body>

2021-11-05T13:56:46.531+05:30	</html> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ... 16 more

2021-11-05T13:56:50.661+05:30	[RecordProcessor-0000] INFO com.snowplowanalytics.stream.loader.clients.ElasticsearchBulkSender - BulkSender force shutdown

2021-11-05T18:16:50.661+05:30	orker.Worker - Sleeping ...

@ihor

The force shutdown by ES loader should present any checkpointing by the KCL so you should be fine in terms of data loss whenever the loader is able to connect to Elasticsearch again successfully.