Stream Loader Read Timed Out Error

For engineers
1

Hey,

I have the stream loader 1.1.0 running as an app engine service. The instances are autoscaled so there was no downtime when the following error occurred.

Error:

com.google.cloud.bigquery.BigQueryException: Read timed out

at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate ( com/google.cloud.bigquery.spi.v2/HttpBigQueryRpc.java:115 )
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll ( com/google.cloud.bigquery.spi.v2/HttpBigQueryRpc.java:494 )
at com.google.cloud.bigquery.BigQueryImpl.insertAll ( com/google.cloud.bigquery/BigQueryImpl.java:1065 )
at com.snowplowanalytics.snowplow.storage.bigquery.streamloader.Bigquery$.$anonfun$mkInsert$2 ( com/snowplowanalytics.snowplow.storage.bigquery.streamloader/Bigquery.scala:65 )

Error indicated a failure when inserting data, does the occurrence of this exception indicate a loss of data?

Thank you!

2

Hi @siv if you are often seeing read time outs like that, you could try changing the bqWriteRequestTimeout in your hocon file.

There will not be data loss when this exception happens. The loader is designed to ack the incoming pubsub message only after the event has been successfully written to BigQuery. We try to design all Snowplow components on this same principle, so that data should never be lost upon unexpected exceptions.

1 Like
3

I just re-read my comment and realised it could sound a bit odd: the error message is about a read timeout, but I suggested changing a write timeout setting. But I still think this is correct – both error message and config setting refer to inserting data into BigQuery, so they are connected.

1 Like
4

Thank you for replying @istreeter! I will increase the time out and give it a try

5

Hey @istreeter ,

I was just wondering if there would be duplicate records in the bq table if the bqloader times out while writing?

Thank you!

6

Hi @siv I think it is very likely, but I don’t know for sure. If you find out, please can you update this thread?

1 Like
7

Hey @istreeter,

BQ Stream Loader faced another issue today, just wanted to know what was causing this and any suggestion on avoiding it.

Error:

java.lang.InterruptedException:

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await ( java/util.concurrent.locks/AbstractQueuedSynchronizer.java:1638 )
at java.util.concurrent.LinkedBlockingQueue.take ( java/util.concurrent/LinkedBlockingQueue.java:435 )
at com.permutive.pubsub.consumer.grpc.internal.PubsubSubscriber$.$anonfun$takeNextElement$4 ( com/permutive.pubsub.consumer.grpc.internal/PubsubSubscriber.scala:76 )
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop ( cats/effect.internals/IORunLoop.scala:108 )
at cats.effect.internals.IORunLoop$.restartCancelable ( cats/effect.internals/IORunLoop.scala:51 )
at cats.effect.internals.IOBracket$BracketStart.run ( cats/effect.internals/IOBracket.scala:100 )
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop ( cats/effect.internals/Trampoline.scala:67 )
at cats.effect.internals.Trampoline.startLoop ( cats/effect.internals/Trampoline.scala:35 )
at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop ( cats/effect.internals/TrampolineEC.scala:90 )
at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1 ( cats/effect.internals/TrampolineEC.scala:90 )
at scala.runtime.java8.JFunction0$mcV$sp.apply ( scala/runtime.java8/JFunction0$mcV$sp.scala:18 )
at scala.concurrent.BlockContext$.withBlockContext ( scala/concurrent/BlockContext.scala:94 )
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop ( cats/effect.internals/TrampolineEC.scala:90 )
at cats.effect.internals.Trampoline.execute ( cats/effect.internals/Trampoline.scala:43 )
at cats.effect.internals.TrampolineEC.execute ( cats/effect.internals/TrampolineEC.scala:42 )
at cats.effect.internals.IOBracket$BracketStart.apply ( cats/effect.internals/IOBracket.scala:80 )
at cats.effect.internals.IOBracket$BracketStart.apply ( cats/effect.internals/IOBracket.scala:58 )
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop ( cats/effect.internals/IORunLoop.scala:192 )
at cats.effect.internals.IORunLoop$RestartCallback.signal ( cats/effect.internals/IORunLoop.scala:480 )
at cats.effect.internals.IORunLoop$RestartCallback.apply ( cats/effect.internals/IORunLoop.scala:501 )
at cats.effect.internals.IORunLoop$RestartCallback.apply ( cats/effect.internals/IORunLoop.scala:439 )
at cats.effect.internals.IOShift$Tick.run ( cats/effect.internals/IOShift.scala:36 )
at cats.effect.internals.PoolUtils$$anon$2$$anon$3.run ( cats/effect.internals/PoolUtils.scala:52 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( java/lang/Thread.java:833 )

I’ll update this thread for sure!

Thank you!

8

Hi @siv I think the exception you shared is not the root cause of the problem. It shows that a thread was interrupted, but the reason for the interruption is probably due to another exception in another thread. Is there anything else in the logs immediately before or after that exception was printed?

9

Hey @istreeter, the BQ read time out error had occurred right before that. I’m guessing this exception popped cause of that?