Enrich PubSub Bad Gateway Error

For engineers
1

Hey,

I have the enrich pubsub 2.0.5 running as an app engine service. I frequently see ‘io.grpc’ errors/exceptions in logs. The instances are still up after the exceptions thanks to autoscaling.

Error 1:

com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: http2 exception

at com.google.api.gax.rpc.ApiExceptionFactory.createException ( com/google.api.gax.rpc/ApiExceptionFactory.java:67 )
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create ( com/google.api.gax.grpc/GrpcApiExceptionFactory.java:72 )
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create ( com/google.api.gax.grpc/GrpcApiExceptionFactory.java:60 )
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure ( com/google.api.gax.grpc/GrpcExceptionCallable.java:97 )
at com.google.api.core.ApiFutures$1.onFailure ( com/google.api.core/ApiFutures.java:68 )
at com.google.common.util.concurrent.Futures$CallbackListener.run ( com/google.common.util.concurrent/Futures.java:1050 )
at com.google.common.util.concurrent.DirectExecutor.execute ( com/google.common.util.concurrent/DirectExecutor.java:30 )
at com.google.common.util.concurrent.AbstractFuture.executeListener ( com/google.common.util.concurrent/AbstractFuture.java:1176 )
at com.google.common.util.concurrent.AbstractFuture.complete ( com/google.common.util.concurrent/AbstractFuture.java:969 )
at com.google.common.util.concurrent.AbstractFuture.setException ( com/google.common.util.concurrent/AbstractFuture.java:760 )
at io.grpc.stub.ClientCalls$GrpcFuture.setException ( io/grpc.stub/ClientCalls.java:545 )
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose ( io/grpc.stub/ClientCalls.java:515 )
at io.grpc.internal.ClientCallImpl.closeObserver ( io/grpc.internal/ClientCallImpl.java:426 )
at io.grpc.internal.ClientCallImpl.access$500 ( io/grpc.internal/ClientCallImpl.java:66 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close ( io/grpc.internal/ClientCallImpl.java:689 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900 ( io/grpc.internal/ClientCallImpl.java:577 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal ( io/grpc.internal/ClientCallImpl.java:751 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext ( io/grpc.internal/ClientCallImpl.java:740 )
at io.grpc.internal.ContextRunnable.run ( io/grpc.internal/ContextRunnable.java:37 )
at io.grpc.internal.SerializingExecutor.run ( io/grpc.internal/SerializingExecutor.java:123 )
at java.util.concurrent.Executors$RunnableAdapter.call ( java/util.concurrent/Executors.java:539 )
at java.util.concurrent.FutureTask.run ( java/util.concurrent/FutureTask.java:264 )
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run ( java/util.concurrent/ScheduledThreadPoolExecutor.java:304 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( java/lang/Thread.java:833 )
Caused by: io.grpc.StatusRuntimeException

at io.grpc.Status.asRuntimeException ( Status.java:533 )
Caused by: io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException

at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError ( Http2Exception.java:167 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel ( DefaultHttp2RemoteFlowController.java:481 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed ( DefaultHttp2RemoteFlowController.java:105 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed ( DefaultHttp2Connection.java:356 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams ( DefaultHttp2Connection.java:1000 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams$2.process ( DefaultHttp2Connection.java:961 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.decrementPendingIterations ( DefaultHttp2Connection.java:1022 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.forEachActiveStream ( DefaultHttp2Connection.java:977 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.forEachActiveStream ( DefaultHttp2Connection.java:208 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.closeStreamsGreaterThanLastKnownStreamId ( DefaultHttp2Connection.java:280 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.goAwaySent ( DefaultHttp2Connection.java:274 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.goAway ( Http2ConnectionHandler.java:823 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.goAway ( Http2ConnectionHandler.java:880 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onConnectionError ( Http2ConnectionHandler.java:672 )
at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onConnectionError ( NettyClientHandler.java:495 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.processRstStreamWriteResult ( Http2ConnectionHandler.java:888 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.access$1000 ( Http2ConnectionHandler.java:66 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$4.operationComplete ( Http2ConnectionHandler.java:809 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$4.operationComplete ( Http2ConnectionHandler.java:806 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0 ( DefaultPromise.java:577 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow ( DefaultPromise.java:551 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners ( DefaultPromise.java:490 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0 ( DefaultPromise.java:615 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0 ( DefaultPromise.java:608 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure ( DefaultPromise.java:117 )
at io.grpc.netty.shaded.io.netty.util.internal.PromiseNotificationUtil.tryFailure ( PromiseNotificationUtil.java:64 )
at io.grpc.netty.shaded.io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete ( DelegatingChannelPromiseNotifier.java:57 )
at io.grpc.netty.shaded.io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete ( DelegatingChannelPromiseNotifier.java:31 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0 ( DefaultPromise.java:577 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow ( DefaultPromise.java:551 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners ( DefaultPromise.java:490 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0 ( DefaultPromise.java:615 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0 ( DefaultPromise.java:608 )
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure ( DefaultPromise.java:117 )
at io.grpc.netty.shaded.io.netty.util.internal.PromiseNotificationUtil.tryFailure ( PromiseNotificationUtil.java:64 )
at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.safeFail ( ChannelOutboundBuffer.java:723 )
at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.remove0 ( ChannelOutboundBuffer.java:308 )
at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.failFlushed ( ChannelOutboundBuffer.java:660 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close ( AbstractChannel.java:740 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0 ( AbstractChannel.java:942 )
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0 ( AbstractEpollChannel.java:519 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush ( AbstractChannel.java:897 )
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.flush ( DefaultChannelPipeline.java:1372 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0 ( AbstractChannelHandlerContext.java:750 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush ( AbstractChannelHandlerContext.java:742 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.flush ( AbstractChannelHandlerContext.java:728 )
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.forceFlush ( SslHandler.java:2040 )
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrapAndFlush ( SslHandler.java:796 )
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.flush ( SslHandler.java:773 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0 ( AbstractChannelHandlerContext.java:750 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush ( AbstractChannelHandlerContext.java:742 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.flush ( AbstractChannelHandlerContext.java:728 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onError ( Http2ConnectionHandler.java:643 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder$FlowControlledData.error ( DefaultHttp2ConnectionEncoder.java:467 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.writeError ( DefaultHttp2RemoteFlowController.java:534 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel ( DefaultHttp2RemoteFlowController.java:484 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed ( DefaultHttp2RemoteFlowController.java:105 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed ( DefaultHttp2Connection.java:356 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams ( DefaultHttp2Connection.java:1000 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams$2.process ( DefaultHttp2Connection.java:961 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.decrementPendingIterations ( DefaultHttp2Connection.java:1022 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.forEachActiveStream ( DefaultHttp2Connection.java:977 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.forEachActiveStream ( DefaultHttp2Connection.java:208 )
at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.goingAway ( NettyClientHandler.java:784 )
at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$300 ( NettyClientHandler.java:91 )
at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$3.onGoAwayReceived ( NettyClientHandler.java:275 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.goAwayReceived ( DefaultHttp2Connection.java:236 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.onGoAwayRead0 ( DefaultHttp2ConnectionDecoder.java:218 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onGoAwayRead ( DefaultHttp2ConnectionDecoder.java:551 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onGoAwayRead ( Http2InboundFrameLogger.java:119 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readGoAwayFrame ( DefaultHttp2FrameReader.java:591 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState ( DefaultHttp2FrameReader.java:272 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame ( DefaultHttp2FrameReader.java:160 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame ( Http2InboundFrameLogger.java:41 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame ( DefaultHttp2ConnectionDecoder.java:174 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode ( Http2ConnectionHandler.java:378 )
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode ( Http2ConnectionHandler.java:438 )
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection ( ByteToMessageDecoder.java:498 )
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode ( ByteToMessageDecoder.java:437 )
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead ( ByteToMessageDecoder.java:276 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead ( AbstractChannelHandlerContext.java:379 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead ( AbstractChannelHandlerContext.java:365 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead ( AbstractChannelHandlerContext.java:357 )
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap ( SslHandler.java:1486 )
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible ( SslHandler.java:1235 )
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode ( SslHandler.java:1282 )
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection ( ByteToMessageDecoder.java:498 )
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode ( ByteToMessageDecoder.java:437 )
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead ( ByteToMessageDecoder.java:276 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead ( AbstractChannelHandlerContext.java:379 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead ( AbstractChannelHandlerContext.java:365 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead ( AbstractChannelHandlerContext.java:357 )
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead ( DefaultChannelPipeline.java:1410 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead ( AbstractChannelHandlerContext.java:379 )
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead ( AbstractChannelHandlerContext.java:365 )
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead ( DefaultChannelPipeline.java:919 )
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady ( AbstractEpollStreamChannel.java:792 )
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady ( EpollEventLoop.java:475 )
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run ( EpollEventLoop.java:378 )
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run ( SingleThreadEventExecutor.java:989 )
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run ( ThreadExecutorMap.java:74 )
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run ( FastThreadLocalRunnable.java:30 )

Error 2:

com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: 502:Bad Gateway

at com.google.api.gax.rpc.ApiExceptionFactory.createException ( com/google.api.gax.rpc/ApiExceptionFactory.java:69 )
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create ( com/google.api.gax.grpc/GrpcApiExceptionFactory.java:72 )
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create ( com/google.api.gax.grpc/GrpcApiExceptionFactory.java:60 )
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure ( com/google.api.gax.grpc/GrpcExceptionCallable.java:97 )
at com.google.api.core.ApiFutures$1.onFailure ( com/google.api.core/ApiFutures.java:68 )
at com.google.common.util.concurrent.Futures$CallbackListener.run ( com/google.common.util.concurrent/Futures.java:1050 )
at com.google.common.util.concurrent.DirectExecutor.execute ( com/google.common.util.concurrent/DirectExecutor.java:30 )
at com.google.common.util.concurrent.AbstractFuture.executeListener ( com/google.common.util.concurrent/AbstractFuture.java:1176 )
at com.google.common.util.concurrent.AbstractFuture.complete ( com/google.common.util.concurrent/AbstractFuture.java:969 )
at com.google.common.util.concurrent.AbstractFuture.setException ( com/google.common.util.concurrent/AbstractFuture.java:760 )
at io.grpc.stub.ClientCalls$GrpcFuture.setException ( io/grpc.stub/ClientCalls.java:545 )
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose ( io/grpc.stub/ClientCalls.java:515 )
at io.grpc.internal.ClientCallImpl.closeObserver ( io/grpc.internal/ClientCallImpl.java:426 )
at io.grpc.internal.ClientCallImpl.access$500 ( io/grpc.internal/ClientCallImpl.java:66 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close ( io/grpc.internal/ClientCallImpl.java:689 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900 ( io/grpc.internal/ClientCallImpl.java:577 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal ( io/grpc.internal/ClientCallImpl.java:751 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext ( io/grpc.internal/ClientCallImpl.java:740 )
at io.grpc.internal.ContextRunnable.run ( io/grpc.internal/ContextRunnable.java:37 )
at io.grpc.internal.SerializingExecutor.run ( io/grpc.internal/SerializingExecutor.java:123 )
at java.util.concurrent.Executors$RunnableAdapter.call ( java/util.concurrent/Executors.java:539 )
at java.util.concurrent.FutureTask.run ( java/util.concurrent/FutureTask.java:264 )
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run ( java/util.concurrent/ScheduledThreadPoolExecutor.java:304 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( java/lang/Thread.java:833 )
Caused by: io.grpc.StatusRuntimeException

at io.grpc.Status.asRuntimeException ( Status.java:533 )

On looking up the error I found out that its related to acknowledging pubsub messages. I had one question though, will the enricher process the pulled messages and send them out before acknowledging them or does it enrich the messages only after acknowledging?

Thank you!

2

As far as I know messages from the enricher are only acked after the data has either been published to the good or the bad sink.

2 Likes
3

So that would mean that duplicates would be sent to bq loader if there is an issue with the enricher acknowledging the messages as pub sub messages won’t be deleted unless they are acknowledged. Am I correct?

4

Yes - if there was some issue in acknowledging messages (but they were being processed correctly to their destination) you will likely end up with at-least-once semantics where you could end up with duplicate events. If the message wasn’t acknowledged within the deadline then PubSub will try and redeliver it.

2 Likes
5

Hi @mike,

The enricher just gave a new exception, still trying to figure this one out, do you know what this means and what could have caused it?

Error:

com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Policy checks are unavailable.

at com.google.api.gax.rpc.ApiExceptionFactory.createException ( com/google.api.gax.rpc/ApiExceptionFactory.java:69 )
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create ( com/google.api.gax.grpc/GrpcApiExceptionFactory.java:72 )
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create ( com/google.api.gax.grpc/GrpcApiExceptionFactory.java:60 )
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure ( com/google.api.gax.grpc/GrpcExceptionCallable.java:97 )
at com.google.api.core.ApiFutures$1.onFailure ( com/google.api.core/ApiFutures.java:68 )
at com.google.common.util.concurrent.Futures$CallbackListener.run ( com/google.common.util.concurrent/Futures.java:1050 )
at com.google.common.util.concurrent.DirectExecutor.execute ( com/google.common.util.concurrent/DirectExecutor.java:30 )
at com.google.common.util.concurrent.AbstractFuture.executeListener ( com/google.common.util.concurrent/AbstractFuture.java:1176 )
at com.google.common.util.concurrent.AbstractFuture.complete ( com/google.common.util.concurrent/AbstractFuture.java:969 )
at com.google.common.util.concurrent.AbstractFuture.setException ( com/google.common.util.concurrent/AbstractFuture.java:760 )
at io.grpc.stub.ClientCalls$GrpcFuture.setException ( io/grpc.stub/ClientCalls.java:545 )
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose ( io/grpc.stub/ClientCalls.java:515 )
at io.grpc.internal.ClientCallImpl.closeObserver ( io/grpc.internal/ClientCallImpl.java:426 )
at io.grpc.internal.ClientCallImpl.access$500 ( io/grpc.internal/ClientCallImpl.java:66 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close ( io/grpc.internal/ClientCallImpl.java:689 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900 ( io/grpc.internal/ClientCallImpl.java:577 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal ( io/grpc.internal/ClientCallImpl.java:751 )
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext ( io/grpc.internal/ClientCallImpl.java:740 )
at io.grpc.internal.ContextRunnable.run ( io/grpc.internal/ContextRunnable.java:37 )
at io.grpc.internal.SerializingExecutor.run ( io/grpc.internal/SerializingExecutor.java:123 )
at java.util.concurrent.Executors$RunnableAdapter.call ( java/util.concurrent/Executors.java:539 )
at java.util.concurrent.FutureTask.run ( java/util.concurrent/FutureTask.java:264 )
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run ( java/util.concurrent/ScheduledThreadPoolExecutor.java:304 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( java/lang/Thread.java:833 )
Caused by: io.grpc.StatusRuntimeException

at io.grpc.Status.asRuntimeException ( Status.java:533 )
6

Not too sure on this one - generally internal 5xx GCP errors tend to be region or feature related. You might need to reach out to GCP support for more detail on that one I’m afraid.

1 Like
7

Hey @mike

I did reach out to GCP and this is the feedback I received. Posting it here:

I understand that you are using terraform to deploy your app on App Engine,
built on top of Snowplow components (enricher, bqloader, mutator and repeater). 
Also, you previously used Dataflow for the same app and you did not experience these errors. 
Please correct me if I misunderstood.

After further investigation, I was able to detect that this issue lies solely between GRPC and 
SnowPlow, which are third party software and not subjected to SLO by Google Cloud Platform 
Support. It was also confirmed on the official webpage for Snowplow. Nevertheless, 
I would be happy to help you on a best effort basis and point you towards the right direction.

The error is present in this thread for which the Snowplow team is yet to find the root cause, 
but there’s a recommendation on the last message to add “--restart=always” in your Dockerfile. 
Also, I saw that the errors are present in the versions 2.0.2 and 2.0.5 of Enrich PubSub. 
It could be a good try to downgrade to 2.0.3 or 2.0.4 and to check if the errors are still present.

Hopefully this information can be of help to you.

Would you recommend downgrading?

Thank you!

8

This is kind of ambiguous. Snowplow won’t be able to do much about 5xx errors (other than gracefully retry) so I think GCP should be in a position to give a bit more of a detailed analysis than this if they have pinpointed the error.

2 Likes
9

Google’s answer is a shame.

All the errors that you showed are gRPC errors that happen in a daemon thread on the JVM so unfortunately we can’t catch them. Does the JVM crash when they happen ?

I’m afraid there isn’t much we can do on our side.

2 Likes
10

Thank you @mike and @BenB,

I haven’t checked if the jvm crashed or not, will check on that.

I’ll also try to downgrade and see if i face these errors.

1 Like