GCS Dataflow Job Error


I have a GCS Loader running as a Dataflow job and faced the following issue.

The job says its streaming but the backlog in the pubsub is increasing and there is no data being written into the bucket.

For the time being I have restarted the job.

The following are the job parameters


Is there a solution for this?

Thank you!

Hey @siv,

Could you tell me what kind of data you are sending to GCS Loader ? Also, could you give us full error stack trace if it is possible ?

1 Like

Hi @enes_aldemir,

I am using multiple GCS loaders actually. The following GCS Loader gave the above mentioned error:

  • GCS Loader to write enricher bad records to bucket
  • GCS Loader to write collector good records to bucket

This is all I can grab from the logs:

Thanks for the additional information @siv. Could you confirm that you have some other GCS Loader instances which works fine but this two is giving the mentioned error ?

All of them give errors, but these two have greater load comparatively, hence the error count is higher.