Final step at shred/load failing with no error

For engineers
1

I’m at final step of shred and load.

shredder is running/completing successfully(shred details) but

  • Not creating shredding_complete.json file
  • Also not sending msg to SQS.

And then when RDB loader is started, it starts with log

INFO RDB Loader 2.0.0 has started. Listening

and then nothing.

Shredder has full access to kinesis to write msgs. and I find no error across any logs in EMR where shredder isn’t able to write msgs.

1 Like
2

Hi @pramod.niralakeri ,

When shredding is complete, shredder tries to write a message to SQS ("queueName": "snowplow-shredder-loader", in your case) for the loader, and then it writes shredding_complete.json to S3.

In your case I suspect that shredder can’t write to SQS.

To check that a message has been written to SQS you can use the Monitoring tab in SQS UI for your queue, you will see Number Of Messages Sent.

3

Hi @BenB thanks for reply.

I’ve tried all the steps you suggested. I see no msg reached to SQS

Shredder has full access to kinesis to write msgs. and I find no error across any logs in EMR where shredder isn’t able to write msgs.
4

Shredder needs to talk to SQS, not Kinesis

5

Sorry I meant, SQS only.

in-short shredder has full access to SQS and kinesis

6

I’m having the same issue.

As the shredder succeeded in writing shredded data to the S3, I’m sure the shredder have enough privilege at least for the S3.