Hi, has anyone else experienced an issue with snowplow-rdb-loader timing out, starting today? It worked during a batch process yesterday, but every time I run it today it’s running over an hour without ever actually querying Redshift.
We are using the official Iglu repositories, might this have anything to do with the Igluctl update released today?
I have tried re-running snowplow-rdb-loader locally with the same config with the same result (it just runs without any output until I kill it).
I also tried running it with --dry-run with the same result. We are running version 0.14.0 which was working perfectly up until today. Did anything else change today?
Thanks in advance for any insight anyone can give!
No, this cannot be related to igluctl release. RDB Loader doesn’t use it directly and igluctl was published long time ago, just announced on discourse yesterday.
We don’t experience this problem with pipelines we manage and the only thing that strikes me in your post is a very old version of RDB Loader. It might have something to do with an outdated JDBC driver or AWS SDK.
One more potential issue is the consistency check. RDB Loader waits until S3 gets into a consistent state with its listKeys output and how long it waits depends on amout of shredded types and volume of atomic data. If by any chance you got too many shredded types, wait time can grow significantly. You can use --skip consistency_check to avoid it. However the formulae improved in latest versions, so updating RDB Loader could be a good idea anyway.
@anton - thank you so much! The consistency check seems to be exactly what was causing the timeout - after taking a look at our parallelism, it was set very high, creating a large number (tens of thousands) of individual shredded output files which added almost an hour waiting for S3 to become consistent (with the v0.4.0 algorithm).
Dialing down the consistency (to 100 in our case, which meant large individual shred files but all-in only a couple of thousand) brought the Redshift Load step down to the expected ~10-15minutes which was actually spent on query time and not just waiting.