Hi there! I’m running RDB loader in kubernetes. I have several rdb_loader apps and there is ~2 hours delay to load data to Redshift. I’ve tried to use more rdb_loader pods, but that did not help. Could you advice how is it possible to speed up SQS messages processing ?
1 Like
Hi @Serhii_Dimchenko, we find the bottleneck with loading into Redshift is almost always the size of the Redshift cluster, rather than any problem with the loader. If you want to load into Redshift more quickly, then pick a larger Redshift cluster.
Furthermore, I would actually discourage you from running several rdb_loader apps concurrently; it is much better to run a single loader for faster load times. I know this is counter-intuitive!
The RDB Loader doesn’t do much “work” itself; all it does is issue the load statements to Redshift. And it deliberately runs only one load command at a time, so that Redshift can spread the load over the cluster. If you run multiple loader apps, then you will find the loaders are competing to issue load statements to Redshift. This means the Redshift cluster is not able to dedicate its full resource to each load. (You might also run into problems with table locks – but I’m not 100% sure about this, I’ve never tried it).
1 Like
Thanks