Hi,
we are setting up the RDB Loader and would like to use postgres as a storage target in order to use Amazon Aurora. However, in the config.hocon it says that Redshift is the only option:
# Warehouse connection details, identical to storage target config
"storage": {
# Database, redshift is the only acceptable option
"type": "redshift",
Is that correct? The Github Documentation suggests that multiple storage targets are possible:
Hi @mgloel ,
Indeed our RDB loader supports only Redshift ATM.
In your case you probably want to use our Postgres loader, that consumes enriched events from Kinesis directly and inserts them into Postgres.
Setup guide:
Hey @BenB ,
thanks a lot for the quick confirmation. Is there a plan that RDB Loader can support other storage targets in the future?
Cheers
Matthias
Hi @mgloel,
No we don’t plan to support Postgres in RDB Loader. In fact in previous versions it always had an extremely limited and experimental support and we tried to not advertise it.
We want to make RDB Loader a kind of “batch OLAP” Loader that is specialized on warehouses where efficient “streaming insert” is not possible and instead one has to prepare a batch of data first and then load it using single COPY statement. Redshift and Snowflake follow this model, but Postgres (as well as BigQuery or Elasticsearch) can have a more real-time approach where every event gets into a warehouse from topic as quick as possible.