Hi, i have setup a snowplow pipeline on aws following quick guide. i have enriched and transformed data in my S3 bucket. how do i load this data to my redshift cluster? Is RDB loader a separate application which is required to be setup? i have snowplow-transformer-server-stsv and snowplow-rs-loader-servers already setup when i setup pipeline using snowplow terraform. i think rs loader isn’t working. there are messages available in sqs queue but nothing in flight.
i am new to snowplow and stuck here. any suggestions and recommendations would be a great help.
The quick start guide already does all of this for you. (As illustrated here: https://docs.snowplow.io/docs/getting-started-on-community-edition/what-is-deployed/?warehouse=redshift&cloud=aws).
If you followed all the steps, you should be able to send events and see data in Redshift.
Hi @stanch thank you for quick response,
yes that’s what i am wondering about. i can not see rs-loader-servers logs in cloudwatch either. according to the docs, transformer server notify the loader (via SQS) and then loader send SQL commands for loading data from S3 bucket. but in my case i can’t see data in redshift schema. i checked the iam access advisor for loader server, seems like it never accessed the sqs and s3
let me try again by creating a new redshift schema and user
Issue is fixed, i just had to clone the snowplow repository again and the complete flow worked. i guess terraform code on my local repo was old or missing something.
1 Like