We are trying to get a snowplow pipeline working using the quickstart “secure” terraform for aws. The target database is snowflake. We have applied the terraform for iglu server, pipeline and snowflake, and it all seems to have worked okay.
However when we run the test curl command, we get data in the S3 buckets, but not snowflake.
There are no errors in cloudwatch, just INFO messages. There are also no “COPY events FROM s3…” messages in cloudwatch, which I think is suspicious.
I think to diagnose this issue it would be helpful if I could understand which part of the quickstart terraform sets up the integratuion between S3 and snowflake.
Reading this in the guide: “… For this purpose, the Snowflake Terraform module has been created. This module creates resources including, but not limited to, Snowflake database, table, user, and role. These resources are needed by the Snowflake Loader to operate correctly.”
Does this also ceate the resources which copy data from S3 to snowflake?
Which resources can I look for in the state and plan to see which might be missing or misconfigured?
Any clues to point us in the right direction would be very helpfule