we used Quickstart Example for AWS including Snowflake Loader. We initiated infrastructure on 2022-06-01 and it worked fine for some time. In the end of June we noticed that in Transformed/Good S3 bucket we have no new events altough in Enriched we have some. We restarted sp-snowflake-loader-server and it started working again.
Today, we found the same issue so I’m asking for help here. See screenshot attached, that enriched folder has events from today 2022-07-26 but transformed folder has events up to 2022-07-19.
I’m not sure to understand your setup, you mention transformer kinesis but I don’t see it on your list of ec2 instances, could it be that it crashed and that’s why fan-out stopped working and transformed data is not written to s3 more ?
BTW if you use transformer kinesis, you don’t need s3 loader for the enriched events, as the transformer consumes the enriched events directly from Kinesis and writes the transformed ones in transformed folder on s3.
thanks a lot for the pointer. You are right that ec2 “sp-transformer-kinesis-enriched-server” stopped working and it missing. I need to restore it and find out why it crashed.
Regarding s3 loader, we used quickstart to set up AWS Snowflake default version. So is it included as backup in the terraform script?
so it turns out that Auto scaling group has trouble scaling to 1 instance of this ec2.
In activity log, it tells me that:
Launching a new EC2 instance. Status Reason: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance. Launching EC2 instance failed.
However, it has been over 14 days and the support doesn’t answer. Have you ever dealt with this?
It might be there because historically Snowflake loader was only able to load data from S3, and then when we introduced the loading directly from Kinesis we haven’t removed S3 loader.
Awesome that you could figure out the reason!
Not to my knowledge, sorry. I let other people of the team jump in if they did.
Case closed. Turns out aws was sending verification email to non-existent email address of aws subaccount. Once we created it we verified and it autoscaled.
