AWS servers setup for kinesis

For engineers AWS real-time pipeline
1

Hey,

Currently we are processing around 2 million events per day, I was wondering if there best practice for server capacity We are basically doing the basic enrichments

ip_lookups
referer_parser.
event_fingerprint_config
user_agent_utils_config
campaign_attribution.

1)What is the required instance type for the stream-collector?
2)What is the required instance type for the stream-enrichment?

Thanks,

2

Looks like T2.medium worked for kjain here:
http://discourse.snowplow.io/t/optimizing-aws-stack-for-medium-load-site/961

One way to test it is to use Snowplow Avalanche (https://github.com/snowplow/avalanche) and do load testing. If you use small to medium instances and put inside an ELB and ASG - you can monitor the logs in cloudwatch dashboard and see if you are hitting capacity or being throttled.

1 Like