Monitoring memory on AWS EC2

For engineers
1

Hi all :slight_smile:
We have a snowlow pipeline running in AWS, which we setup using the terraform modules. We would like to monitor the EC2 instances, particularly the memory usage.

We would normally use CloudWatch as this would provide historical data suitable for graphing the memory usage as well as alerting if a threshold is exceeded.

From what I can see there is no easy way of getting the CloudWatch agent onto the instances together with the required policies onto the instance roles. We could use custom AMI’s to provide an image with the cloudwatch agent already installed, but the role would still be lacking the policies to allow it to work.

Any suggestions?

2

Hi @chris so I can see a few ways to tackle this:

  1. You can fork all of the OSS Terraform modules to support this use-case for yourself internally;
  2. Addition of ability to pass-through custom user-data + custom IAM policies for each module;
  3. Addition specifically of the CloudWatch agent + memory monitoring for each module

Personally I would likely opt for “Option 2” given it expands the capabilities the best and could be used for this use-case as well as installing any other arbitrary system in the modules.

The extra abilities would first need to be added to the base “service” module here: GitHub - snowplow-devops/terraform-aws-service-ec2

From there you would need to update each individual service to support the new capabilities you are interested in (assuming you wanted to stay using the OSS modules on offer).

Happy to review a PR for this capability if you end up pursuing this!

3

Hi @josh appologies for the late reply.

Thanks for the suggestions.

We would also like to go down the route of allowing custom user-data and policies.

The first step would be to get our current config using the latest versions of the modules. Trying the latest module versions in our terraform code results in a plan which proposes a lot of changes, this will involve lots of testing and downtime during the update on prod.

Not sure if its going to be feasable at the moment.

Thanks anayway!

4

Hi @chris the easiest option to upgrade is likely to deploy a completely new pipeline and to then switch your traffic to the new Load Balancer running on the latest pipeline (green / blue switchover).

You can achieve this pretty easily with a CNAME / A record switch to the new pipeline load balancer and carries far less risk than a rolling update if you are not confident in what the end result will be - as you can rollback immediately to the still active old pipeline if anything is not working as you want.

Good luck on the upgrade if you do go down that route!