GCP setup - Instance is being verified and Health Check Status Timeout Warning

For engineers
1

Hello there,

I’m trying to setup Snowplow with GCP. I’m following snowplow ressources as well as simo ahava’s guide.
I’m stuck when creating an instance group. I’ve tried creating new projects, tweak around firewall rules but can’t get nowhere.

For the configuration of the Instance Template, the firewall rules and the Instance group I’m following exactly the instructions given in the GCP Snowplow setup guide.

The issue I have is that when looking at the Instance Group Members list I can see that the instance created is labeled with two warnings:

  • Instance is being verified (under status)
  • Timeout (under Health Check Status)

Obviously when trying to run curl http://<EXTERNAL_IP_HERE>:8080/health from my terminal It doesn’t work. I have en error message Connection refused.

When creating the Instance Group without adding any Health Checks I’m not seeing any errors but still have the Connection refused error.

What can I do to try to debug this?

Thanks.

2

Hi @simonbreton that sounds like your instance startup-script is not executing correctly and that your Collector is never coming up correctly.

You would likely need to SSH into the server to check whether the launch script has executed correctly and that the Collector is actually running on the server - your VM instances generally also publish these logs directly to GCP Monitoring so checking for errors there would also be a good idea!

If you are using the GCP OSS modules we recently ran into an issue where the Docker package we were using got removed which could cause this failure loop. These modules were recently patched to fix this issue: Bump GCP CE based modules to latest (close #20) · snowplow/quickstart-examples@0c53555 · GitHub

1 Like
3

Hey @josh,

thanks for your message. I took a look at the log and I’m not seeing anything that could help me. and no error. However I think you are right when you say that the script is not executed correctly. I’ve ssh into the server and I can see for example that the application.conf is not even there. But when I’m executing the command manually it is working and I can see the file.

So for example here is the simple startup script I’m using to test it:

#! /bin/bash
gsutil cp gs://path-here/application.conf .

and like I said when entering the command manually it is working. The image I’m using is ubuntu-1604.
I have the same issue when creating the instance directly.

What am I missing here?

4

Hey @simonbreton - not sure why its not executing - it might be worth reaching out to GCP Support for help here?

We have a working example with an instance group here that you can use for reference: terraform-google-collector-pubsub-ce/startup-script.sh.tmpl at main · snowplow-devops/terraform-google-collector-pubsub-ce · GitHub

5

I wish I could close this topic as I was only a matter of miss configuration between gcp firewall, vm instance setup and application.conf

1 Like
6

Happens to the best of us! Glad you got it sorted in the end !!

7

I am also facing the same issue where the instance is showing Instance is being verified (under status) & Timeout (under Health Check Status). I am following the same steps as mentioned in the document . My other fellow also facing the same issue , could you please let me know if there is any solution for this issue .

8

Could you please help me , I am also into same situation and unable to find out what exactly the change needed between gcp firewall, vm instance setup and application.conf

9

Hi @amit_shrivastava are you having issues with the Terraform modules or are you trying to do this from scratch and running into problems there?