Terraform quick start on GCP is not working , Trying to find manual set up page but not able to locate it in documentation

Amit_Shrivastava1 · August 21, 2023, 6:22am

I have tried to set up snowplow on gcp using the terraform module provided in the snowplow documentation. In the very first step of setting up iglu server the terraform job is failing due to health check failure , health check is continuely failing however the VM is health and running. I have enabled all the required API as mentioned in some of the solution on snowplow website, I have also tried to setting it up using linux machine but both ways i am getting into same error.

Could you please provide me the link having the steps to manually set up snowplow on gcp .

mike · August 21, 2023, 8:25am

What do the logs say for the Iglu Server for that VM? Generally if it’s failing the health check there should be something in the logs to indicate where the failure is occurring and as a result the rest of the stack is not provisioned.

Amit_Shrivastava1 · August 24, 2023, 4:39am

Hi Mike, I have reproduced the same error msg and captured the error in the attached image and error log from console as well as some log from cloud logging are copied below:
Snowplow-Error
running vm

Error msg on terminal

Error: timeout while waiting for state to become 'created' (last state: 'creating', timeout: 20m0s)
│
│   with module.iglu_server.google_compute_region_instance_group_manager.grp,
│   on .terraform\modules\iglu_server\main.tf line 230, in resource "google_compute_region_instance_group_manager" "grp":│  230: resource "google_compute_region_instance_group_manager" "grp" {
│
Cloud logging – warning msgs
Time 7:33 IST last log 
{
  "textPayload": "2023-08-23 14:03:30.213 UTC [617]: [2-1] db=,user= LOG:  automatic analyze of table \"cloudsqladmin.public.heartbeat\" system usage: CPU 0.00s/0.00u sec elapsed 0.00 sec",
  "insertId": "s=72ed0b03e32b4b62afddaf86f0cdf8f8;i=4184;b=b0a40380d639435d8dc7d5221fe3429a;m=5e347988;t=6039793859418;x=a212a1a48609141-0@a1",
  "resource": {
    "type": "cloudsql_database",
    "labels": {
      "region": "us-central",
      "project_id": "snowplow-poc-123",
      "database_id": "snowplow-poc-123:sp-iglu-db-558d4bfb"
    }
  },
  "timestamp": "2023-08-23T14:03:30.213912Z",
  "severity": "INFO",
  "labels": {
    "INSTANCE_UID": "1-852779d9-c055-4474-b749-6af25de04d34",
    "LOG_BUCKET_NUM": "68"
  },
  "logName": "projects/snowplow-poc-123/logs/cloudsql.googleapis.com%2Fpostgres.log",
  "receiveTimestamp": "2023-08-23T14:04:12.100784364Z"
}

Time - 7:23 pm IST
{
  "insertId": "3655777108714915225622482866447037347",
  "jsonPayload": {
    "@type": "type.googleapis.com/compute.InstanceGroupManagerEvent",
    "instanceHealthStateChange": {
      "instanceWithId": "projects/881847669974/zones/us-central1-c/instances/5622482866447037347",
      "previousDetailedHealthState": "UNKNOWN",
      "ipAddress": "10.128.0.2",
      "networkWithId": "projects/881847669974/global/networks/3725966222115212588",
      "notificationTime": "2023-08-23T13:53:50.624Z",
      "network": "projects/snowplow-poc-123/global/networks/default",
      "instance": "projects/snowplow-poc-123/zones/us-central1-c/instances/sp-iglu-server-8b6c",
      "healthCheck": "projects/snowplow-poc-123/global/healthChecks/sp-iglu-server",
      "detailedHealthState": "TIMEOUT"
    }
  },
  "resource": {
    "type": "gce_instance_group_manager",
    "labels": {
      "instance_group_manager_id": "365577710871491522",
      "location": "us-central1",
      "project_id": "snowplow-poc-123",
      "instance_group_manager_name": "sp-iglu-server-grp"
    }
  },
  "timestamp": "2023-08-23T13:53:50.624Z",
  "severity": "WARNING",
  "labels": {
    "compute.googleapis.com/instance_location": "us-central1-c",
    "compute.googleapis.com/instance_name": "sp-iglu-server-8b6c",
    "compute.googleapis.com/instance_id": "5622482866447037347"
  },
  "logName": "projects/snowplow-poc-123/logs/compute.googleapis.com%2Finstance_group_manager_events",
  "receiveTimestamp": "2023-08-23T13:53:51.806943432Z"
}



Time - 7:15 pm IST

{
  "insertId": "3655777108714915225622482866447037347",
  "jsonPayload": {
    "@type": "type.googleapis.com/compute.InstanceGroupManagerEvent",
    "instanceHealthStateChange": {
      "instance": "projects/snowplow-poc-123/zones/us-central1-c/instances/sp-iglu-server-8b6c",
      "healthCheck": "projects/snowplow-poc-123/global/healthChecks/sp-iglu-server",
      "instanceWithId": "projects/881847669974/zones/us-central1-c/instances/5622482866447037347",
      "ipAddress": "10.128.0.2",
      "networkWithId": "projects/881847669974/global/networks/3725966222115212588",
      "detailedHealthState": "TIMEOUT",
      "previousDetailedHealthState": "UNKNOWN",
      "notificationTime": "2023-08-23T13:45:13.588Z",
      "network": "projects/snowplow-poc-123/global/networks/default"
    }
  },
  "resource": {
    "type": "gce_instance_group_manager",
    "labels": {
      "instance_group_manager_id": "365577710871491522",
      "project_id": "snowplow-poc-123",
      "instance_group_manager_name": "sp-iglu-server-grp",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-08-23T13:45:13.588Z",
  "severity": "WARNING",
  "labels": {
    "compute.googleapis.com/instance_location": "us-central1-c",
    "compute.googleapis.com/instance_name": "sp-iglu-server-8b6c",
    "compute.googleapis.com/instance_id": "5622482866447037347"
  },
  "logName": "projects/snowplow-poc-123/logs/compute.googleapis.com%2Finstance_group_manager_events",
  "receiveTimestamp": "2023-08-23T13:45:14.429114442Z"
}



{
  "insertId": "3655777108714915225622482866447037347",
  "jsonPayload": {
    "instanceHealthStateChange": {
      "instanceWithId": "projects/881847669974/zones/us-central1-c/instances/5622482866447037347",
      "detailedHealthState": "TIMEOUT",
      "healthCheck": "projects/snowplow-poc-123/global/healthChecks/sp-iglu-server",
      "instance": "projects/snowplow-poc-123/zones/us-central1-c/instances/sp-iglu-server-8b6c",
      "networkWithId": "projects/881847669974/global/networks/3725966222115212588",
      "ipAddress": "10.128.0.2",
      "network": "projects/snowplow-poc-123/global/networks/default",
      "notificationTime": "2023-08-23T13:45:13.588Z",
      "previousDetailedHealthState": "UNKNOWN"
    },
    "@type": "type.googleapis.com/compute.InstanceGroupManagerEvent"
  },
  "resource": {
    "type": "gce_instance_group_manager",
    "labels": {
      "instance_group_manager_name": "sp-iglu-server-grp",
      "project_id": "snowplow-poc-123",
      "location": "us-central1",
      "instance_group_manager_id": "365577710871491522"
    }
  },
  "timestamp": "2023-08-23T13:45:13.588Z",
  "severity": "WARNING",
  "labels": {
    "compute.googleapis.com/instance_name": "sp-iglu-server-8b6c",
    "compute.googleapis.com/instance_location": "us-central1-c",
    "compute.googleapis.com/instance_id": "5622482866447037347"
  },
  "logName": "projects/snowplow-poc-123/logs/compute.googleapis.com%2Finstance_group_manager_events",
  "receiveTimestamp": "2023-08-23T13:45:14.429114442Z"
}


{
  "textPayload": "2023-08-23 13:43:28.193 UTC [30]: [1-1] db=cloudsqladmin,user=cloudsqladmin ERROR:  relation \"public.heartbeat\" does not exist at character 13",
  "insertId": "s=72ed0b03e32b4b62afddaf86f0cdf8f8;i=32c5;b=b0a40380d639435d8dc7d5221fe3429a;m=16784f23;t=603974bc969b3;x=9dd17553cb4db030-0@a2",
  "resource": {
    "type": "cloudsql_database",
    "labels": {
      "project_id": "snowplow-poc-123",
      "region": "us-central",
      "database_id": "snowplow-poc-123:sp-iglu-db-558d4bfb"
    }
  },
  "timestamp": "2023-08-23T13:43:28.199493Z",
  "severity": "ERROR",
  "labels": {
    "LOG_BUCKET_NUM": "68",
    "INSTANCE_UID": "1-852779d9-c055-4474-b749-6af25de04d34"
  },
  "logName": "projects/snowplow-poc-123/logs/cloudsql.googleapis.com%2Fpostgres.log",
  "receiveTimestamp": "2023-08-23T13:43:29.873172918Z"
}


Time - 7:13 pm IST

{
  "textPayload": "2023-08-23 13:43:28.193 UTC [30]: [1-1] db=cloudsqladmin,user=cloudsqladmin ERROR:  relation \"public.heartbeat\" does not exist at character 13",
  "insertId": "s=72ed0b03e32b4b62afddaf86f0cdf8f8;i=32c5;b=b0a40380d639435d8dc7d5221fe3429a;m=16784f23;t=603974bc969b3;x=9dd17553cb4db030-0@a2",
  "resource": {
    "type": "cloudsql_database",
    "labels": {
      "region": "us-central",
      "project_id": "snowplow-poc-123",
      "database_id": "snowplow-poc-123:sp-iglu-db-558d4bfb"
    }
  },
  "timestamp": "2023-08-23T13:43:28.199493Z",
  "severity": "ERROR",
  "labels": {
    "INSTANCE_UID": "1-852779d9-c055-4474-b749-6af25de04d34",
    "LOG_BUCKET_NUM": "68"
  },
  "logName": "projects/snowplow-poc-123/logs/cloudsql.googleapis.com%2Fpostgres.log",
  "receiveTimestamp": "2023-08-23T13:43:29.873172918Z"
}

Snowplow-Error

Amit_Shrivastava1 · August 25, 2023, 9:44am

@mike , would you be able to suggest some solution for the error I mentioned above along .

josh · August 27, 2023, 8:18pm

Hi @Amit_Shrivastava1 the logs we would need to help debug this are the actual logs from the VM instance that was being deployed. This will tell us why it did not launch correctly.

Would you mind sharing those with us?

Amit_Shrivastava1 · August 31, 2023, 12:17pm

Hi @josh , There is no error log for VM so i have export full log as well as the VM log for the terraform quickstart execution and here is the git link from where you can access the json log files .

please have a look and suggest .

josh · August 31, 2023, 11:22pm

Hi @Amit_Shrivastava1 the logs we need can be found by navigating to the VM instance in the “Compute Engine” UI. From there:

Select the VM;
Select the “Observability” panel;
Select “Logs”

This should give you the log stream from the VM instance which includes output from the launch script that is being executed on the VM and hopefully give us some insight into what is going wrong.