Terraform Apply timeout error

For engineers GCP pipeline
1

Hi! I am trying to set snowplow on gcp using terraform and iglu. But when I run terraform apply, I get this error:

Screenshot at Mar 30 09-30-20
Screenshot at Mar 30 09-30-201037×148 25.8 KB

I tried to find where the error occurs. Found the instance group and instance, but when I checked their logs, found no errors. The only place where I found errors is Monitoring → Observabiliy Logging → Logs Explorer:

ALERT 2024-03-30T00:19:54.736599Z 2024-03-30 00:19:54.736 UTC [21]: [1-1] db=cloudsqladmin,user=cloudsqladmin FATAL: the database system is starting up
ALERT 2024-03-30T00:19:54.736799Z 2024-03-30 00:19:54.736 UTC [23]: [1-1] db=cloudsqladmin,user=cloudsqladmin FATAL: the database system is starting up
ALERT 2024-03-30T00:19:54.737653Z 2024-03-30 00:19:54.737 UTC [22]: [1-1] db=cloudsqladmin,user=cloudsqladmin FATAL: the database system is starting up
INFO 2024-03-30T00:19:54.743852Z 2024-03-30 00:19:54.743 UTC [20]: [2-1] db=,user= LOG: MultiXact member wraparound protections are now enabled
INFO 2024-03-30T00:19:54.970640Z 2024-03-30 00:19:54.970 UTC [7]: [1-1] db=,user= LOG: database system is ready to accept connections
INFO 2024-03-30T00:19:54.971477Z 2024-03-30 00:19:54.971 UTC [27]: [1-1] db=,user= LOG: autovacuum launcher started
INFO 2024-03-30T00:19:58.485710Z 2024-03-30 00:19:58.484 UTC [7]: [2-1] db=,user= LOG: received SIGHUP, reloading configuration files
ERROR 2024-03-30T00:19:58.492171Z 2024-03-30 00:19:58.491 UTC [31]: [1-1] db=cloudsqladmin,user=cloudsqladmin ERROR: relation "public.heartbeat" does not exist at character 13
INFO 2024-03-30T00:19:58.492302Z 2024-03-30 00:19:58.491 UTC [31]: [2-1] db=cloudsqladmin,user=cloudsqladmin STATEMENT: INSERT INTO public.heartbeat(id, master_time) VALUES (1, now() at time zone 'utc') ON CONFLICT (id) DO UPDATE SET master_time = (now() at time zone 'utc');
NOTICE 2024-03-30T00:20:25.179758Z [protoPayload.serviceName: cloudsql.googleapis.com] [protoPayload.methodName: cloudsql.instances.create] [protoPayload.resourceName: projects/snowplowmaksim] [protoPayload.authenticationInfo.principalEmail: xxxx@xxxx.com] audit_log, method: "cloudsql.instances.create", principal_email: "xxxx@xxxx.com"
2

Hi @maksim can you confirm which version (githash) of the quickstart you are using? Can you also verify if you have done anything custom with the networking layer that the modules are being deployed into?

3

Thanks @josh for the reply.

  1. 23.10-patch.1-4-g0461951
  2. I didn’t customize networking layer
4

Hi @maksim for some reason the latest release tag for Quickstart was never pushed out properly! I have now tagged 24.02 which has a range of updates for all three clouds we support - though none that stand out to explain this issue.

The reason the instance group times out is because none of the servers it is trying to deploy every become “healthy”. If the instances are not becoming healthy this either means that:

  1. We could not pull the appropriate images from DockerHub as part of the setup script OR;
  2. We could pull the image but they couldn’t connect to the CloudSQL database instance

Can you check if the CloudSQL instance deployed properly?

5

One other possible cause is your GCP account might have reached the quota limit of the max number of instances can be created.