"terraform apply" fail with timeout error

[Rick_Yang]

We have a quickstart open source snowplow deployments on GCP. It has run a month. Just now, I updated the terraform.tfvars files. While I “terraform apply” on my jump server, I received this timeout error.

│ Error: timeout while waiting for state to become ‘created’ (last state: ‘creating’, timeout: 20m0s)
│
│ with module.iglu_server.google_compute_region_instance_group_manager.grp,
│ on .terraform/modules/iglu_server/main.tf line 228, in resource “google_compute_region_instance_group_manager” “grp”:
│ 228: resource “google_compute_region_instance_group_manager” “grp” {
│

[josh]

Hi @Rick_Yang generally this means that your server is failing to launch so the health check never passes. Its worth checking the instance group on the GCP Console + the logs for the server that is being launched to check for errors.

[Rick_Yang]

I found the error from the log of the iglu server. It looks like iglu server fail to pass authentication of iglu db.

org.postgresql.util.PSQLException: FATAL: password authentication failed for user “iglu”

Right. We modified the password in terraform.tfvars. Terraform was supposed to update iglu server and iglu db while applying the changes. However, now iglu db has been changed to the new password and iglu server fails to connect.

[Rick_Yang]

I resolved the issue by ssh the vm and updating the password under /opt/snowplow/config/iglu-server.hocon manually.
A very odd thing is that I notice the password in vm’s startup-script is correct. However, the password under /opt/snowplow/config/iglu-server.hocon is wrong.
I am still not sure about the root reason. It may due to a bug or my incorrect operation.

The correct pwd: H$peY#UuQVHxE9Ga
The wrong password: H#UuQVHxE9Ga

[PaulBoocock]

Looks like $peY is being interpreted as a variable somewhere, so this is likely a bug.

[Rick_Yang]

Right. Just now, I updated the password in terrafrom.tfvars one more time. Then I deployed with “terraform apply”. Terraform runs smoothly.