“terraform apply” fail with timeout error with QuickStart Guide on GCP because Cloud SQL Instances takes too long

Hi there,

I’ve been working on setting up the Quick Start Installation Guide on GCP with someone and multiple time we have been stuck when running the set up of the Iglu Server because of terraform timeout error.

I know that there is multiple timeout error post but I think that what I’ve been facing here was a bit different.

Basically the first time we were running the terraform apply cmd was failing and working the second time (with exactly the same configuration). What I think happens is that the Cloud SQL Instances takes too long to initiate at first then is faster during the second run.

One way to fix this would be to extend the terraform timeout by 5min.

I don’t know if what I’m saying make sense. I’m sharing my experience (with multiple test). Hopefully it can help.

2 Likes

We had the same problem in 2 setups and could not resolve it.

We had one setup where it worked but no idea why…

This happened to me for the “secure” deployment strategy on GCP—at first.
Solved by adding NAT to my VPC :see_no_evil:

Hi Timo, yeah as Simon said itis linked to the timing of the SQL warm up (:slight_smile: First attempt takes systematically more than 20 minutes. If you leave your instance live and restart your terraform apply, it bugs too. You need to delete it before applying the plan again.

Hi @simonbreton appreciate the feedback on this. CloudSQL is for sure not the fastest service! Although in all my tests I never hit the timeout of 30 minutes - which region where you using in your deployments?

Have opened this ticket in the module for the CloudSQL instance here if you want to add extra comments there would be helpful (Increase timeouts for CloudSQL database creation · Issue #1 · snowplow-devops/terraform-google-cloud-sql · GitHub).

@dsolito would you mind sharing the exact steps you take to resolve this? Is it that on the first apply it hits a timeout and then on the second one you are somehow cleaning the state up so that it can use the deployed instance?

Thanks @josh… actually the timeout was at 20min not 30min

I see - looks like the older provider version (Terraform Registry) had 20 minutes whereas the latest provider version has a timeout of 30 minutes.

Hi @josh. I did 3 projects and each time the first attempt never pass the timeout (it’s 2-3 minutes after the 20minutes that the instance is ready). The solution is to delete the instance and run terraform apply again. If you don’t delete the instance, terraform waits for a running message that never come (the state is not adapted because the server is ready after the timeout). You just need to pray that it takes shorter time to be ready. To be more precise I don’t modify the tfstate file.

1 Like

Okay good to know @dsolito - have put a patch out today which should solve this.

If you see this commit: 22.01-Patch.2 Release by jbeemster · Pull Request #42 · snowplow/quickstart-examples · GitHub

Just bump the Cloud SQL module to v0.1.1 and the timeout problem should be solved for your deployments. If someone has time to give it a test drive to validate would be great!

1 Like

Thks Josh. Will give a try asap.

1 Like

Any luck @dsolito ?

Not yet @josh. Busy to learn the working pipeline :-).
I will give a try this week-end.
David

1 Like

Hello @josh. I confirm that the upgrade for the iglu_db module to 0.1.1 is working :slight_smile:
I had no timeout error at the first try.

module.iglu_db.google_sql_database_instance.instance: Creation complete after 20m27s

Iglu server terraform apply finished without trouble

David

1 Like

Awesome! Thanks for reporting back @dsolito and good luck hacking around on the working quickstart pipeline!

1 Like