Terraform: 503 error on creating several CloudSQL databases (fails after third was actually created)

[Blasterdick]

.tf for a CloudSQL instance and it's databases:

resource "google_sql_database_instance" "default" {
    region = "europe-west1"
    database_version = "MYSQL_5_5"
    settings {
        tier = "D1"
        activation_policy = "ON_DEMAND"
        authorized_gae_applications = []
        pricing_plan = "PER_USE"
        replication_type = "SYNCHRONOUS"

        backup_configuration {
            binary_log_enabled = true
            enabled = true
            start_time = "18:00"
        }   

        ip_configuration {
            ipv4_enabled = true
            authorized_networks {
                name = "office_network"
                value = "123.456.789.000"
            }   
        }   

        location_preference {
            zone = "europe-west1-b"
        } 
    }   
}   

resource "google_sql_database" "db-temp-test1" {
    name = "db-temp-test1"
    instance = "${google_sql_database_instance.default.name}"
}

resource "google_sql_database" "db-temp-test2" {
    name = "db-temp-test2"
    instance = "${google_sql_database_instance.default.name}"
}

resource "google_sql_database" "db-temp-test3" {
    name = "db-temp-test3"
    instance = "${google_sql_database_instance.default.name}"
}

Terraform plan sample:

+ google_sql_database.db-temp-test1
    instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
    name:      "" => "db-temp-test1"
    self_link: "" => "<computed>"

+ google_sql_database.db-temp-test2
    instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
    name:      "" => "db-temp-test2"
    self_link: "" => "<computed>"

+ google_sql_database.db-temp-test3
    instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
    name:      "" => "db-temp-test3"
    self_link: "" => "<computed>"

Terraform apply sample:

google_sql_database.db-temp-test3: Creating...
  instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
  name:      "" => "db-temp-test3"
  self_link: "" => "<computed>"
google_sql_database.db-temp-test1: Creating...
  instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
  name:      "" => "db-temp-test1"
  self_link: "" => "<computed>"
google_sql_database.db-temp-test2: Creating...
  instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
  name:      "" => "db-temp-test2"
  self_link: "" => "<computed>"

Error message:

* google_sql_database.db-temp-test2: Error, failed to insert database db-temp-test2 into instance terraform-kez4yt4ivzazjcokccu4zw6wni: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.db-temp-test1: Error, failed to insert database db-temp-test1 into instance terraform-kez4yt4ivzazjcokccu4zw6wni: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.db-temp-test3: Error, failure waiting for insertion of db-temp-test3 into terraform-kez4yt4ivzazjcokccu4zw6wni: speckle::MySQL exception: mysql_query You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-temp-test3' at line 1

[Blasterdick]

After changing db name with _ instead of - and deleting an old one (with instance actually, cause it failed to delete it even through web-interface) a new error appears:

3 error(s) occurred:

* google_sql_database.db_temp_test1: Error, failed to insert database db_temp_test1 into instance terraform-neugp7b2nvdahcebbsb3ynikyi: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.db_temp_test2: Error, failed to insert database db_temp_test2 into instance terraform-neugp7b2nvdahcebbsb3ynikyi: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.db_temp_test3: Error, failed to insert database db_temp_test3 into instance terraform-neugp7b2nvdahcebbsb3ynikyi: googleapi: Error 503: Service temporarily unavailable., serverException

Terraform apply:

google_sql_database_instance.default: Creation complete
google_sql_database.db_temp_test1: Creating...
  instance:  "" => "terraform-neugp7b2nvdahcebbsb3ynikyi"
  name:      "" => "db_temp_test1"
  self_link: "" => "<computed>"
google_sql_database.db_temp_test2: Creating...
  instance:  "" => "terraform-neugp7b2nvdahcebbsb3ynikyi"
  name:      "" => "db_temp_test2"
  self_link: "" => "<computed>"
google_sql_database.db_temp_test3: Creating...
  instance:  "" => "terraform-neugp7b2nvdahcebbsb3ynikyi"
  name:      "" => "db_temp_test3"
  self_link: "" => "<computed>"

while all databases appeared in web-interface.

[Blasterdick]

Next running terraform plan trying to create databases again, just like they are doesn't exists.

[Blasterdick]

Clean-run:

3 error(s) occurred:

* google_sql_database.test_db_test: Error, failed to insert database test_db_test into instance terraform-yeiopvrpwbez5b3b5r6jpq3zn4: googleapi: Error 403: Operation failed because another operation was already in progress., operationInProgress
* google_sql_database.test_db_local: Error, failed to insert database test_db_local into instance terraform-yeiopvrpwbez5b3b5r6jpq3zn4: googleapi: Error 403: Operation failed because another operation was already in progress., operationInProgress
* google_sql_database.test_db: Error, failed to insert database test_db into instance terraform-yeiopvrpwbez5b3b5r6jpq3zn4: googleapi: Error 503: Service temporarily unavailable., serverException

Second plan/apply

2 error(s) occurred:

* google_sql_database.test_db_local: Error, failed to insert database test_db_local into instance terraform-yeiopvrpwbez5b3b5r6jpq3zn4: googleap
i: Error 503: Service temporarily unavailable., serverException
* google_sql_database.test_db_test: Error, failed to insert database test_db_test into instance terraform-yeiopvrpwbez5b3b5r6jpq3zn4: googleapi:
 Error 503: Service temporarily unavailable., serverException

Third plan/apply:

2 error(s) occurred:

* google_sql_database.test_db_test: Error, failed to insert database test_db_test into instance terraform-yeiopvrpwbez5b3b5r6jpq3zn4: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.test_db_local: Error, failure waiting for insertion of test_db_local into terraform-yeiopvrpwbez5b3b5r6jpq3zn4: speckle::MySQL exception: mysql_query Can't create database 'test_db_local'; database exists

[evandbrown]

This looks like an issue with concurrent databases.insert calls to the CloudSQL API. I was able to replicate your issue, and was able to fix it with a parallelism=1 flag.

For example:

$ terraform apply --parallelism=1

and

$ terraform destroy --parallelism=1

This is a temporary workaround. I'm working with the CloudSQL team to investigate further, and I'll keep this issue updated.

[Blasterdick]

@evandbrown thank you, mate, I'm looking forward for your update.

[aditya87]

Hi, any updates on this issue? We are seeing this intermittently as well.

[guidowb]

Started seeing this about 18 hours ago. Scripts that ran successfully 21 hours ago started failing with these types of concurrency errors and haven't been completing successfully since. Manually serializing by adding 'depends_on' did not fix the issue. Neither did --parallellism=1.
Terraform version and templates did not change. Seems like API behavior did?

[virtmerlin]

Still seeing this as well, however manually forcing serialization via depends_on does seem to work around the issue for now in our pipelines. We did however have to serialize ALL objects after creation of the cloud sql instance (users&dbs). Like @guidowb posted above, this issue manifested itself to our pipelines on the 21 of Dec with no changes to Terraform templates. It does however coincide with the Dec 21 release of terraform cli 0.8.2 (December 21, 2016) which our pipelines did pick up.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.