Terraform-provider-google: BigQuery Table Hive Partitioning not Working with Explicit Schema (autodetect=false)

Created on 25 Jun 2020  ·  14Comments  ·  Source: hashicorp/terraform-provider-google


Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave _+1_ or _me too_ comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.18
+ provider.archive v1.3.0
+ provider.google v3.26.0
+ provider.kubernetes v1.11.3
+ provider.random v2.2.1

Affected Resource(s)

  • google_bigquery_table

Terraform Configuration Files

variable "schema_types" {
  type    = list(object({schema = string, batched = string}))
  default = [{schema = "improbable", batched = "-batched"},
             {schema = "playfab", batched = "-batched"},
             {schema = "improbable", batched = ""},
             {schema = "playfab", batched = ""}]
}

resource "google_bigquery_table" "table_events_external" {
  count      = length(var.schema_types)
  dataset_id = google_bigquery_dataset.dataset_external.dataset_id
  table_id   = "events_${var.schema_types[count.index].schema}${replace(var.schema_types[count.index].batched, "-", "_")}"

  external_data_configuration {
    autodetect            = false
    ignore_unknown_values = true
    compression           = "GZIP"
    source_format         = "NEWLINE_DELIMITED_JSON"

    hive_partitioning_options {
      mode              = "CUSTOM"
      source_uri_prefix = "gs://${var.gcloud_project}-analytics-${var.environment}${var.schema_types[count.index].batched}/data_type=jsonl/event_schema=${var.schema_types[count.index].schema}/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
    }

    source_uris = [
      "gs://${var.gcloud_project}-analytics-${var.environment}${var.schema_types[count.index].batched}/data_type=jsonl/event_schema=${var.schema_types[count.index].schema}/*"
    ]
  }
  schema = var.schema_types[count.index].schema == "improbable" ? var.schema_improbable : var.schema_playfab
}

Debug Output

Terraform did not produce an error.

Expected Behavior

I should have gotten tables _with_ Hive partition columns, however the tables that were successfully created did not show the hive partitions.

Actual Behavior

This is an example table that Terraform produced:

Screenshot 2020-06-25 11 57 13

Screenshot 2020-06-25 11 57 28

Steps to Reproduce

I have tried applying the same without passing it an explicit schema, and set autodetect = true. Now the tables produced do have the Hive partitions, however:

  • Some tables do not have all columns I'd like them to have (one table has a subset of columns of the other which I'd like to be NULL for easy UNION'ing between the two).
  • Autodetect in this case didn't work as it erroneously classified some columns as int whereas it is of type string, which when BigQuery encounters it, it can't cast to INT and throws an error.

For these reasons I'd like to pass it an explicit schema.

I get it working correctly via:

# Step 1 - Create the table definition file:
bq mkdef \
  --ignore_unknown_values \
  --source_format=NEWLINE_DELIMITED_JSON \
  --hive_partitioning_mode=CUSTOM \
  --hive_partitioning_source_uri_prefix=gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING} \
  gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/\* \
  AnalyticsEnvironment:STRING,PlayFabEnvironment:STRING,SourceType:STRING,Source:STRING,EventNamespace:STRING,TitleId:STRING,GroupBatchId:STRING,BatchId:STRING,EventId:STRING,EventName:STRING,EntityType:STRING,EntityId:STRING,Timestamp:TIMESTAMP,ReceivedTimestamp:TIMESTAMP,BatchedTimestamp:TIMESTAMP,BatchJobName:STRING,EventAttributes:STRING \
  > /Users/loek/Desktop/events_playfab_testing_hive_partitioned_batched

# Step 2 - Apply the definition file:
bq mk --table --location=US --external_table_definition=/Users/loek/Desktop/events_playfab_testing_hive_partitioned_batched external.events_playfab_hive_partitioned_batched

The definition file looks like this:

{
  "autodetect": true,
  "hivePartitioningOptions": {
    "mode": "CUSTOM",
    "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
  },
  "ignoreUnknownValues": true,
  "schema": {
    "fields": [
      {
        "name": "AnalyticsEnvironment",
        "type": "STRING"
      },
      {
        "name": "PlayFabEnvironment",
        "type": "STRING"
      },
      {
        "name": "SourceType",
        "type": "STRING"
      },
      {
        "name": "Source",
        "type": "STRING"
      },
      {
        "name": "EventNamespace",
        "type": "STRING"
      },
      {
        "name": "TitleId",
        "type": "STRING"
      },
      {
        "name": "GroupBatchId",
        "type": "STRING"
      },
      {
        "name": "BatchId",
        "type": "STRING"
      },
      {
        "name": "EventId",
        "type": "STRING"
      },
      {
        "name": "EventName",
        "type": "STRING"
      },
      {
        "name": "EntityType",
        "type": "STRING"
      },
      {
        "name": "EntityId",
        "type": "STRING"
      },
      {
        "name": "Timestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "ReceivedTimestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "BatchedTimestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "BatchJobName",
        "type": "STRING"
      },
      {
        "name": "EventAttributes",
        "type": "STRING"
      }
    ]
  },
  "sourceFormat": "NEWLINE_DELIMITED_JSON",
  "sourceUris": [
    "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"
  ]
}

Note that in bq mkdef I did not set --autodetect, yet the file still has "autodetect": true (other bug -- Google Cloud SDK 298.0.0 & bq 2.0.58?). Changing this to false, or removing it altogether doesn't change anything though, it still works:

Screenshot 2020-06-25 12 22 43

Screenshot 2020-06-25 12 22 35

Finally,

  • One remaining discrepancy between the tables I can see is that the faulty one has Compression = GZIP, whereas the working one does not have this set. But I'm not sure how this would translate to the missing partitions. Also note I am using decompressive transcoding (all files in Cloud Storage have Content-Encoding: gzip set).
  • I _suspect_ that when the schema is explicitly passed with Terraform, it is potentially applied later (compared to bq mk) and is overriding the hive path partitions?

References

bug
Source
picture LoekL

All 14 comments

@LoekL it would help if you could run this with TF_LOG=debug terraform apply and bq --apilog "" mk ... so we can compare the payloads and investigate further.

ffung picture ffung on 25 Jun 2020

@ffung, dropped a table & then:

export TF_LOG=debug
terraform apply -target=module.bigquery

<!-- removed some log lines here where Terraform is figuring out what to apply --!>

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

2020/06/25 16:15:45 [DEBUG] command: asking for input: "Do you want to perform these actions?"
  # module.bigquery.google_bigquery_table.table_events_external[1] will be created
  + resource "google_bigquery_table" "table_events_external" {
      + creation_time       = (known after apply)
      + dataset_id          = "external"
      + etag                = (known after apply)
      + expiration_time     = (known after apply)
      + id                  = (known after apply)
      + last_modified_time  = (known after apply)
      + location            = (known after apply)
      + num_bytes           = (known after apply)
      + num_long_term_bytes = (known after apply)
      + num_rows            = (known after apply)
      + project             = (known after apply)
      + schema              = jsonencode(
            [
              + {
                  + name = "AnalyticsEnvironment"
                  + type = "STRING"
                },
              + {
                  + name = "PlayFabEnvironment"
                  + type = "STRING"
                },
              + {
                  + name = "SourceType"
                  + type = "STRING"
                },
              + {
                  + name = "Source"
                  + type = "STRING"
                },
              + {
                  + name = "EventNamespace"
                  + type = "STRING"
                },
              + {
                  + name = "TitleId"
                  + type = "STRING"
                },
              + {
                  + name = "GroupBatchId"
                  + type = "STRING"
                },
              + {
                  + name = "BatchId"
                  + type = "STRING"
                },
              + {
                  + name = "EventId"
                  + type = "STRING"
                },
              + {
                  + name = "EventName"
                  + type = "STRING"
                },
              + {
                  + name = "EntityType"
                  + type = "STRING"
                },
              + {
                  + name = "EntityId"
                  + type = "STRING"
                },
              + {
                  + name = "Timestamp"
                  + type = "TIMESTAMP"
                },
              + {
                  + name = "ReceivedTimestamp"
                  + type = "TIMESTAMP"
                },
              + {
                  + name = "BatchedTimestamp"
                  + type = "TIMESTAMP"
                },
              + {
                  + name = "BatchJobName"
                  + type = "STRING"
                },
              + {
                  + name = "EventAttributes"
                  + type = "STRING"
                },
            ]
        )
      + self_link           = (known after apply)
      + table_id            = "events_playfab_batched"
      + type                = (known after apply)

      + external_data_configuration {
          + autodetect            = false
          + compression           = "GZIP"
          + ignore_unknown_values = true
          + source_format         = "NEWLINE_DELIMITED_JSON"
          + source_uris           = [
              + "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*",
            ]

          + hive_partitioning_options {
              + mode              = "CUSTOM"
              + source_uri_prefix = "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.


Warning: Resource targeting is in effect

You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.

The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

2020/06/25 16:15:48 [INFO] backend/local: apply calling Apply
2020/06/25 16:15:48 [INFO] terraform: building graph: GraphTypeApply

<!-- removing more irrelevant logs !-->

2020/06/25 16:15:48 [WARN] Provider "registry.terraform.io/-/google" produced an invalid plan for module.bigquery.google_bigquery_table.table_events_external[1], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
",\n      \"type\": \"STRING\"\n    },\n    {\n      \"name\": \"EntityId\",\n      \"type\": \"STRING\"\n    },\n    {\n      \"name\": \"Timestamp\",\n      \"type\": \"TIMESTAMP\"\n    },\n    {\n      \"name\": \"ReceivedTimestamp\",\n      \"type\": \"TIMESTAMP\"\n    },\n    {\n      \"name\": \"BatchedTimestamp\",\n      \"type\": \"TIMESTAMP\"\n    },\n    {\n      \"name\": \"BatchJobName\",\n      \"type\": \"STRING\"\n    },\n    {\n      \"name\": \"EventAttributes\",\n      \"type\": \"STRING\"\n    }\n  ]\n")
module.bigquery.google_bigquery_table.table_events_external[1]: Creating...
2020/06/25 16:15:48 [DEBUG] module.bigquery.google_bigquery_table.table_events_external[1]: applying the planned Create change
2020-06-25T16:15:48.841+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:48 [INFO] Creating BigQuery table: events_playfab_batched
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:48 [DEBUG] Retry Transport: starting RoundTrip retry loop
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:48 [DEBUG] Retry Transport: request attempt 0
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:48 [DEBUG] Google API Request Details:
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: ---[ REQUEST ]---------------------------------------
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: POST /bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables?alt=json&prettyPrint=false HTTP/1.1
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Host: www.googleapis.com
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.18 (+https://www.terraform.io) Terraform Plugin SDK/1.11.0 terraform-provider-google/3.26.0
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Content-Length: 1308
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Content-Type: application/json
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Goog-Api-Client: gl-go/1.14.3 gdcl/20200514
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Accept-Encoding: gzip
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "externalDataConfiguration": {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "compression": "GZIP",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "hivePartitioningOptions": {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "mode": "CUSTOM",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "ignoreUnknownValues": true,
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "sourceFormat": "NEWLINE_DELIMITED_JSON",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "sourceUris": [
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   ]
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "schema": {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "fields": [
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "AnalyticsEnvironment",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "PlayFabEnvironment",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "SourceType",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "Source",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventNamespace",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "TitleId",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "GroupBatchId",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchId",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventId",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventName",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EntityType",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EntityId",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "Timestamp",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "ReceivedTimestamp",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchedTimestamp",
MP"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchJobName",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventAttributes",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    }
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   ]
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "tableReference": {
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "datasetId": "external",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "projectId": "tidal-turbine-222414",
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "tableId": "events_playfab_batched"
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  }
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: }
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:48.842+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: -----------------------------------------------------
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Google API Response Details:
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: ---[ RESPONSE ]--------------------------------------
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: HTTP/1.1 200 OK
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Connection: close
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Transfer-Encoding: chunked
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Alt-Svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Cache-Control: private
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Content-Type: application/json; charset=UTF-8
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Date: Thu, 25 Jun 2020 14:15:49 GMT
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Etag: g8TTwrRydbLOIDJmPI0u3Q==
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Server: ESF
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Vary: Origin
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Vary: X-Origin
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Vary: Referer
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Content-Type-Options: nosniff
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Frame-Options: SAMEORIGIN
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Xss-Protection: 0
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 6ab
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "kind": "bigquery#table",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "etag": "g8TTwrRydbLOIDJmPI0u3Q==",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "id": "tidal-turbine-222414:external.events_playfab_batched",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "selfLink": "https://www.googleapis.com/bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables/events_playfab_batched",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "tableReference": {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "projectId": "tidal-turbine-222414",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "datasetId": "external",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "tableId": "events_playfab_batched"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "schema": {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "fields": [
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "AnalyticsEnvironment",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "PlayFabEnvironment",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "SourceType",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "Source",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventNamespace",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
Id",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "GroupBatchId",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchId",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventId",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventName",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EntityType",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EntityId",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "Timestamp",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "ReceivedTimestamp",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchedTimestamp",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchJobName",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
erraform-provider-google_v3.26.0_x5:     "name": "EventAttributes",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    }
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   ]
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "numBytes": "0",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "numLongTermBytes": "0",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "numRows": "0",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "creationTime": "1593094549288",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "lastModifiedTime": "1593094549341",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "type": "EXTERNAL",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "externalDataConfiguration": {
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "sourceUris": [
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   ],
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "sourceFormat": "NEWLINE_DELIMITED_JSON",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "ignoreUnknownValues": true,
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "compression": "GZIP",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "hivePartitioningOptions": {
G] plugin.terraform-provider-google_v3.26.0_x5:    "mode": "CUSTOM",
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   }
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "location": "US"
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: }
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 0
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: -----------------------------------------------------
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Retry Transport: Stopping retries, last request was successful
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Retry Transport: Returning after 1 attempts
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [INFO] BigQuery table tidal-turbine-222414:external.events_playfab_batched has been created
2020-06-25T16:15:49.518+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [INFO] Reading BigQuery table: projects/tidal-turbine-222414/datasets/external/tables/events_playfab_batched
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Retry Transport: starting RoundTrip retry loop
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Retry Transport: request attempt 0
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Google API Request Details:
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: ---[ REQUEST ]---------------------------------------
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: GET /bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables/events_playfab_batched?alt=json&prettyPrint=false HTTP/1.1
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Host: www.googleapis.com
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.18 (+https://www.terraform.io) Terraform Plugin SDK/1.11.0 terraform-provider-google/3.26.0
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Goog-Api-Client: gl-go/1.14.3 gdcl/20200514
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Accept-Encoding: gzip
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.519+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: -----------------------------------------------------
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Google API Response Details:
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: ---[ RESPONSE ]--------------------------------------
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: HTTP/1.1 200 OK
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Connection: close
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Transfer-Encoding: chunked
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Alt-Svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Cache-Control: private
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Content-Type: application/json; charset=UTF-8
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Date: Thu, 25 Jun 2020 14:15:49 GMT
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Etag: g8TTwrRydbLOIDJmPI0u3Q==
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Server: ESF
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Vary: Origin
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Vary: X-Origin
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: Vary: Referer
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Content-Type-Options: nosniff
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Frame-Options: SAMEORIGIN
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: X-Xss-Protection: 0
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 6ab
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "kind": "bigquery#table",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "etag": "g8TTwrRydbLOIDJmPI0u3Q==",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "id": "tidal-turbine-222414:external.events_playfab_batched",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "selfLink": "https://www.googleapis.com/bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables/events_playfab_batched",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "tableReference": {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "projectId": "tidal-turbine-222414",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "datasetId": "external",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "tableId": "events_playfab_batched"
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "schema": {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "fields": [
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "AnalyticsEnvironment",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "PlayFabEnvironment",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "SourceType",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "Source",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventNamespace",
2020-06-25T16:15:49.778+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "TitleId",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "GroupBatchId",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchId",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventId",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventName",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
ugin.terraform-provider-google_v3.26.0_x5:     "name": "EntityType",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EntityId",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "Timestamp",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "ReceivedTimestamp",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchedTimestamp",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "TIMESTAMP"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "BatchJobName",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "name": "EventAttributes",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:     "type": "STRING"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    }
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   ]
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "numBytes": "0",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "numLongTermBytes": "0",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "numRows": "0",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "creationTime": "1593094549288",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "lastModifiedTime": "1593094549341",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "type": "EXTERNAL",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "externalDataConfiguration": {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "sourceUris": [
onl/event_schema=playfab/*"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   ],
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "sourceFormat": "NEWLINE_DELIMITED_JSON",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "ignoreUnknownValues": true,
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "compression": "GZIP",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   "hivePartitioningOptions": {
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "mode": "CUSTOM",
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:    "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:   }
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  },
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5:  "location": "US"
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: }
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 0
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: -----------------------------------------------------
2020-06-25T16:15:49.779+0200 [DEBUG] plugin.terraform-provider-google_v3.26.0_x5: 2020/06/25 16:15:49 [DEBUG] Retry Transport: Stopping retries, last request was successful
pts
2020/06/25 16:15:49 [WARN] Provider "registry.terraform.io/-/google" produced an unexpected new value for module.bigquery.google_bigquery_table.table_events_external[1], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .description: was null, but now cty.StringVal("")
      - .friendly_name: was null, but now cty.StringVal("")
      - .external_data_configuration[0].max_bad_records: was null, but now cty.NumberIntVal(0)
module.bigquery.google_bigquery_table.table_events_external[1]: Creation complete after 1s [id=projects/tidal-turbine-222414/datasets/external/tables/events_playfab_batched]
2020-06-25T16:15:49.784+0200 [DEBUG] plugin: plugin process exited: path=/Users/loek/analysts/scavengers-analytics/terraform/testing/.terraform/plugins/darwin_amd64/terraform-provider-google_v3.26.0_x5 pid=60856
2020-06-25T16:15:49.784+0200 [DEBUG] plugin: plugin exited

Warning: Applied changes may be incomplete

The plan was created with the -target option in effect, so some changes
requested in the configuration may have been ignored and the output values may
not be fully updated. Run the following command to verify that no other
changes are pending:
    terraform plan

Note that the -target option is not suitable for routine use, and is provided
only for exceptional situations such as recovering from errors or mistakes, or
when Terraform specifically suggests to use it as part of an error message.


Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Other one:

 bq --apilog "" mk --table --location=US --external_table_definition=/Users/loek/Desktop/events_playfab_testing_hive_partitioned_batched external.events_playfab_hive_partitioned_batched
I0625 16:20:13.090822 4451456448 bigquery_client.py:1114] Requesting discovery document from https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest
I0625 16:20:13.235872 4451456448 model.py:111] --request-start--
I0625 16:20:13.235983 4451456448 model.py:112] -headers-start-
I0625 16:20:13.236035 4451456448 model.py:114] accept: application/json
I0625 16:20:13.236080 4451456448 model.py:114] accept-encoding: gzip, deflate
I0625 16:20:13.236119 4451456448 model.py:114] user-agent: google-api-python-client/1.7.10 (gzip)
I0625 16:20:13.236156 4451456448 model.py:115] -headers-end-
I0625 16:20:13.236196 4451456448 model.py:116] -path-parameters-start-
I0625 16:20:13.236255 4451456448 model.py:118] tableId: events_playfab_hive_partitioned_batched
I0625 16:20:13.236299 4451456448 model.py:118] datasetId: external
I0625 16:20:13.236337 4451456448 model.py:118] projectId: tidal-turbine-222414
I0625 16:20:13.236373 4451456448 model.py:119] -path-parameters-end-
I0625 16:20:13.236410 4451456448 model.py:120] body: None
I0625 16:20:13.236511 4451456448 model.py:121] query: ?alt=json
I0625 16:20:13.236566 4451456448 model.py:122] --request-end--
I0625 16:20:13.236742 4451456448 discovery.py:868] URL being requested: GET https://bigquery.googleapis.com/bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables/events_playfab_hive_partitioned_batched?alt=json
I0625 16:20:13.307837 4451456448 transport.py:186] Refreshing due to a 401 (attempt 1/2)
I0625 16:20:13.308290 4451456448 reauth_creds.py:103] Refreshing access_token
I0625 16:20:13.581171 4451456448 model.py:182] --response-start--
I0625 16:20:13.581322 4451456448 model.py:184] vary: Origin, X-Origin, Referer
I0625 16:20:13.581402 4451456448 model.py:184] content-type: application/json; charset=UTF-8
I0625 16:20:13.581518 4451456448 model.py:184] date: Thu, 25 Jun 2020 14:20:13 GMT
I0625 16:20:13.581581 4451456448 model.py:184] server: ESF
I0625 16:20:13.581640 4451456448 model.py:184] cache-control: private
I0625 16:20:13.581696 4451456448 model.py:184] x-xss-protection: 0
I0625 16:20:13.581773 4451456448 model.py:184] x-frame-options: SAMEORIGIN
I0625 16:20:13.581852 4451456448 model.py:184] x-content-type-options: nosniff
I0625 16:20:13.582002 4451456448 model.py:184] alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
I0625 16:20:13.582065 4451456448 model.py:184] transfer-encoding: chunked
I0625 16:20:13.582122 4451456448 model.py:184] status: 404
I0625 16:20:13.582175 4451456448 model.py:184] content-length: 374
I0625 16:20:13.582231 4451456448 model.py:184] -content-encoding: gzip
I0625 16:20:13.582287 4451456448 model.py:186] b'{\n  "error": {\n    "code": 404,\n    "message": "Not found: Table tidal-turbine-222414:external.events_playfab_hive_partitioned_batched",\n    "errors": [\n      {\n        "message": "Not found: Table tidal-turbine-222414:external.events_playfab_hive_partitioned_batched",\n        "domain": "global",\n        "reason": "notFound"\n      }\n    ],\n    "status": "NOT_FOUND"\n  }\n}\n'
I0625 16:20:13.582353 4451456448 model.py:187] --response-end--
I0625 16:20:13.595350 4451456448 model.py:111] --request-start--
I0625 16:20:13.595453 4451456448 model.py:112] -headers-start-
I0625 16:20:13.595505 4451456448 model.py:114] accept: application/json
I0625 16:20:13.595548 4451456448 model.py:114] accept-encoding: gzip, deflate
I0625 16:20:13.595589 4451456448 model.py:114] user-agent: google-api-python-client/1.7.10 (gzip)
I0625 16:20:13.595628 4451456448 model.py:114] content-type: application/json
I0625 16:20:13.595666 4451456448 model.py:115] -headers-end-
I0625 16:20:13.595705 4451456448 model.py:116] -path-parameters-start-
I0625 16:20:13.595742 4451456448 model.py:118] datasetId: external
I0625 16:20:13.595781 4451456448 model.py:118] projectId: tidal-turbine-222414
I0625 16:20:13.595819 4451456448 model.py:119] -path-parameters-end-
I0625 16:20:13.595856 4451456448 model.py:120] body: {"tableReference": {"tableId": "events_playfab_hive_partitioned_batched", "datasetId": "external", "projectId": "tidal-turbine-222414"}, "externalDataConfiguration": {"autodetect": true, "hivePartitioningOptions": {"mode": "CUSTOM", "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"}, "ignoreUnknownValues": true, "schema": {"fields": [{"name": "AnalyticsEnvironment", "type": "STRING"}, {"name": "PlayFabEnvironment", "type": "STRING"}, {"name": "SourceType", "type": "STRING"}, {"name": "Source", "type": "STRING"}, {"name": "EventNamespace", "type": "STRING"}, {"name": "TitleId", "type": "STRING"}, {"name": "GroupBatchId", "type": "STRING"}, {"name": "BatchId", "type": "STRING"}, {"name": "EventId", "type": "STRING"}, {"name": "EventName", "type": "STRING"}, {"name": "EntityType", "type": "STRING"}, {"name": "EntityId", "type": "STRING"}, {"name": "Timestamp", "type": "TIMESTAMP"}, {"name": "ReceivedTimestamp", "type": "TIMESTAMP"}, {"name": "BatchedTimestamp", "type": "TIMESTAMP"}, {"name": "BatchJobName", "type": "STRING"}, {"name": "EventAttributes", "type": "STRING"}]}, "sourceFormat": "NEWLINE_DELIMITED_JSON", "sourceUris": ["gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"]}}
I0625 16:20:13.595925 4451456448 model.py:121] query: ?alt=json
I0625 16:20:13.595968 4451456448 model.py:122] --request-end--
I0625 16:20:13.596068 4451456448 discovery.py:868] URL being requested: POST https://bigquery.googleapis.com/bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables?alt=json
I0625 16:20:32.278898 4451456448 model.py:182] --response-start--
I0625 16:20:32.279092 4451456448 model.py:184] etag: kNHTC7HumZ1+PhWEA0jFTg==
I0625 16:20:32.279177 4451456448 model.py:184] content-type: application/json; charset=UTF-8
I0625 16:20:32.279252 4451456448 model.py:184] vary: Origin, X-Origin, Referer
I0625 16:20:32.279322 4451456448 model.py:184] date: Thu, 25 Jun 2020 14:20:32 GMT
I0625 16:20:32.279388 4451456448 model.py:184] server: ESF
I0625 16:20:32.279455 4451456448 model.py:184] cache-control: private
I0625 16:20:32.279521 4451456448 model.py:184] x-xss-protection: 0
I0625 16:20:32.279589 4451456448 model.py:184] x-frame-options: SAMEORIGIN
I0625 16:20:32.279654 4451456448 model.py:184] x-content-type-options: nosniff
I0625 16:20:32.279719 4451456448 model.py:184] alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
I0625 16:20:32.279788 4451456448 model.py:184] transfer-encoding: chunked
I0625 16:20:32.279853 4451456448 model.py:184] status: 200
I0625 16:20:32.279918 4451456448 model.py:184] content-length: 3633
I0625 16:20:32.279984 4451456448 model.py:184] -content-encoding: gzip
I0625 16:20:32.280052 4451456448 model.py:186] b'{\n  "kind": "bigquery#table",\n  "etag": "kNHTC7HumZ1+PhWEA0jFTg==",\n  "id": "tidal-turbine-222414:external.events_playfab_hive_partitioned_batched",\n  "selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/tidal-turbine-222414/datasets/external/tables/events_playfab_hive_partitioned_batched",\n  "tableReference": {\n    "projectId": "tidal-turbine-222414",\n    "datasetId": "external",\n    "tableId": "events_playfab_hive_partitioned_batched"\n  },\n  "schema": {\n    "fields": [\n      {\n        "name": "AnalyticsEnvironment",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "PlayFabEnvironment",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "SourceType",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "Source",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "EventNamespace",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "TitleId",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "GroupBatchId",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "BatchId",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "EventId",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "EventName",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "EntityType",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "EntityId",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "Timestamp",\n        "type": "TIMESTAMP",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "ReceivedTimestamp",\n        "type": "TIMESTAMP",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "BatchedTimestamp",\n        "type": "TIMESTAMP",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "BatchJobName",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "EventAttributes",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "event_category",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "event_environment",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "event_date",\n        "type": "DATE",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "event_hour",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      },\n      {\n        "name": "event_minute",\n        "type": "STRING",\n        "mode": "NULLABLE"\n      }\n    ]\n  },\n  "numBytes": "0",\n  "numLongTermBytes": "0",\n  "numRows": "0",\n  "creationTime": "1593094813736",\n  "lastModifiedTime": "1593094832076",\n  "type": "EXTERNAL",\n  "externalDataConfiguration": {\n    "sourceUris": [\n      "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"\n    ],\n    "sourceFormat": "NEWLINE_DELIMITED_JSON",\n    "autodetect": true,\n    "ignoreUnknownValues": true,\n    "hivePartitioningOptions": {\n      "mode": "CUSTOM",\n      "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}",\n      "fields": [\n        "event_category",\n        "event_environment",\n        "event_date",\n        "event_hour",\n        "event_minute"\n      ]\n    }\n  },\n  "location": "US"\n}\n'
I0625 16:20:32.280496 4451456448 model.py:187] --response-end--
Table 'tidal-turbine-222414:external.events_playfab_hive_partitioned_batched' successfully created.
LoekL picture LoekL on 25 Jun 2020

@ffung It appears CUSTOM has not been implemented in the https://github.com/GoogleCloudPlatform/magic-modules/pull/3335. Do you want to finish that? Thanks

edwardmedia picture edwardmedia on 25 Jun 2020

if you look at the bq payload, autodetect is set to true in this case instead of false for the terraform payload, can you rerun using bq with autodetect set to false and see if this also correctly creates the table?

Besides that I currently don't see any differences between the terraform payload and the bq one. 

{
  "tableReference": {
    "tableId": "events_playfab_hive_partitioned_batched",
    "datasetId": "external",
    "projectId": "tidal-turbine-222414"
  },
  "externalDataConfiguration": {
    "autodetect": true,
    "hivePartitioningOptions": {
      "mode": "CUSTOM",
      "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
    },
    "ignoreUnknownValues": true,
    "schema": {
      "fields": [
        {
          "name": "AnalyticsEnvironment",
          "type": "STRING"
        },
        {
          "name": "PlayFabEnvironment",
          "type": "STRING"
        },
        {
          "name": "SourceType",
          "type": "STRING"
        },
        {
          "name": "Source",
          "type": "STRING"
        },
        {
          "name": "EventNamespace",
          "type": "STRING"
        },
        {
          "name": "TitleId",
          "type": "STRING"
        },
        {
          "name": "GroupBatchId",
          "type": "STRING"
        },
        {
          "name": "BatchId",
          "type": "STRING"
        },
        {
          "name": "EventId",
          "type": "STRING"
        },
        {
          "name": "EventName",
          "type": "STRING"
        },
        {
          "name": "EntityType",
          "type": "STRING"
        },
        {
          "name": "EntityId",
          "type": "STRING"
        },
        {
          "name": "Timestamp",
          "type": "TIMESTAMP"
        },
        {
          "name": "ReceivedTimestamp",
          "type": "TIMESTAMP"
        },
        {
          "name": "BatchedTimestamp",
          "type": "TIMESTAMP"
        },
        {
          "name": "BatchJobName",
          "type": "STRING"
        },
        {
          "name": "EventAttributes",
          "type": "STRING"
        }
      ]
    },
    "sourceFormat": "NEWLINE_DELIMITED_JSON",
    "sourceUris": [
      "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"
    ]
  }
}
ffung picture ffung on 26 Jun 2020

@ffung see first post, bq mkdef seems to always set this to true, even if --autodetect isn't passed. That said, I've tried setting it to false and removing it altogether, each work.

Using "autodetect": false

{
  "autodetect": false,
  "hivePartitioningOptions": {
    "mode": "CUSTOM",
    "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
  },
  "ignoreUnknownValues": true,
  "schema": {
    "fields": [
      {
        "name": "AnalyticsEnvironment",
        "type": "STRING"
      },
      {
        "name": "PlayFabEnvironment",
        "type": "STRING"
      },
      {
        "name": "SourceType",
        "type": "STRING"
      },
      {
        "name": "Source",
        "type": "STRING"
      },
      {
        "name": "EventNamespace",
        "type": "STRING"
      },
      {
        "name": "TitleId",
        "type": "STRING"
      },
      {
        "name": "GroupBatchId",
        "type": "STRING"
      },
      {
        "name": "BatchId",
        "type": "STRING"
      },
      {
        "name": "EventId",
        "type": "STRING"
      },
      {
        "name": "EventName",
        "type": "STRING"
      },
      {
        "name": "EntityType",
        "type": "STRING"
      },
      {
        "name": "EntityId",
        "type": "STRING"
      },
      {
        "name": "Timestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "ReceivedTimestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "BatchedTimestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "BatchJobName",
        "type": "STRING"
      },
      {
        "name": "EventAttributes",
        "type": "STRING"
      }
    ]
  },
  "sourceFormat": "NEWLINE_DELIMITED_JSON",
  "sourceUris": [
    "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"
  ]
}

Screenshot 2020-06-26 10 16 57

Screenshot 2020-06-26 10 17 18

Removed

{
  "hivePartitioningOptions": {
    "mode": "CUSTOM",
    "sourceUriPrefix": "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING}"
  },
  "ignoreUnknownValues": true,
  "schema": {
    "fields": [
      {
        "name": "AnalyticsEnvironment",
        "type": "STRING"
      },
      {
        "name": "PlayFabEnvironment",
        "type": "STRING"
      },
      {
        "name": "SourceType",
        "type": "STRING"
      },
      {
        "name": "Source",
        "type": "STRING"
      },
      {
        "name": "EventNamespace",
        "type": "STRING"
      },
      {
        "name": "TitleId",
        "type": "STRING"
      },
      {
        "name": "GroupBatchId",
        "type": "STRING"
      },
      {
        "name": "BatchId",
        "type": "STRING"
      },
      {
        "name": "EventId",
        "type": "STRING"
      },
      {
        "name": "EventName",
        "type": "STRING"
      },
      {
        "name": "EntityType",
        "type": "STRING"
      },
      {
        "name": "EntityId",
        "type": "STRING"
      },
      {
        "name": "Timestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "ReceivedTimestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "BatchedTimestamp",
        "type": "TIMESTAMP"
      },
      {
        "name": "BatchJobName",
        "type": "STRING"
      },
      {
        "name": "EventAttributes",
        "type": "STRING"
      }
    ]
  },
  "sourceFormat": "NEWLINE_DELIMITED_JSON",
  "sourceUris": [
    "gs://tidal-turbine-222414-analytics-testing-batched/data_type=jsonl/event_schema=playfab/*"
  ]
}

Screenshot 2020-06-26 10 20 27

Screenshot 2020-06-26 10 20 44

I also successfully ran queries against both tables.

LoekL picture LoekL on 26 Jun 2020

I also just tried adding "compression": "GZIP", to the bq definition file (for which bq mkdef does not have an option yet) and using bq mk ..., again works.

LoekL picture LoekL on 26 Jun 2020

@LoekL thanks for the info, I had a closer look at the payloads and there's a difference, bq uses the nested schema object in externalDataConfiguration instead of the toplevel one, apparently this is only applicable for CSV and JSON. I'll see if I can try to create a fix for this this, but no guarantees, we don't make use of this and this is not my day-job ;-).

ffung picture ffung on 26 Jun 2020
👍1

I see, so it should work if we move schema = ... into the external_data_configuration {} block? When I try this now I get An argument named "schema" is not expected here. Fingers crossed for the fix 🤞

LoekL picture LoekL on 26 Jun 2020

I see, so it should work if we move schema = ... into the external_data_configuration {} block? When I try this now I get An argument named "schema" is not expected here. Fingers crossed for the fix 🤞

yes, the terraform schema for external_data_configuration doesn't support the schema field yet.

ffung picture ffung on 26 Jun 2020
👍1

@LoekL https://github.com/GoogleCloudPlatform/magic-modules/pull/3717

ffung picture ffung on 29 Jun 2020
🎉1

Seems like the docs for this didn't make it to the Terraform docs site? It's not showing it's possible to specify schema under external_data_configuration block

preston-hf picture preston-hf on 27 Jul 2020

Try a hard refresh? I see it there.
mxiaraTGC3r

danawillow picture danawillow on 27 Jul 2020

It's not clear what it was but it's there now. Thanks for looking into it.

preston-hf picture preston-hf on 27 Jul 2020

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

hashibot[bot] picture hashibot[bot] on 9 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Preete picture Preete  ·  28Comments
shaulishaked picture shaulishaked  ·  24Comments
Lirt picture Lirt  ·  33Comments
pdaborowski-playtech picture pdaborowski-playtech  ·  24Comments
EssentialMix picture EssentialMix  ·  23Comments