Terraform-provider-cloudflare: cloudflare_filter not respecting $TF_VAR_cloudflare_email or $TF_VAR_cloudflare_token

[pieterza]

Current TF version
0.12.29

Providers:

$ terraform -v
Terraform v0.12.29

• provider.aws v2.70.0
• provider.cloudflare v2.13.1
• provider.helm v1.3.2
• provider.kubernetes v1.13.3
• provider.null v2.1.2
• provider.random v2.1.2
• provider.template v2.1.2

Error received:

11:11:12  
11:11:12  Error: cloudflare_api_key and cloudflare_email are required for validating filter expressions but they are missing
11:11:12  
11:11:12    on .terraform_base/modules/bot_blacklist_REDACT_vn/main.tf line 5, in resource "cloudflare_filter" "bot_blacklist_filter":
11:11:12     5: resource "cloudflare_filter" "bot_blacklist_filter" {
11:11:12  
11:11:12  
11:11:12  
11:11:12  Error: cloudflare_api_key and cloudflare_email are required for validating filter expressions but they are missing
11:11:12  
11:11:12    on .terraform_base/modules/bot_blacklist_REDACT_mm/main.tf line 5, in resource "cloudflare_filter" "bot_blacklist_filter":
11:11:12     5: resource "cloudflare_filter" "bot_blacklist_filter" {
11:11:12  
11:11:12  

We have our cloudflare provider set to automatically upgrade version >2:

provider "cloudflare" {
  version = "~> 2.0"

Our ENV sets:
$TF_VAR_cloudflare_email
$TF_VAR_cloudflare_token

This is working with version v2.12.0, pointing to a bug in v2.13.x

[kravvcu]

We set our cloudflare_token directly in the provider configuration

provider "cloudflare" {
  api_token = data.google_secret_manager_secret_version.cloudflare_token.secret_data
}

Works for 2.11, 2.12. Fails for 2.13.x. I presume it's the same issue. The description mentions only TF_VAR usage, which I guess might be pointing in a different direction (? I'm unfamiliar with the code.).

[pieterza]

We set our cloudflare_token directly in the provider configuration

provider "cloudflare" {
  api_token = data.google_secret_manager_secret_version.cloudflare_token.secret_data
}

Works for 2.11, 2.12. Fails for 2.13.x. I presume it's the same issue, but the description mentions only TF_VAR applications (which I guess might be pointing in a different direction; I'm unfamiliar with the code).

It does work for 2.13.x if I set it manually as follow:

variable "cloudflare_email" {
  type    = string
  default = "xxxx"
}

variable "cloudflare_token" {
  type    = string
  default = "xxxxx"
}

But we'd obviously like to keep the TF_var usage that works up to 2.12.0

[pieterza]

It does seem limited to filter calls resource "cloudflare_filter" too.

[kravvcu]

We set our cloudflare_token directly in the provider configuration

provider "cloudflare" {
  api_token = data.google_secret_manager_secret_version.cloudflare_token.secret_data
}

Works for 2.11, 2.12. Fails for 2.13.x. I presume it's the same issue, but the description mentions only TF_VAR applications (which I guess might be pointing in a different direction; I'm unfamiliar with the code).

It does work for 2.13.x if I set it manually as follow:

variable "cloudflare_email" {
  type    = string
  default = "xxxx"
}

variable "cloudflare_token" {
  type    = string
  default = "xxxxx"
}

But we'd obviously like to keep the TF_var usage that works up to 2.12.0

Something is off then..
The docs clearly state (and I believe I've checked this in the past in the Cloudflare docs) that it should be either:

• cloudflare api key + email
  or
• cloudflare token

So cloudlfare token + email doesn't make sense (?).

[aidenvaines-bjss]

Can confirm. My Cloudflare provider crentials are sourced from an AWS SSM parameter like

provider "cloudflare" {
  email   = data.aws_ssm_parameter.cloudflare_api_auth_email.value
  api_key = data.aws_ssm_parameter.cloudflare_api_auth_key.value
}

this worked fine in 2.12.0 when last run on 04-Nov-2020 09:31. This morning with 2.13.1 I now get the message stated by others in thread.

Error: cloudflare_api_key and cloudflare_email are required for validating filter expressions but they are missing

  on cloudflare_filter.tf line 1, in resource "cloudflare_filter" "resource-name":
   1: resource "cloudflare_filter" "resource-name" {

It seems limited to cloudflare_filter resources

[pieterza]

We set our cloudflare_token directly in the provider configuration

provider "cloudflare" {
  api_token = data.google_secret_manager_secret_version.cloudflare_token.secret_data
}

Works for 2.11, 2.12. Fails for 2.13.x. I presume it's the same issue, but the description mentions only TF_VAR applications (which I guess might be pointing in a different direction; I'm unfamiliar with the code).

It does work for 2.13.x if I set it manually as follow:

variable "cloudflare_email" {
  type    = string
  default = "xxxx"
}

variable "cloudflare_token" {
  type    = string
  default = "xxxxx"
}

But we'd obviously like to keep the TF_var usage that works up to 2.12.0

Something is off then..
The docs clearly state (and I believe I've checked this in the past in the Cloudflare docs) that it should be either:

• cloudflare api key + email
  or
• cloudflare token

So cloudlfare token + email doesn't make sense (?).

Only setting cloudflare_token leads to:

Error: email is not set correctly

  on _providers.tf line 59, in provider "cloudflare":
  59: provider "cloudflare" {

So I agree it doesn't make sense in this case.

But setting the ENV's was working up to 2.13.0 as per others in thread.
And it's only broken on cloudflare_filter resource.

[kravvcu]

@pieterza For me (only cloudflare_token set) it doesn't fail on the provider configuration, rather later on the cloudflare_filter resource

Error: cloudflare_api_key and cloudflare_email are required for validating filter expressions but they are missing

Weird that we have different errors.

[addysidda]

We are also seeing this error, but in our case we are storing and pulling the API key and email in vault.

Error: cloudflare_api_key and cloudflare_email are required for validating filter expressions but they are missing

It worked when we changed the version to 2.12.

[jacobbednarz]

Thanks for raising this one. The issue you've raised is indeed related to cloudflare_filter which in v2.13.0 got an update to how the expression is validated (#848). As a part of that change, it makes a call to the Cloudflare API to validate the expression. Unfortunately, the way the request is built is unique to that scenario due to how what is available to ValidateFunc so the client is stand alone and the values are pulled only from the environment variables which appears to be an oversight in how many people are using that approach. To make matters a little more confusing, the endpoint isn't supported by API tokens so an API key/email combination is required to perform it.

Once 2.13.0 was released, I noticed on our internal test suite a bunch of new errors which later turned into the fix #860 to propagate the exception properly to the end-user.

I'm in the process of seeing if we can reuse the existing client which will allow TF_VAR_..., environment variables and the provider configuration blocks to work however at this stage, it will still require API key/email combo to make the call. Once the endpoint supports API tokens, we can remove this specific check.

To doubley check this hasn't changed in the last week, I also just confirmed API token support is still missing.

$ curl -s -H "Authorization: Bearer $CLOUDLARE_API_TOKEN" "https://api.cloudflare.com/client/v4/filters/validate-expr?expression=ip.src==34" | jq
{
  "success": false,
  "errors": [
    {
      "code": 10000,
      "message": "API Tokens are not supported by this API for now"
    }
  ]
}

[edragic]

I was not clear on the intended follow up based on your comment @jacobbednarz, but it sounds like your proposal is to continue forwards with this breaking change to allowed authentication method?

What is the feasibility of returning to a method which allows token usage until this new endpoint supports that access?

[jacobbednarz]

I started digging into a workaround for this one and managed to find an approach that Terraform allows, however, would leave us supporting some functionality that Terraform core have indicated they aren't keen on continuing and will eventually end up deprecated. Instead of pursuing that, I'm going to revert #848 and #860 in #863 until such time we aren't managing a standalone client specifically for filters and can instead reuse the existing client with the endpoint supporting all authentication methods. This also means #846 is reopened for tracking the feature.

What is the feasibility of returning to a method which allows token usage until this new endpoint supports that access?

@edragic I'm afraid there isn't a state (previous or proposed) that would allow only token use. The endpoint simply doesn't support it hence why API key/email was originally required. I've raised 2017235 with Cloudflare Support to see if we can get an ETA on the support of this to revisit this in #846.

is there any way of reverting to the previous structure to avoid breaking production workflows until the filters API
[...] or at least utilise the configuration in the tf provider resource

@aidenvaines-bjss I covered it in a bit more detail in a previous comment and at the beginning of this comment but the gist is that the inbuilt client isn't exposed to the schema validation methods -- they currently only have access the old and new values so it wasn't possible hence the standalone client. On a somewhat tangential note, I'd highly recommend not automatically using the latest of any piece of software as a part of your production workflow. All software eventually encounters edge cases so it's best to pin to a known good version and increment after confirming working elsewhere. If you want to run bleeding edge (such as we do), have a duplicate pipeline/workflow that runs the latest as a litmus test.

863 is automatically going to close this issue and a new release will be available shortly but feel free to ask any questions you have and I'll respond where I can.