Terraform: Literal hexadecimal notation isn't properly converted by 0.12upgrade

[ebekker]

Terraform Version

Terraform v0.12.0-beta1

Terraform Configuration Files

## No need for any providers, can simulate
## the issue with a variable and an output;
## this works in 0.11.x but breaks in latest 0.12
variable "v1" {
    default = 0x99
}
output "input-v1" {
    value = "${var.v1}"
}

Expected Behavior

Should have parsed the 0x99 hexadecimal value as decimal value 153.

Actual Behavior

Got a parse error:

Error: Missing newline after argument

  on 000-main.tf line 3, in variable "v1":
   3:     default = 0x99

An argument definition must end with a newline.

Steps to Reproduce

This works with TF 0.11.* but produces an error with latest TF 0.12 release (beta1)

terraform init
terraform apply

[mildwonkey]

Hi @ebekker !
Thank you for opening this issue.
I'd like to ask you a little more about your configuration. I wasn't able to get your example working in terraform 0.11 without putting quotes around the hex value:

variable "v1" {
  default = "0x99"
}

output "input-v1" {
  value = "${var.v1}"
}

Next, I ran terraform 0.12upgrade (with terraform 0.12-beta), which only changed the variable reference in the output:

$ cat hex.tf
variable "v1" {
  default = "0x99"
}

output "input-v1" {
  value = var.v1
}

The output from terraform apply:

Outputs:

input-v1 = 0x99

This leads to two questions: Did you have a configuration that worked without quotes around the hex value? Does quoting the hex value and continuing with your terraform 0.12 apply produce the result you expect?

[ebekker]

Yes, without quotes, I'm on 0.11.13. As per the Syntax section of the Configuration Language Guide:

Numbers are assumed to be base 10. If you prefix a number with 0x, it is treated as a hexadecimal number.

Here is a sample session, please note, I'm on Windows:

PS C:\XXXXX\terrascape\src\Terrascape.WinLocalProvider\tf1> ls


    Directory: C:\XXXXX\Terrascape.WinLocalProvider\tf1


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----         4/4/2019   2:46 PM             89 000-main.tf

PS C:\XXXXX\terrascape\src\Terrascape.WinLocalProvider\tf1> cat .\000-main.tf
variable "v1" {
    default = 0x99
}
output "input-v1" {
    value = "${var.v1}"
}



md5-be88dbe12f2c949192874618f3df73f5



```pwsh
PS C:\XXXXX\terrascape\src\Terrascape.WinLocalProvider\tf1> C:\local\bin\terraform.exe apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

input-v1 = 153

[ebekker]

Just did a quick test on Ubuntu (under WSL) and confirmed the same findings using 0.11.13 and 0.12-beta1:

ebekker@XXXXX:/mnt/c/XXXXX/terrascape/src/Terrascape.WinLocalProvider/tf1$ ~/terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

input-v1 = 153

ebekker@XXXXX:/mnt/c/XXXXX/terrascape/src/Terrascape.WinLocalProvider/tf1$ ~/terraform012 apply

Error: Missing newline after argument

  on 000-main.tf line 2, in variable "v1":
   2:     default = 0x99

An argument definition must end with a newline.

[ebekker]

If you're finding different results with 0.11 and you're not using the latest (0.11.13), let me know, and I can confirm on my end.

[mildwonkey]

Ah, the error was on my end. I have a few too many versions of terraform floating around, apologies.
Just as you said (and thank you for confirming!), this is working in terraform 0.11.13:

variable "v1" {
  default = 0x99
}

output "input-v1" {
  value = "${var.v1}"
}

$ terraform apply -auto-approve

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

input-v1 = 153

In terraform 0.12, numbers can only be digits, so we would expect to see hexidecimal values wrapped in strings (default = "0x99"). I will flag this as both a potential bug for the configupgrade tool and a documentation issue. Thanks again!

[ebekker]

So if a hex number is represented as a string in a TF config, does that mean that it is the provider's responsibility to parse the string as a hex number? (before with 0.11, TF was supplying the integer value, e.g. an integer with a decimal value of 153)

[ebekker]

Also, how would that work if the hex value is embedded as part of an expression (new feature in 0.12)?

For example, I could do:

variable "foo" {
  default = 3 + 3
}

And that would correctly resolve to 6, but I couldn't do that if the number value was in a quoted hex representation. Is this just not supported?

[apparentlymart]

Hi @ebekker,

The language in Terraform 0.12 is an entirely new implementation based on the foundations of the two libraries that powered the language in prior versions: HCL and HIL. In merging the two languages to form this new language, we also sought to simplify it by consolidating around the most commonly-used features and patterns, as a way to somewhat balance with the additional complexity created by the new features.

The ability to specify numbers in bases other than decimal was one of the possibilities that is intentionally no longer present, and the terraform 0.12upgrade tool is intended to recognize numbers in other bases and convert them to decimal for use with Terraform 0.12. If that isn't working then that'll be a bug to fix!

I notice also that the language parser is giving a poor error message for attempts to use the hex syntax: it's considering 0x33 to be the number 0 followed by the variable x33, which is a syntax error. We can and should improve this by having the parser produce a more specific error for this case, but I expect we will address that in a subsequent release since v0.12.0 is nearing completion and so we're trying to keep changes to a minimum.

For the rare situation where a number value is more readable in a different base -- such as Unix-style file modes, which are conventionally written in Octal -- we recommend that provider developers design the provider to expect a string containing digits in the relevant base, which therefore ensures that the value shown in output such as from terraform output or terraform plan will also be reflected in the same base. As you've seen in the examples shared above, with numbers given in hex in previous versions they would appear in output in decimal, thus defeating whatever motivated writing those numbers in a different base in the first place.

That _does_ require special support on the part of the provider. If a provider specifies that an argument has type number then values for that argument must be specified either directly as a number (written in decimal in the language) or as a string containing decimal digits which will then be converted to a number. Generally I would expect a provider to choose a single intuitive string representation accepted for a value if a decimal representation is not sufficient, similarly to how providers will expect dotted-decimal IPv4 addresses without any automatic conversion from a 32-bit integer available.

If the terraform 0.12upgrade command isn't automatically rewriting your 0x99 as 153, please let me know; we'd love to fix that up before the final v0.12.0 release, if it's not working.

[ebekker]

Sounds good, and perfectly reasonable, just wasn't aware of this particular config language change for 0.12.

I tested out the 0.12upgrade command and the resulting config file produced was incorrect.

So to be clear, given this input config:

variable "v1" {
    default = 0x99
}
output "input-v1" {
    value = "${var.v1}"
}

the upgrade command produced this:

variable "v1" {
  default = 0 x99
}

output "input-v1" {
  value = var.v1
}

[ebekker]

Since you've addressed my concern with the config notation, I've renamed the issue to address the upgrade error.

[apparentlymart]

Thanks for checking that, @ebekker! Looks like there's a missing conversion rule there. We'll get that fixed up.

[apparentlymart]

Looks like the problem is here:

https://github.com/hashicorp/terraform/blob/ac2052f0bbcedb8bb5c54fd0e8dc5adff9d31825/configs/configupgrade/upgrade_expr.go#L146-L148

It's assuming that any valid NUMBER (really: integer) token from HCL 1 is byte-for-byte compatible with HCL 2, which is not true. Instead, it will need to call Token.Value and then format the result using something like strconv.FormatInt. While we're there writing that anyway, we might as well also do similarly for the FLOAT token type using [strconv.FormatFloat]https://godoc.org/strconv#FormatFloat, just to ensure the result is normalized in all cases.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.