Terraform: Freeze on Windows when applying with apt provision

[george-richardson]

Terraform Version

Terraform v0.11.7
+ provider.aws v1.19.0

I am running on Windows 10 in Powershell 5.1.17134.48

Terraform Configuration Files

resource "aws_instance" "failing_provision" {
  ami                         = "ami-5718412e" # Ubuntu 17.10 in eu-west-1
  instance_type               = "t2.micro"
  vpc_security_group_ids      = ["..."]
  subnet_id                   = "..."
  key_name                    = "..."
  associate_public_ip_address = true

  connection {
    type        = "ssh"
    host        = "${aws_instance.failing_provision.private_ip}"
    user        = "ubuntu"
    private_key = "..."
  }

  provisioner "remote-exec" {
    inline = ["set -ex && sleep 20 && sudo apt update && sudo apt install -y pigz"]
  }
}

Debug Output

N/A

Explained below in additional context.

Crash Output

N/A

Expected Behavior

Terraform provision a Ubuntu 17.10 instance and installs pigz

Actual Behavior

Terraform gets to:

aws_instance.failing_provision (remote-exec): Fetched 55.3 kB in 0s (0 B/s)
                                                Selecting previously unselected package pigz.

Where it will hang permanently. Only killing the terraform process through task manager will allow you to continue with the session. Terraform state will not be updated with the failed provision.

Steps to Reproduce

1. Get Terraform on Windows and open powershell.
2. apply the above configuration or another one where apt install is used.
3. Wait for it to hang when it starts doing the actual install.

Additional Context

Interestingly the script will continue to run in the background and complete. You can confirm this by connecting to the new machine and seeing if the package is installed. The hang seems to occur the first time a carriage return is produced in the output so the issue may be related to that, however I was unable to reproduce this with a printf '\r' or similar.

Furthermore, I discovered when writing this issue that redirecting the output of the command (to create the trace file) seems to workaround the freeze and let Terraform complete properly e.g. terraform apply *> output.log. If $env:TF_LOG="debug" is set and run without redirection it will fail in the exact same way. The trace is larger than my terminal buffer so I didn't submit it. If needed I can see if I can workout a way of getting it without redirection.

[tuwid]

@george-richardson Keep in mind that aws instance on boot up have this automatic 'apt-get update' that might get your apt things locked. It happens randomly after a couple of minutes of the machine has started ( the random part is by design ).
We do something like this on AMI creation so we wont have to deal this this issue:

systemctl stop apt-daily-upgrade.service
systemctl disable apt-daily-upgrade.service
systemctl kill apt-daily-upgrade.service
systemctl daemon-reload

[george-richardson]

@tuwid Yes I am aware. This configuration was cut down from larger configuration which accounts for that, I added the sleep 20 here as a way of avoiding this problem while keeping the reproduction simple. This is not the problem however as the script does complete on the remote host, it is only the terraform client that freezes.

[apparentlymart]

Hi @george-richardson! Sorry this isn't working properly.

Your theory about the carriage return character gave me a different idea: I wonder if apt is producing progress indicators with VT100 terminal escape sequences that are somehow confusing Terraform's CLI output. The Windows build of Terraform has some code that attempts to emulate VT100 escape sequences to adapt to the Windows Console API, and it's possible that this filtering subsystem is not fully robust against all possible escape sequences.

Could you try running apt-get with the -q option to disable the ongoing progress messages and produce just the "log"-type output? If my theory is correct, I think that will allow the process to complete without hanging.

Thanks!

[george-richardson]

Hi @apparentlymart Using the -q option with apt-get does allow the apply to continue successfully. Thanks for the help!

[apparentlymart]

Great! I'm glad that workaround was helpful.

That also gives us a clue for where to look for the solution: making sure that terminal escape codes returned by provisioners don't get printed out into the logs where they can then upset the VT100-to-Windows-Console emulation. Thanks for following up!

[bflad]

This downstream provider issue seems very related to this issue and has some great information/screenshots. Specifically, its yum -y install with what looks like some color escaping sequence issue: https://github.com/terraform-providers/terraform-provider-aws/issues/3293

[NickMRamirez]

Maybe when it says WARNING: apt does not have a stable CLI interface. Use with caution in scripts. it means it? :-)

I was struggling with this same behavior until I directed all apt output to /dev/null (for both apt update and apt install).