Terraform: Terraform Remote-Exec Provisioner Glitch on User Switch

[Tensho]

Terraform Version

v0.11.11

Terraform Configuration Files

provider "aws" {
  ...
}

resource "aws_instance" "precious" {
  ...
  provisioner "remote-exec" {
    connection {
      user = "ubuntu"

      bastion_host = "100.100.100.100"
      bastion_user = "bastion-user"
    }

    inline = [
      "pwd",
      "whoami",
      "sudo su deploy",
      "pwd",
      "whoami"
    ]
  }
}

Output

...
module.instance.aws_instance.instance: Still creating... (10s elapsed)
module.instance.aws_instance.instance: Still creating... (20s elapsed)
module.instance.aws_instance.instance: Still creating... (30s elapsed)
module.instance.aws_instance.instance: Still creating... (40s elapsed)
module.instance.aws_instance.instance: Provisioning with 'remote-exec'...
module.instance.aws_instance.instance (remote-exec): Connecting to remote host via SSH...
module.instance.aws_instance.instance (remote-exec):   Host: 100.100.100.100
module.instance.aws_instance.instance (remote-exec):   User: ubuntu
module.instance.aws_instance.instance (remote-exec):   Password: false
module.instance.aws_instance.instance (remote-exec):   Private key: false
module.instance.aws_instance.instance (remote-exec):   SSH Agent: true
module.instance.aws_instance.instance (remote-exec):   Checking Host Key: false
module.instance.aws_instance.instance (remote-exec): Using configured bastion host...
module.instance.aws_instance.instance (remote-exec):   Host: 200.200.200.200
module.instance.aws_instance.instance (remote-exec):   User: bastion-user
module.instance.aws_instance.instance (remote-exec):   Password: false
module.instance.aws_instance.instance (remote-exec):   Private key: false
module.instance.aws_instance.instance (remote-exec):   SSH Agent: true
module.instance.aws_instance.instance (remote-exec):   Checking Host Key: false
module.instance.aws_instance.instance: Still creating... (50s elapsed)
module.instance.aws_instance.instance (remote-exec): Connected!
module.instance.aws_instance.instance (remote-exec): /home/ubuntu
module.instance.aws_instance.instance (remote-exec): ubuntu
module.instance.aws_instance.instance (remote-exec): deploy@ip-100-100-100-100:/home/ubuntu$
module.instance.aws_instance.instance: Still creating... (1m0s elapsed)
module.instance.aws_instance.instance: Still creating... (1m10s elapsed)
module.instance.aws_instance.instance: Still creating... (1m20s elapsed)
module.instance.aws_instance.instance: Still creating... (1m30s elapsed)
module.instance.aws_instance.instance: Still creating... (1m40s elapsed)
module.instance.aws_instance.instance: Still creating... (1m50s elapsed)
module.instance.aws_instance.instance: Still creating... (2m0s elapsed)
...

Debug Ouput

https://gist.github.com/Tensho/190a3b8e2ed32e45614252a349c7c899

Expected Behavior

Remote shell provisioner successfully switches to deploy user and proceed with other commands described in inline array.

Actual Behavior

Stuck (waiting endlessly) right after user switch in sudo su deploy command.

Additional Context (Considerations)

If I understand correctly, Terraform packs all inline commands to one script as a temp file, uploads it to the server and runs on behalf of user described in provisioner connection block (in my case ubuntu). That means the shell context should be the same for all described inline commands and I may freely switch user (su) and run subsequent commands on behalf of it (in my case deploy). Is it correct?

[apparentlymart]

Hi @Tensho,

The command sudo su ... creates a child shell, which in your case is then showing its interactive prompt and waiting for further input. The rest of the script never runs because the child shell doesn't exit.

As far as I know, there isn't a way to ask the shell to switch to a different user in-process and then continue executing the same script: any command you run that changes user will create a child process, which will either block the rest of the script from executing until it returns or replace the script process entirely (if you were to use exec, for example.)

For situations like this I would probably either just run each subsequent command itself in sudo, or use the file provisioner to upload a separate script file and then use remote-exec to sudo-execute _that_ script.

This is not really a Terraform-specific problem; if you run the commands you shared as a script directly on the remote host I expect you'll see the same behavior. I'd suggest developing your script and testing it directly (by running it yourself from the shell on a similarly-configured host) before having Terraform execute it, since it'll be easier to get feedback on how the script is behaving when you can interact directly any interactive prompts it might display, etc.

[Tensho]

@apparentlymart Thank you for the thorough explanation. Agree, the question is not related to Terraform itself. Tested with file + remote-exec variant as you suggested and everything works as expected.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.