Terraform: Starting commands with screen or nohup are not kept running.

[snorberhuis]

I provision several machines using Terraform. I also want to use Terraform to start the service it has to provide using ssh and a command, so that I not need something else to do this. The behaviour I expect is that Terraform is able to do this using screen or nohup:

provisioner "remote-exec" {
inline = "nohup sudo command &"
}

or

provisioner "remote-exec" {
 inline = "screen -d -m sudo command"
}

But these commands are not kept running for no apparent reason. I checked the machines and the commands are not kept running. When I ssh manually and execute the same command the commands are kept running.

You can reproduce this simply by having a terraform config with these provisioning commands and a command to your liking.

Why can this not be done and is this a bug?

[jangrewe]

This fix worked for me: http://stackoverflow.com/questions/36207752/how-can-i-start-a-remote-service-using-terraform-provisioning

[mitchellh]

That looks like a decent workaround.

[simpsonjon]

I'm still seeing this issue. Sleeping for a few seconds is not working for me. I can see the command started but it was killed the moment the provisioner finished that line of remote-exec.

[jangrewe]

@simpsonjon just sleep probably won't be sufficient, as all processes will be killed on disconnect.

[simpsonjon]

@jangrewe Do you have a suggestion on how to handle running a command through screen/nohup and having terraform allow it to keep running?

[jangrewe]

@simpsonjon the link i posted above did work fine for me, but maybe you're doing something that won't work this way. you should probably show what you're trying to do.

[simpsonjon]

I'm in the process of learning terraform so I have a small app deploy a scaling group and performs a load test.

inline = [
            "sudo apt-get update",
            "curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -",
            "sudo apt-get install -y nodejs",
            "cd /opt/app",
            "npm install",
            "sudo npm install -g forever",
            "nohup forever main.js --url=http://${aws_elb.web.dns_name} &",
            "sleep 1",
        ]

connection {
            type = "ssh"
            user = "ubuntu"
}

I've also tried replacing 'sleep1', with another apt-get update call or a longer sleep. Neither worked.

[snorberhuis]

@simpsonjon I have been working on a lot of processes detaching and the same problems keep coming up.

Some problems arise if you try to write to a stdout or stderr. These streams might be destroyed after detaching resulting in error's in your code. Some examples of this occurring might be simple prints.

[kgruszka]

Shouldn't it be reopened? I have the same problem with

inline = [
  "sudo nohup npm start &",
  "sleep 1"
]

this works fine:

inline = [
  "sudo forever start index.js"
]

[ghmulti]

@simpsonjon sleep did not work for me either, adding local-exec after remote-exec and run nohup over ssh manually worked for me though:

provisioner "remote-exec" {
  inline = [
    ...
  ]
  connection {
    type = "ssh"
    user = "ec2-user"
    private_key = "${file(var.aws_key_path)}"
  }
}

provisioner "local-exec" {
  command = "ssh -o 'StrictHostKeyChecking no' -i ${var.aws_key_path} ec2-user@${self.public_ip} 'nohup some-command </dev/null >/dev/null 2>&1 &'"
}

[omarabid]

@mitchellh The sleep "work-around" is not decent. This should be considered as a bug since it is not the expected behavior.

[ghost]

The problem is still present. Looks like for a workaround it is possible to create and enable systemD service which will handle command. See example here https://github.com/vfreefly/ec2_proxies/blob/master/setup.sh

[kaizenlabs]

Is anybody on track to open a PR to fix this? Terraform is pretty pointless as an automated provisioning system if you can't run processes as daemons as you need to ssh manually to restart them. Also why is this issue closed? Definitely not a closed bug.

[VanTamNguyen]

I'm still seeing this issue in August of 2018. Terraform please fix this!

[kaizenlabs]

Agreed, please fix. Terraform cannot be used on GCP in a production environment right now.

[delirius325]

This is still an issue and @omarabid is right, the workaround shouldn't be considered as the right method.

[juanjoserivera]

@mitchellh This is definitely an unexpected behavior, while "sleep 1" workaround work for me, shouldn't be standard solution.

[trihoangvo]

Sleep 1 does not work for npm start. Anyone have a solution that works with npm?
If I use forever, I have to specify a js script. And I have to parse the package.json (which is known by npm) for the script.

[mgarstecki]

I managed to get this to work using a disown after the nohup ... command (as per https://superuser.com/a/705448):

nohup ... &; disown

I specifically use bash as interpreter in my commands.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.