nomad-driver-containerd: Issue with ExecTask API

[shishir-a412ed]

Hi,

I am writing a custom nomad driver for containerd.
More details in #5417

All the other API's are working fine, however, I am not able to reach ExecTask API.

Since we are still in the process of open-sourcing this project, I reproduced this issue in nomad-skeleton-driver-project

I did the following:

1) In the StartTask I changed the command from echo to sleep so that I can keep the job running, and try exec.

echoCmd := fmt.Sprintf(`sleep "%s"`, driverConfig.Greeting)

2) Added a logger in ExecTask and don't return an error.

func (d *HelloDriverPlugin) ExecTask(taskID string, cmd []string, timeout time.Duration) (*drivers.ExecTaskResult, error) { d.logger.Info("HELLO: ExecTask") return nil, nil }

I just wanted to see if ExecTask is being called when I exec.

3) nomad job run example.nomad

````
job "example" {
datacenters = ["dc1"]
type = "batch"

group "example" {
task "hello-world" {
driver = "hello-world-example"

  config {
    greeting = "360s"
  }
}

}
}
````

4) Once the job is running:
$ nomad alloc exec -i -t <alloc_id> /bin/bash

Error: Failed to exec into the task: RPC error: code = Unknown desc = driver does not support exec
I also see the same error logged in the nomad logs.

I was hoping to at least hit ExecTask API and see the logger (HELLO: ExecTask) printed.

[shishir-a412ed]

ping @notnoop @tgross

[notnoop]

This is very exciting! Cannot wait to see it open-sourced!

To support nomad exec command, you'll need to implement drivers.ExecTaskStreamingDriver (or ExecTaskStreamingRawDriver that's more complicated). Docker offers a sample implementation in https://github.com/hashicorp/nomad/blob/v0.11.2/drivers/docker/driver.go#L1424-L1511 .

Let me know if that helps!

[shishir-a412ed]

@notnoop Thanks for the quick response!

Just to be clear, My issue is not on how to implement Exec, but that I am not even able to reach ExecTask (HELLO: ExecTask not getting printed) when I do nomad alloc -i -t <alloc_id> /bin/bash

Once I hit the API, I was planning to use containerd task.Exec since containerd is the supervisor process and should be handling the exec for that container.

Are you saying I need to implement drivers.ExecTaskStreamingDriver to even hit ExecTask from nomad client?

[notnoop]

Correct. For nomad exec CLI invocation, nomad invokes ExecTaskStreaming, not ExecTask. ExecTask is used for performing task service checks instead.

We'll update the documentation and consider unifying these two functions so only one is needed.

[shishir-a412ed]

@notnoop Thanks! that works, and I can hit ExecTaskStreaming when I do nomad alloc -i -t <alloc_id> /bin/bash.

Making the actual exec work will be a little more complicated :)
nomad docker driver exec implementation might not be helpful here as it uses go-dockerclient which is just reaching out to docker daemon exec endpoint.

Since our case is to bypass the docker completely, I cannot use the docker daemon exec endpoint. I am exploring how containerd or docker implements exec, and see if I could use some of that.

For now, Is it alright if I keep this issue open? So If I have any more questions related to ExecTask I can ask here. Once I have a working version, I will close this issue myself.

[bithavoc]

@shishir-a412ed I’m rooting for you and this driver, I wasn’t counting on having exec support for Containerd, damn!!! 🚀

[shishir-a412ed]

I have a working version of exec. Closing the issue now.