Swarmkit: Infinite loop with incorrect image name

[FrenchBen]

Expected

When I provide an image to swarmkit that is an incorrect image, I should get an error indicating that this image doesn't exist.

Actual

Swarmkit ends up in an endless loop of errors and retries

Steps:

Create a service with an image name typo: (notice an ID is generate)

$ swarmctl service create --name redis --image redi 
a2ffgwgdu88ntjqtu9bjt5izv

Try to look at the service ls: (notice empty replicas)

$ swarmctl service ls
ID                         Name   Image  Replicas
--                         ----   -----  --------
a2ffgwgdu88ntjqtu9bjt5izv  redis  redi   0/1

Look at tasks:

$ swarmctl task ls
ID                         Service  Desired State  Last State              Node
--                         -------  -------------  ----------              ----
edcqq2a5auf14fhrz2dp07ugn  redis.1  ACCEPTED       ACCEPTED 2 seconds ago  node-1

$ swarmctl task ls
ID                         Service  Desired State  Last State               Node
--                         -------  -------------  ----------               ----
5mmmtytjd5v7x8ireaskilgqn  redis.1  RUNNING        PREPARING 6 seconds ago  node-1

Look at your logs:

ERRO[0236] pulling image failed                          error=Error: image library/redi:latest not found module=taskmanager task.id=4pxjrqr74x4cezumz9nwlu91e
ERRO[0236] fatal task error                              error=Error: No such image: redi module=taskmanager task.id=4pxjrqr74x4cezumz9nwlu91e
WARN[0236] task updates not yet supported                module=taskmanager task.id=4pxjrqr74x4cezumz9nwlu91e
INFO[0238] grpc: Server.Serve failed to complete security handshake from "192.168.200.200:55361": remote error: bad certificate
WARN[0241] task updates not yet supported                module=taskmanager task.id=17g2jti8kmn56wowfhld644wu
ERRO[0243] pulling image failed                          error=Error: image library/redi:latest not found module=taskmanager task.id=17g2jti8kmn56wowfhld644wu
ERRO[0243] fatal task error                              error=Error: No such image: redi module=taskmanager task.id=17g2jti8kmn56wowfhld644wu
WARN[0243] task updates not yet supported                module=taskmanager task.id=17g2jti8kmn56wowfhld644wu
INFO[0246] grpc: Server.Serve failed to complete security handshake from "192.168.200.200:55371": remote error: bad certificate
WARN[0248] task updates not yet supported                module=taskmanager task.id=2um3ezqmqnre2f5k710rcrkvl
ERRO[0250] pulling image failed                          error=Error: image library/redi:latest not found module=taskmanager task.id=2um3ezqmqnre2f5k710rcrkvl
ERRO[0250] fatal task error                              error=Error: No such image: redi module=taskmanager task.id=2um3ezqmqnre2f5k710rcrkvl
WARN[0250] task updates not yet supported                module=taskmanager task.id=2um3ezqmqnre2f5k710rcrkvl
....

[stevvooe]

@FrenchBen this is the right behavior. Inspect one of the rejected tasks to see the error.

[FrenchBen]

$ swarmctl task inspect redis.1
Error: task redis not found
$ swarmctl task inspect edcqq2a5auf14fhrz2dp07ugn
Error: task edcqq2a5auf14fhrz2dp07ugn not found

Is this the expected output?

EDIT:
It looks like the task ID keeps rotating on the endless loop, which means that I'll get the above error if I don't inspect at the right time.
Catching it at the right time, I get the proper output:

swarmctl task inspect b6n4dh6ex6unzijd8vv3tbgmc
ID                     : b6n4dh6ex6unzijd8vv3tbgmc
Slot                   : 1
Service                : redis
Status
  Desired State        : SHUTDOWN
  Last State           : REJECTED
  Timestamp            : 2016-06-17T19:12:23.027037109Z
  Message              : preparing
  Error                : Error: No such image: redi
  ExitCode             : 0
Node                   : node-1
Spec
  Image                : redi

===> Task Parents
Task ID                      Service    Slot    Image    Desired State    Last State                 Node
-------                      -------    ----    -----    -------------    ----------                 ----
7rq7o67ebiva8gq1eezc1j3cd    redis      1       redi     ACCEPTED         ACCEPTED 1 second ago      node-1
b6n4dh6ex6unzijd8vv3tbgmc    redis      1       redi     SHUTDOWN         REJECTED 8 seconds ago     node-1
78k57wpvy8wr3hlc7vbjoe2oh    redis      1       redi     SHUTDOWN         REJECTED 15 seconds ago    node-1
al6ra425c66gyc2tga9vkw38r    redis      1       redi     SHUTDOWN         REJECTED 22 seconds ago    node-1
94kqzztg5deszzznftas1dye7    redis      1       redi     SHUTDOWN         REJECTED 29 seconds ago    node-1
8tpxkohde2ce13hvhajuxryc8    redis      1       redi     SHUTDOWN         REJECTED 36 seconds ago    node-1
358edaq812f5rha3dqbxaigu2    redis      1       redi     SHUTDOWN         REJECTED 43 seconds ago    node-1
7hii6cb9fepzz7blag9wh6pwp    redis      1       redi     SHUTDOWN         REJECTED 50 seconds ago    node-1
3b20f2bzbrn5e9jrw8pjau4vq    redis      1       redi     SHUTDOWN         REJECTED 57 seconds ago    node-1
cgek0q4o3r7423vuhm76o9lox    redis      1       redi     SHUTDOWN         REJECTED 1 minute ago      node-1

[stevvooe]

It looks like the task ID keeps rotating on the endless loop, which means that I'll get the above error if I don't inspect at the right time.

Each time, a new task gets dispatched.

To investigate this, I usually inspect the task, then find the task that existed before it, but we should automate this.

It looks like we need a UX for aggregating service errors. Something like swarmctl service errors <service>.

[nathanleclaire]

It looks like we need a UX for aggregating service errors. Something like swarmctl service errors .

Yes please!

[dperny]

@nathanleclaire docker/docker#23706 is where we're tracking this same issue on docker/docker, and according to the meeting we had yesterday, a general and sustainable solution is planned.

[aluzzardi]

@FrenchBen @nathanleclaire I think this was fixed in 1.12 with the enhanced docker service ps output. Can you confirm?

[nathanleclaire]

I'm still not persuaded that retrying forever is the right thing to do, but yes it's much easier to detect the service create is failing due to wrong image in the docker service ps output 👍