Trident: Stack deployments fail when using UCP 2.2.5 and Collections

[adkerr]

_Originally created by @lordgnu as netapp/netappdvp#117_

We upgraded to UCP 2.2.5 this morning and suddenly all of our deployments are failing when using the netappdvp plugin and collections. We always seem to get an error similar to this: failed to update service 7c1e4382_vtest_db-redis: Error response from daemon: located 1 sets of existing volumes named "7c1e4382_vtest_db_redis" with different collection labels. Please make sure all existing volumes with the same name share the same collection or don't have one at all

Our deployment system inspects services about to be deployed and creates the volumes ahead of time (with the correct collection labels) in the event they don't already exists. This was because (prior to UCP 2.2.5) volume labels were ignored when doing a docker stack deploy to a swarm. It seems the labels are now honored, but past the initial deployment, all subsequent deployments fail with the aforementioned error.

We are using version 17.10.1 of the netappdvp plugin with UCP 2.2.5 and Docker 17.06.2-ee-6

Below is some output from our deployment system showing the errors returned (as well as what labels were applied to the service). Following this output is the result of a docker volume inspect of the same volume raising the error from a management node, and that shows that the collection labels all match.

[2018-01-18T12:38:51-06:00][start] Beginning deployment 
[2018-01-18T12:38:51-06:00][info] Cloning repository from REDACTED 
[2018-01-18T12:38:52-06:00][info] Generating swarm.yml 
[2018-01-18T12:38:52-06:00][info] Resolved swarm.yml path as /tmp/swarm-test-7c50c1c67a2f4eba6f4afcfee2b8d163e42c942adbedb0adafb098960cb1e96f/docker/swarm/swarm.yml 
[2018-01-18T12:38:52-06:00][info] Processing services for image / network replacement 
[2018-01-18T12:38:52-06:00][info] Checking service: db-redis 
[2018-01-18T12:38:52-06:00][info] Overriding image: REDACTED/platform/db-redis:0.0.26 
[2018-01-18T12:38:52-06:00][info] Checking networks 
[2018-01-18T12:38:52-06:00][info] Assigning deployment to collection: /vtest 
[2018-01-18T12:38:52-06:00][info] Tagging deployment with user: dbauer 
[2018-01-18T12:38:52-06:00][info] Appending YML labels 
[2018-01-18T12:38:52-06:00][info] - Adding label: delivery.user=dbauer 
[2018-01-18T12:38:52-06:00][info] - Adding label: delivery.deployment=swarm-test-7c50c1c67a2f4eba6f4afcfee2b8d163e42c942adbedb0adafb098960cb1e96f 
[2018-01-18T12:38:52-06:00][info] - Adding label: delivery.master=true 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection.root=true 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true 
[2018-01-18T12:38:52-06:00][info] Checking root volume definitions 
[2018-01-18T12:38:52-06:00][info] - Checking volume: db_redis (7c1e4382_vtest_db_redis) 
[2018-01-18T12:38:52-06:00][info] -- Found volume: db_redis (7c1e4382_vtest_db_redis) 
[2018-01-18T12:38:52-06:00][info] Checking root network definitions 
[2018-01-18T12:38:52-06:00][info] Processing network: 7c1e4382-redis-network 
[2018-01-18T12:38:52-06:00][info] Deploying service to swarm 
[2018-01-18T12:38:53-06:00][stderr] Updating service 7c1e4382_vtest_db-redis (id: nk918xdmi97hm9st35gygbd98) 
[2018-01-18T12:38:53-06:00][stderr] failed to update service 7c1e4382_vtest_db-redis: Error response from daemon: located 1 sets of existing volumes named "7c1e4382_vtest_db_redis" with different collection labels. Please make sure all existing volumes with the same name share the same collection or don't have one at all 
[2018-01-18T12:38:53-06:00][stderr] exit status 1 
[2018-01-18T12:38:53-06:00][error] Deployment has failed: an error was encountered during swarm deployment: exit status 1 
[2018-01-18T12:38:53-06:00][complete] Deployment failed 

Volume inspection output:

docker volume inspect 7c1e4382_vtest_db_redis
[
    {
        "Driver": "netapp_sync:latest",
        "Labels": {
            "com.docker.swarm.whitelists": "[\"node==REDACTED\"]",
            "com.docker.ucp.access.label": "/vtest",
            "com.docker.ucp.collection": "b8dee80c-6b58-4d79-a33f-236c6fc8b4bb",
            "com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb": "true",
            "com.docker.ucp.collection.root": "true",
            "com.docker.ucp.collection.swarm": "true",
            "delivery.user": "dbauer"
        },
        "Mountpoint": "/var/lib/docker/plugins/65801f6b71e89c320b3fde10638395ed676beeb1cbaaa7fc6f15ac06695c3af3/rootfs",
        "Name": "7c1e4382_vtest_db_redis",
        "Options": {
            "size": "2G"
        },
        "Scope": "global",
        "Status": {
            "Snapshots": []
        }
    }
]

We have attempted this every way we can think of (starting from scratch) and all followup deployments seem to fail whether or not we manually create the volume or let docker handle that for us in the initial deployment; whether or not we omit the volume labels entirely, or only add "com.docker.ucp.access.label", or verbosely add every ucp label...

---

@lordgnu:
Update on this... It appears as long as you explicitly add ALL UCP collection labels to both the service and the volumes in the swarm template, the initial deployment works as well as the follow-up deployments. Caveat is you MUST create the volume ahead of time with all those labels defined as well, and all pre-existing volumes, even if they belong to the correct collection according to UCP must be destroyed and recreated (meaning whole stacks need to be torn down) in order for the update deployments to operate as expected.

Here's my logs from the attempt that proved to be successful:

######################################################################################
# Attempt : Deployment with ALL collections labels on volumes/services, pre-creation #
######################################################################################

Initial Deployment (Successful):

[2018-01-18T14:35:45-06:00][start] Beginning deployment
[2018-01-18T14:35:45-06:00][info] Cloning repository from ssh://REDACTED/db-redis.git
[2018-01-18T14:35:46-06:00][info] Generating swarm.yml
[2018-01-18T14:35:46-06:00][info] Resolved swarm.yml path as /tmp/swarm-test-e54b41ad3d63aae0ba06f9a9c5cf1c26a372d7d74331716b2696b1a9514a0035/docker/swarm/swarm.yml
[2018-01-18T14:35:46-06:00][info] Processing services for image / network replacement
[2018-01-18T14:35:46-06:00][info] Checking service: db-redis
[2018-01-18T14:35:46-06:00][info] Overriding image: REDACTED/platform/db-redis:master
[2018-01-18T14:35:46-06:00][info] Checking networks
[2018-01-18T14:35:48-06:00][info] Assigning deployment to collection: /vtest
[2018-01-18T14:35:48-06:00][info] Tagging deployment with user: dbauer
[2018-01-18T14:35:49-06:00][info] Appending YML labels at path services.db-redis.deploy.labels
[2018-01-18T14:35:49-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:35:49-06:00][info] - Adding label: delivery.deployment=swarm-test-e54b41ad3d63aae0ba06f9a9c5cf1c26a372d7d74331716b2696b1a9514a0035
[2018-01-18T14:35:49-06:00][info] - Adding label: delivery.master=true
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:35:49-06:00][info] Checking root volume definitions
[2018-01-18T14:35:49-06:00][info] - Checking volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:35:49-06:00][info] -- Creating volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:35:53-06:00][info] Appending YML labels at path volumes.db_redis.labels
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:35:53-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:35:53-06:00][info] Checking root network definitions
[2018-01-18T14:35:53-06:00][info] Processing network: 7c1e4382-redis-network
[2018-01-18T14:35:53-06:00][info] Deploying service to swarm
[2018-01-18T14:35:54-06:00][stdout] Creating service 7c1e4382_vtest_db-redis
[2018-01-18T14:35:54-06:00][info] Waiting for service to become healthy
[2018-01-18T14:35:54-06:00][info] Will poll once every 30 seconds for up to 20 minutes...

Volume Inspection:

# docker volume inspect 7c1e4382_vtest_db_redis
[
    {
        "Driver": "netapp_sync:latest",
        "Labels": {
            "com.docker.swarm.whitelists": "[\"node==REDACTED\"]",
            "com.docker.ucp.access.label": "/vtest",
            "com.docker.ucp.collection": "b8dee80c-6b58-4d79-a33f-236c6fc8b4bb",
            "com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb": "true",
            "com.docker.ucp.collection.root": "true",
            "com.docker.ucp.collection.swarm": "true",
            "delivery.user": "dbauer"
        },
        "Mountpoint": "/var/lib/docker/plugins/9d30000b1042920d3636aade2291c2a8d54cd1521320ef58ca54b593daef8414/rootfs",
        "Name": "7c1e4382_vtest_db_redis",
        "Options": {
            "size": "2G"
        },
        "Scope": "global",
        "Status": {
            "Snapshots": []
        }
    }
]

Follow-up deployment with same settings:

[2018-01-18T14:37:23-06:00][start] Beginning deployment
[2018-01-18T14:37:23-06:00][info] Cloning repository from ssh://REDACTED/db-redis.git
[2018-01-18T14:37:24-06:00][info] Generating swarm.yml
[2018-01-18T14:37:24-06:00][info] Resolved swarm.yml path as /tmp/swarm-test-8f0cf7534731795a1f9cc7cd7f206f1af65ec49b37e839a0fdb579e57735d89b/docker/swarm/swarm.yml
[2018-01-18T14:37:24-06:00][info] Processing services for image / network replacement
[2018-01-18T14:37:24-06:00][info] Checking service: db-redis
[2018-01-18T14:37:24-06:00][info] Overriding image: REDACTED/platform/db-redis:0.0.26
[2018-01-18T14:37:24-06:00][info] Checking networks
[2018-01-18T14:37:24-06:00][info] Assigning deployment to collection: /vtest
[2018-01-18T14:37:24-06:00][info] Tagging deployment with user: dbauer
[2018-01-18T14:37:24-06:00][info] Appending YML labels at path services.db-redis.deploy.labels
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.deployment=swarm-test-8f0cf7534731795a1f9cc7cd7f206f1af65ec49b37e839a0fdb579e57735d89b
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.master=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:37:24-06:00][info] Checking root volume definitions
[2018-01-18T14:37:24-06:00][info] - Checking volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:37:24-06:00][info] -- Found volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:37:24-06:00][info] Appending YML labels at path volumes.db_redis.labels
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:37:24-06:00][info] Checking root network definitions
[2018-01-18T14:37:24-06:00][info] Processing network: 7c1e4382-redis-network
[2018-01-18T14:37:24-06:00][info] Deploying service to swarm
[2018-01-18T14:37:25-06:00][stdout] Updating service 7c1e4382_vtest_db-redis (id: q3t32wmyoyku2bkgmrs4a8z43)
[2018-01-18T14:37:25-06:00][info] Waiting for service to become healthy
[2018-01-18T14:37:25-06:00][info] Will poll once every 30 seconds for up to 20 minutes...

[RemyCH]

+1
Same issue here since we upgraded in 2.2.5. Critical for my customer, we can't mount external volume since the RBAC on volume is enabled.

The issue is : inspecting the same volume from different nodes shows that the metadata aren't consistent from one node to an other.

• creation via bundle :

bash-4.2$ docker volume create -d netapp --opt size=1g meta_slave_test2 --label com.docker.ucp.access.label="/Run/Std/rmq-168113"
meta_slave_test2

• inspect volume without bundle. You see that all informations are correct on node 001 you have no "name", no "size" or no collection info on node 002.
  We did multiple create, the issue is not specific to the node 002. Sometimes the 002 got the info, and not the 001 by example.. Those 2 nodes are worker nodes.

node001$ docker volume inspect meta_slave_test2
[
    {
        "Driver": "netapp:latest",
        "Labels": {
            "com.docker.swarm.whitelists": "[\"node==node002|node001\"]",
            "com.docker.ucp.access.label": "/Run/Std/rmq-168113",
            "com.docker.ucp.collection": "5adeb0a9-4a63-4f9b-9312-738b591db100",
            "com.docker.ucp.collection.24d2e68d-9403-4ff9-8be3-70d4e2c65270": "true",
            "com.docker.ucp.collection.5adeb0a9-4a63-4f9b-9312-738b591db100": "true",
            "com.docker.ucp.collection.c8b78227-32f4-4cdb-a000-bd9f15592ac4": "true",
            "com.docker.ucp.collection.root": "true",
            "com.docker.ucp.collection.swarm": "true"
        },
        "Mountpoint": "/var/lib/docker/plugins/5c0c607f825693c600c14c0277697e31b47800c0293df6b4933d88934b8d1f27/rootfs",
        "Name": "meta_slave_test2",
        "Options": {
            "size": "1g"
        },
        "Scope": "global",
        "Status": {
            "Snapshots": []
        }
    }
]

node002$ docker volume inspect meta_slave_test2
[
    {
        "Driver": "netapp:latest",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/plugins/b6cb40b5dfe69743d31d2f2028589e6b827ebc5ab9c2cf958b63978f9d3bc1ca/rootfs",
        "Name": "meta_slave_test2",
        "Options": {},
        "Scope": "global",
        "Status": {
            "Snapshots": []
        }
    }
]

• And now if you check the metadata.db on each node :

node001$ strings metadata.db | grep meta_slave_test2
meta_slave_test2{"Name":"meta_slave_test2","Driver":"netapp:latest","Labels":{"com.docker.swarm.whitelists":"[\"node==node002|node001\"]","com.docker.ucp.access.label":"/Run/Std/rmq-168113","com.docker.ucp.collection":"5adeb0a9-4a63-4f9b-9312-738b591db100","com.docker.ucp.collection.24d2e68d-9403-4ff9-8be3-70d4e2c65270":"true","com.docker.ucp.collection.5adeb0a9-4a63-4f9b-9312-738b591db100":"true","com.docker.ucp.collection.c8b78227-32f4-4cdb-a000-bd9f15592ac4":"true","com.docker.ucp.collection.root":"true","com.docker.ucp.collection.swarm":"true"},"Options":{"size":"1g"}} ...

node002$ strings metadata.db | grep meta_slave_test2
meta_slave_test2{"Name":"","Driver":"netapp:latest","Labels":null,"Options":null}

So you'll find every volume via "docker volume ls", but metadata aren't correctly setted everywhere.
Once again it's not specific to the 17.06, we already saw this kind of things in 17.03 with older nDVP versions. As long as we had no RBAC on volume enabled it worked.

[Jukie]

+1 on this with 2.2.5, did you find any situation that _does_ works consistently?
Possibly pre-creating volume via bundle client side or via ssh to worker/manager?

[clintonk]

This is a Docker issue that affects all external volume plugins. We're waiting on a fix from Docker that will allow us to update Trident and support this deployment model.

[Jukie]

Yeah for example I'm using cloudstor, I meant to mention that in my comment. I understand it's a docker problem but I was wondering if anyone has a workaround until 2.2.6/3.0.

[kevtainer]

i believe there is no official workaround. we ended up backing up UCP on our test cluster, reinstalling 2.2.4, and restoring from the 2.2.5 backup. our production cluster wasn't impacted since we discovered this issue on our test environment.

our rep @ docker has indicated that a fix for this is in QA, with no official timeline for release.

[innergy]

Docker just added a release note about this: https://docs.docker.com/datacenter/ucp/2.2/guides/release-notes/#version-225

Specifically, under "known issues":

UCP doesn’t allow enforcing role-based access control for third-party volume plugins. If you’re using third-party volume plugins and are planning on upgrade UCP, you can skip 2.2.5 and wait for the upcomming 2.2.6 release, which will provide an option to turn of RBAC enforcement for volumes.

[kevtainer]

They've updated their release notes. Our TAM has conveyed that this issue has a fix undergoing tests at this time.

https://docs.docker.com/datacenter/ucp/2.2/guides/release-notes/#version-225

UCP role-based access control is not compatible with all third-party volume plugins. If you’re using certain third-party volume plugins (such as Netapp) and are planning on upgrading UCP, you can skip 2.2.5 and wait for the upcoming 2.2.6 release, which will provide an alternative way to turn on RBAC enforcement for volumes.

[korenaren]

UCP 2.2.6 is released. This should now be fixed.