Che: [Bug] Snapshot removing failed. Snapshot attributes are not valid

Created on 21 Mar 2017 · 23Comments · Source: eclipse/che

Cannot delete snapshots from the remote registry at docker hub. Pushes and pulls to registry just fine, however it fails to delete old snapshots from registry resulting in ALOT of orphan snapshots in registry. I have a multi-machine workspace consisting of 5 machines and it left around 40 orphan snapshots over time.

Reproduction Steps:

configure che to work with a docker hub organization repository for saving snapshots
start a multi-machine workspace
stop the workspace, verify that it saves to registry
start the workspace and load from snapshot
stop the workspace without saving snapshot

OS and version:

Che running on AWS EC2 t2.medium.
Ubuntu 16.04
Eclipse Che 5.4

Diagnostics:
Log:

2017-03-21 18:16:26,551[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshot7r82ivljtss3ls1n', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,551[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,552[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshotyq6mieud1voj4cxf', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,553[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,553[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshot15c4j45me2k0lt2p', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]

kinbug

Source

rhughes89

Most helpful comment

Done. Fixed bug about invalid snapshot attributes. Added automate deletion of snapshots stored in Docker Hub.

mmorhun on 11 Apr 2017

🎉2

All 23 comments

Couple of questions:

Are you using 5.4.0-SNAPSHOT?
Can you provide more logs around this time?

skabashnyuk on 21 Mar 2017

@skabashnyuk
Yes i am using 5.4.0-SNAPSHOT. I can get the full log to you once i get back to my dev machine

rhughes89 on 21 Mar 2017

some more logs from before

2017-03-21 18:11:55,960[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.ssh' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:11:56,053[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'com.codenvy.rsync_in_machine' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:12:01,098[aceSharedPool-0]  [INFO ] [ineInnerRsyncAgentLauncherImpl 69]   - Docker machine has been deployed. ID 'machinebj7423o4nweryjab'. Workspace ID 'workspacem52qibob1zahcdiy'. Container ID '
7e40e857c4094f320d0a135e50283460401002287430ed05bbd206762867e21c'. Node host '10.20.6.143'. Node IP 'null'
2017-03-21 18:12:01,099[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.terminal' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:12:03,210[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.ws-agent' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:12:14,275[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.ls.json' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:12:14,429[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 757]    - Workspace 'testuser/test' with id 'workspacem52qibob1zahcdiy' started by user 'testuser'
2017-03-21 18:12:34,277[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 796]    - Workspace 'testuser/test' with id 'workspacem52qibob1zahcdiy' is being stopped by user 'testuser'
2017-03-21 18:12:34,280[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 863]   - Creating snapshot of workspace 'workspacem52qibob1zahcdiy', machines to snapshot: '5'
2017-03-21 18:14:39,134[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 886]   - Saving new snapshots metadata, workspace id 'workspacem52qibob1zahcdiy'
2017-03-21 18:14:41,453[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 827]    - Workspace 'testuser/test' with id 'workspacem52qibob1zahcdiy' stopped by user 'testuser'
2017-03-21 18:15:03,390[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.ssh' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:15:03,462[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'com.codenvy.rsync_in_machine' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:15:08,438[aceSharedPool-2]  [INFO ] [ineInnerRsyncAgentLauncherImpl 69]   - Docker machine has been deployed. ID 'machinec0f4g4zkbhh4f5mt'. Workspace ID 'workspacem52qibob1zahcdiy'. Container ID '
74db6fceea430619e403a2927c62651327fa8b6e495d496fa308c4c557d31b41'. Node host '10.20.6.143'. Node IP 'null'
2017-03-21 18:15:08,439[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.terminal' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:15:10,550[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.ws-agent' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:15:21,549[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 692]   - Launching 'org.eclipse.che.ls.json' agent at workspace workspacem52qibob1zahcdiy
2017-03-21 18:15:21,666[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 757]    - Workspace 'testuser/test' with id 'workspacem52qibob1zahcdiy' started by user 'testuser'
2017-03-21 18:15:31,634[aceSharedPool-3]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 796]    - Workspace 'testuser/test' with id 'workspacem52qibob1zahcdiy' is being stopped by user 'testuser'
2017-03-21 18:15:31,634[aceSharedPool-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 863]   - Creating snapshot of workspace 'workspacem52qibob1zahcdiy', machines to snapshot: '5'
2017-03-21 18:16:26,538[aceSharedPool-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 886]   - Saving new snapshots metadata, workspace id 'workspacem52qibob1zahcdiy'
2017-03-21 18:16:26,545[aceSharedPool-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 892]   - Removing old snapshots binaries, workspace id 'workspacem52qibob1zahcdiy', snapshots to remove '5'
2017-03-21 18:16:26,548[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,551[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshot7r82ivljtss3ls1n', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,551[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,552[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshotyq6mieud1voj4cxf', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
2017-03-21 18:16:26,552[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshotyq6mieud1voj4cxf', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,553[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,553[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshot15c4j45me2k0lt2p', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,554[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,555[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshottmq1vziu7pyg3zo9', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,555[aceSharedPool-3]  [ERROR] [c.p.d.m.DockerInstanceProvider 148]  - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,559[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshot6zdqouayarhvoa76', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:29,258[aceSharedPool-3]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 827]    - Workspace 'testuser/test' with id 'workspacem52qibob1zahcdiy' stopped by user 'testuser'
2017-03-21 19:08:44,329[ted-scheduler-7]  [INFO ] [ockerAbandonedResourcesCleaner 124]  - List containers registered in the api: []
2017-03-21 20:08:44,329[ted-scheduler-6]  [INFO ] [ockerAbandonedResourcesCleaner 124]  - List containers registered in the api: []
2017-03-21 20:23:02,998[Thread-5]         [INFO ] [o.a.c.http11.Http11NioProtocol 503]  - Pausing ProtocolHandler ["http-nio-8080"]
2017-03-21 20:23:03,049[Thread-5]         [INFO ] [o.a.c.core.StandardService 496]      - Stopping service Catalina
2017-03-21 20:23:03,122[ost-startStop-2]  [INFO ] [o.e.c.a.s.server.SystemManager 92]   - Preparing system to shutdown
2017-03-21 20:23:03,128[ost-startStop-2]  [INFO ] [o.e.c.a.s.s.ServiceTerminator 55]    - Shutting down 'workspace' service
2017-03-21 20:23:03,136[ost-startStop-2]  [INFO ] [.e.c.a.w.s.WorkspaceSharedPool 114]  - Shutdown workspace threads pool, wait 30s to stop normally
2017-03-21 20:23:03,136[ost-startStop-2]  [INFO ] [.e.c.a.w.s.WorkspaceSharedPool 126]  - Workspace threads pool is terminated
2017-03-21 20:23:03,136[ost-startStop-2]  [INFO ] [o.e.c.a.s.s.ServiceTerminator 63]    - Service 'workspace' is shut down
2017-03-21 20:23:03,137[ost-startStop-2]  [INFO ] [o.e.c.a.s.server.SystemManager 98]   - System is ready to shutdown
2017-03-21 20:23:03,138[ost-startStop-2]  [INFO ] [o.e.c.a.core.util.FileCleaner 77]    - File cleaner is stopped
2017-03-21 20:23:03,149[Thread-5]         [INFO ] [o.a.c.http11.Http11NioProtocol 532]  - Stopping ProtocolHandler ["http-nio-8080"]
2017-03-21 20:23:03,150[Thread-5]         [INFO ] [o.a.c.http11.Http11NioProtocol 547]  - Destroying ProtocolHandler ["http-nio-8080"]

ll'}
2017-03-21 18:16:26,555[aceSharedPool-3]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529]   - Couldn't remove snapshot 'snapshottmq1vziu7pyg3zo9', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]

rhughes89 on 21 Mar 2017

Correct me if I'm mistaken. You are saving snapshots at official docker registry, right? At https://hub.docker.com/, right?
If this is not secure information can you provide examples of orphaned images you found on docker, like my-registry.com:5000/myrepo/myimage:mytag@mydigest or myrepo/myimage:mytag. Basically how to pull such an image from ther registry. You can even remove the image and provide it, so nobody will be able to pull it.
Can you also provide values from your che.env
CHE_DOCKER.REGISTRY
CHE_DOCKER_NAMESPACE

garagatyi on 22 Mar 2017

@garagatyi Yes that's correct, I'm using the official docker registry. I deleted all the sensitive data on the machines and they are located here: https://hub.docker.com/u/rhughes89/. my Workspace is a multi-machine workspace with 5 machines. so I started and stopped it twice with resulting in 10 images stored in my docker hub account. The first 5 should have been deleted after I stopped I the second time.

CHE_DOCKER.REGISTRY = rhughes89
CHE_DOCKER_NAMESPACE was not set

rhughes89 on 22 Mar 2017

Also, note that today I checkedout the latest version; 5.6-SNAPSHOT and am still having the same issue

rhughes89 on 22 Mar 2017

@rhughes89 I think providing your DockerHub namespace is worth trying.

ghost on 22 Mar 2017

Yes. Please try
CHE_DOCKER_NAMESPACE=rhughes89
Not sure about value of property CHE_DOCKER.REGISTRY, need to check.
As far as I remember CHE_DOCKER.REGISTRY should have value the same as
CHE_DOCKER_REGISTRY_AUTH__USERNAME=rhughes89
Example, not sure if everything is 100% correct, let @eivantsov correct me if I mistaken
CHE_DOCKER_NAMESPACE=rhughes89
CHE_DOCKER_REGISTRY_AUTH_docker.io_URL=https://index.docker.io/v1/
CHE_DOCKER_REGISTRY_AUTH_docker.io_USERNAME=rhughes89
CHE_DOCKER_REGISTRY_AUTH_docker.io_PASSWORD=your password

garagatyi on 22 Mar 2017

Yes, this config seems to be ok

ghost on 22 Mar 2017

Ok guys, I tried the following and I'm still getting the same errors:

CHE_DOCKER.REGISTRY=https://index.docker.io/v1/
CHE_DOCKER_NAMESPACE=rhughes89
CHE_DOCKER_REGISTRY_AUTH_REGISTRY1_URL=https://index.docker.io/v1/
CHE_DOCKER_REGISTRY_AUTH_REGISTRY1_USERNAME=rhughes89
CHE_DOCKER_REGISTRY_AUTH_REGISTRY1_PASSWORD=your password

a few things to note. setting CHE_DOCKER.REGISTRY to rhughes89 didn't seem to work, I ended up changing it to https://index.docker.io/v1/ to allowed me to push and pull images from the registry.

also i could not set CHE_DOCKER_REGISTRY_AUTH_docker.io_URL. I was getting an error about the "." in the name on the config.

Is there a way to use the host docker to save snapshots without using a remote registry? I think I may have an issue because I'm running che on a server with no public ip address

rhughes89 on 22 Mar 2017

Quick update,

i was able to set this up using an insecure registry that comes with che instead of using docker hub and I no longer get the error message about unable to delete the machine snapshot. however, when I go to this URL:

http://{{my_private_ip}}:5000/v2/_catalog

I'm still seeing the old orphaned workspace machine snapshots. every time I boot and shutdown a workspace, it adds 5 new machine snapshots which will stay there indefinitely ( until i manually delete them ).

I can foresee this being an issue at some point

rhughes89 on 22 Mar 2017

@mmorhun can you correct what is wrong with last properties provided by @rhughes89 ? Maybe something like that?

CHE_DOCKER.REGISTRY=REGISTRY1
CHE_DOCKER_NAMESPACE=rhughes89
CHE_DOCKER_REGISTRY_AUTH_REGISTRY1_URL=https://index.docker.io/v1/
CHE_DOCKER_REGISTRY_AUTH_REGISTRY1_USERNAME=rhughes89
CHE_DOCKER_REGISTRY_AUTH_REGISTRY1_PASSWORD=your password

Is there a way to use the host docker to save snapshots without using a remote registry?

There is a property to save snapshots on local docker storage without pushing images into registry.
CHE_DOCKER_REGISTRY__FOR__SNAPSHOTS=false

I'm still seeing the old orphaned workspace machine snapshots.

Official docker registry image doesn't cleanup image layers, just links to them. I know it's frustrating, but it is behavior of registry by default. Maybe it is solved somehow in the newest versions of registry of there is an issue in docker repos, but I'm not sure.

garagatyi on 23 Mar 2017

@rhughes89 you almost right here. The proper name of docker registry property is CHE_DOCKER_REGISTRY (underscore instead of dot). Just checked the configuration - it works (also I've set CHE_WORKSPACE_AUTO__SNAPSHOT and CHE_WORKSPACE_AUTO__RESTORE to true).
By the way, you can use alias docker.io instead of full https://index.docker.io/v1/

mmorhun on 23 Mar 2017

@rhughes89 As about snapshot deleting from docker hub, seems we have a problem. Thank you for reporting.

mmorhun on 23 Mar 2017

@garagatyi I set CHE_DOCKER_REGISTRY__FOR__SNAPSHOTS=false and it seems to be removing the snapshots on the local docker alright. This works perfectly for me for the time being.

@mmorhun thank you. glad I can contribute in some sort of way.

rhughes89 on 23 Mar 2017

@mmorhun - so the issue exists when trying to save snapshots to DockerHub from any Che instance or only from a private Che instance?

bmicklea on 29 Mar 2017

@bmicklea removal of images/repos is different for different docker registries. In fact, this scenario hasn't been tested. For example, DockerHub may have certain way to remove repos while GCR or canister.io do it in a different way.

ghost on 29 Mar 2017

@bmicklea snapshots are saved but never deleted by CHE. This problem is more general than DockerHub only. A user can configure arbitrary external secured registry for snapshots. The problem is that auth methods in this registries can be different, even different from push/pull auth. So, DockerHub behaves in some way, Canister.io can have another behaviour, some other registry can have yet another behaviour.
Our current model doesn't support this. We need to refactor some staff and add support for some fixed set of registries and may be some general way in the end (but I am not sure that it is a good idea). Anyway, it should be discussed.

mmorhun on 29 Mar 2017

If I can make a suggestion. Perhaps having a brand new repository for each snapshot is a bit overkill? I think that maybe if use a repository for each workspace machine that is reused when booting and stopping your workspace. you would only have to delete the repository when the workspace is deleted. when you start and stop your workspace, it should save to that repository with the latest tag.

a repository name can be something like: {{CHE_DOCKER_NAMESPACE}}_{{WORKSPACENAME}}_{{MACHINE_NAME}}

so even if it fails to delete the repository, a system admin can manually remove the obsolete repositories easily because it will be logically named OR there can be a maintenance script to remove all orphan repositories.

rhughes89 on 29 Mar 2017

@rhughes89 well, yeah.. if we disable previous snapshot removal before pushing a new one, it will solve the problem, although I am not sure this is a good way to go.

ghost on 29 Mar 2017

Now we have to refactor some stuff, I'll proceed on this issue after it.

mmorhun on 31 Mar 2017

@rhughes89 thank you for the suggestion! But main goal here is keeping registry clear. When you push new image over old one, you just overwrite manifests, but blobs is still there and it is the problem. You don't see it, but it still uses your disk space. So it solves problem of a user, but not admin. In case of CHE, often user is admin too. Of course, there are clean up scripts, but they bring some other troubles...

mmorhun on 31 Mar 2017

Done. Fixed bug about invalid snapshot attributes. Added automate deletion of snapshots stored in Docker Hub.

mmorhun on 11 Apr 2017

🎉2

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Compare rendering performance terminal widget vs CommandOutputConsole.

AndrienkoAleksandr · 3Comments

Dashboard must not send keycloak token to plugin and devfile registries

sleshchenko · 3Comments

UI usage experience

skabashnyuk · 3Comments

How to add custom editor image to Che?

InterestedInTechAndCake · 3Comments

Deployment is not cleaned up if a workspace start fails because of pod creation timeout

sleshchenko · 3Comments