Lxd: Clarification on Image expiration - lxc config set images.remote_cache_expiry 1 not working?
Hello,
As per previous discussions I have set the config option lxc config set images.remote_cache_expiry 1 in order to expire old images after 1 day. This in conjunction with lxc config set images.auto_update_interval 0 which forces LXD to fetch new images whenever they are updated, the hope was that I could have an always up-to-date image repository which wouldn't bloat up to crazy levels over time.
This is how my local image repo looks after a few remote updates however (abridged):
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
| | b20779efd4db | no | Webdock Ubuntu LEMP PHP 7.1 | x86_64 | 800.85MB | Aug 3, 2017 at 7:44pm (UTC) |
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
| | fa9d31e111c5 | no | Webdock Ubuntu LEMP PHP 7.1 | x86_64 | 807.84MB | Aug 16, 2017 at 11:44am (UTC) |
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
Why are the old images not being deleted? Is this because there are active containers which were provisioned from these images once? Why are they being kept around in that case? In order to prevent the rootfs unmount bug in ZFS we've seen before, maybe?
In any case, I'd very much like not having to manually clean up old images (which will never be used for new containers) from my servers. Any hints here? :)
PS: The old images are most definitely deleted on the remote, and the alias resolves to the new image fingerprint.
Kramerican
All 16 comments
Can you paste "lxc image info b20779efd4db"?
stgraber
on 21 Aug 2017
Sure thing
root@jadzia ~ # lxc image info b20779efd4db
Fingerprint: b20779efd4db91e42e3b85e9f1bd045d6b992fa34f2548c7bcb0a39d28125c41
Size: 800.85MB
Architecture: x86_64
Public: no
Timestamps:
Uploaded: 2017/08/03 19:44 UTC
Expires: never
Last used: 2017/08/16 08:55 UTC
Properties:
description: Webdock Ubuntu LEMP PHP 7.1
Aliases:
Auto update: enabled
Source:
Server: https://krellide.webdock.io:8443
Protocol: lxd
Alias: webdock-xenial-lemp-php71
Hmm I see ... Expires Never?
Kramerican
on 21 Aug 2017
The "Expires: never" part isn't particularly relevant in this case, what should be is the "Last used: 2017/08/16 08:55 UTC" part, which should then have caused LXD to flush the image a day later on the 17th.
Can you paste "lxc config show" and "lxc info", that should contain what I need to reproduce this.
stgraber
on 21 Aug 2017
Merci :O)
root@jadzia ~ # lxc config show
config:
images.auto_update_interval: "0"
images.remote_cache_expiry: "1"
root@jadzia ~ # lxc info
config:
images.auto_update_interval: "0"
images.remote_cache_expiry: "1"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIFajCCA1KgAwIBAgIQcwHc0e5worXrTbwADIGr/zANBgkqhkiG9w0BAQsFADA0
MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGph
ZHppYTAeFw0xNzA4MDMxNzQ4MDhaFw0yNzA4MDExNzQ4MDhaMDQxHDAaBgNVBAoT
E2xpbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAamFkemlhMIICIjAN
BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAp5TkOY7Bcibak+HEH41zy2OHlmZf
edzHFxkESH5MkspomMO7D0aRHJXlUmq5rgN3U2ZagbXKD7C/kVLVxrVm2b4qx6kf
/aPTl70dnVT87Wz49Gn0+uPcIA3wW4YBjPZoPojuhREO1M0Z3sfZHoK6Cu3O5STb
PUUMe8SYbHeiNcrkPRqxW8eue8x22LtgHBCglVvTZ9/aGKszxhQY9wus8MmFxXcG
oImAsh0Fans0f4aNL437zG3GsUqfHMOZ7ELJLYBQKrXfHbQvqDKPtItw9r137aH2
FP7c/i+a/HhNZ3rKGkEuiDZiLFITUvLv++maQBbjofNVBvQdhj/UgmE1I443xrlj
h6OfxmqmWHlfhrtP/6UWq5odTyPjYbrIB73vNlcjo8Sa6usyIcOaP1VdxglLI+xT
lDT/TEihLAPjq/APnJEnMqLVMa2PPNWi0mxdK0Mxk8TviI6mWR+Ey1w4JpR2TU6v
PuWSU7B2RC5mZvyGjUaq7uMbxqFSa9IBAqhVE2IY5olZP7luYWAQQDNyHyYFtBwJ
XzzyGOqpGZyUz9NZdZLS7KNbjNU/4gR5mnmSoYIZitdcroESQlHCAuoza/JvVUWg
Nv6hT8iM+EObonbFAYtZKVi9cBU1diaQ24sw/ahE6hMmMkIUyGr9MC0Stu3g/sjS
IEjpeKjbs7Mjmi8CAwEAAaN4MHYwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG
CCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwQQYDVR0RBDowOIIGamFkemlhhwSI8yyM
hxAqAQT4AhIMqgAAAAAAAAAChwSI8yyMhxAqAQT4AhIMqgAAAAAAAAACMA0GCSqG
SIb3DQEBCwUAA4ICAQCPlnIrJhUm9e/zwMekA4TglPvwwbnfCC+XRIBUVyJ3l6JN
EiMA/65OovXhgxvjOknuvJz/unGlk7J2a8V0wM81mlNNK22mxW7Xt7R9tk43UKxh
BI0WFVmD8/ITf/9XTQNVcqdVOa5HHoSpL5H/18fir3stTxgNBkdA2rbUJc4sOq26
ZqyKfM0fEGzCKecwTbhLFf439GpoqQK+J81/GxBXjNDkiLoG8ltMRiDQdXs2iVgI
/bsp+noRziIG474YVhSeWyLC8vxhH3MnfPKEzk38QnmWVP+1y+ArihSp1+SkSowQ
hcHlWIy3xJ7Ffo5uYmM3vaCzRvRh6ojIPY9fwZ+iiQeSFKufMTEUozoZRDsCAMMg
pgfRSbBvwtds99MYs/SrB98VK2OAOxOmf3x3ULTRsHB4OlEEv9R6TszSAnJBiYB6
XyTeFzqT4NhsVlgRKz+ZgonzPA8E52/F55whMk7SFEPPvJg/IBTSf/ORZTRdnAqI
UVhx6nhypDeQNUyec0u9o4kNJu0bZxpFyh74C69JVKGUmbcLZergyaFUjQ/FyN6O
1FwzsU0DEPfZdbMvoOf6+AAIAI48WTHfFgiZHirG4oOyZhPMVhZyR5JvJj+yPBS4
+VDMvwxFs8yTCDGknfIHM1sCfgwWDbPa9hg1YTUqg15dKGtNTcjUEPVwFCbNXg==
-----END CERTIFICATE-----
certificate_fingerprint: e2c4942cdc881a3ca09387ba573e2aee3bd2066ebd2b84143e78fd8efdbfe5b1
driver: lxc
driver_version: 2.0.8
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.10.0-30-generic
server: lxd
server_pid: 7580
server_version: "2.16"
storage: zfs
storage_version: 0.6.5.9-2
Ok, I think I found the issue, for some reason the internal "cached" property doesn't appear to be set to the right value, at least in some cases...
Tracking down exactly what's going on now.
stgraber
on 21 Aug 2017
my current guess is that once an image gets auto-updated, its "cached" property gets cleared, preventing it from later expiring...
stgraber
on 21 Aug 2017
Cool - I love it when I help out finding bugs, and it's not just my own retardedness at work :laughing:
Let me know what the solution is, whenever you get it tracked down / sorted. And ... Thanks!
Kramerican
on 21 Aug 2017
I still see this happening even after upgrading to v2.17
I am however in doubt whether the images I have, which are "stale", pre-date the v2.17 upgrade, and thus whatever flags need to be set have not been set properly, preventing the new version from cleaning them up.
I am doing a test today on a fresh 2.17 system, which will receive an updated image today and should therefore nuke the old image tomorrow.
Please let me know if there is any specific information I can provide here to help
Kramerican
on 12 Sep 2017
I can confirm this is still not resolved as discussed. On a clean v2.17 system a stale image does not expire:
root@quark ~ # lxc image list
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
| | 12b72c47c361 | no | Webdock Ubuntu LEMP PHP 7.1 | x86_64 | 832.70MB | Sep 12, 2017 at 12:14pm (UTC) |
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
| | 989db48304a7 | no | Webdock Ubuntu LEMP PHP 7.1 | x86_64 | 809.81MB | Sep 11, 2017 at 7:49pm (UTC) |
+-------+--------------+--------+-----------------------------+--------+----------+-------------------------------+
root@quark ~ # lxc config show
config:
images.auto_update_interval: "0"
images.remote_cache_expiry: "1"
root@quark ~ # lxc info
config:
images.auto_update_interval: "0"
images.remote_cache_expiry: "1"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIFUDCCAzigAwIBAgIRAP6buNA4N35aP8wL3+a/73YwDQYJKoZIhvcNAQELBQAw
MzEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBx
dWFyazAeFw0xNzA5MTExMzE5MTlaFw0yNzA5MDkxMzE5MTlaMDMxHDAaBgNVBAoT
E2xpbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAcXVhcmswggIiMA0G
CSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDWsrtS9zNAO9Fmo7B/U+K9nB4U12hS
pzdR9R32ViwRmg+O3hXejJzQImeBfylzMFekaXeoIQ958Nux+slXhNccP2u+iQtc
Su9zJc1JtWAYeNxiEGvolrZWYnbbowU8aTr+BHuaLF1HHcofyI9aCSZxqB2K5NVS
BtnRMXkM7yHxtEM53INTPOnF+nNIssftwqvZr8cZApGeDpo6SLCS6AshBCjQOl+m
BmAdKh2xaHYsNJfmP9Y9Y+dGAC0TRbYDirmG/Q/HKzt1TPsscz4O289tdoSaFu8F
kcmX/4KMJxj58YBkYRqIbLAo9WkzAyYSr/fAACWD0tUKPsUnouRJN7+01w87rQJX
xw+TZKxX+gy/dSJe4dHCS7cFoNjb29onefj6HcLgMJ2sUT3A1bOZ+mL6omk+ggLh
6pd2tmuqls/pPmIaWYmHoB4HzMA063PV+a+j2lPjyIXF5ohs5tg0lS9T9B0Z67yU
kTOXKnFzHlbW5nq1ZHntKGxPKjnoKF4+XhHP5Y5Z+KRjuGGeppwaeiWATmViD5jq
DqZIxl3OCY1Z4j/59CPOz4lBB3H1t792/pK6v2PnQ6VhleNcCgwkpVMXRSveHz/3
TnIiGYOkc4tcdchvTBmteq6q1nDpE0Btkyblo5T9j8ZUmAbJf84j4qfMYewYy/1T
7fGeoNOjp0CDQQIDAQABo18wXTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI
KwYBBQUHAwEwDAYDVR0TAQH/BAIwADAoBgNVHREEITAfggVxdWFya4cEXoIProcQ
KgEE+AELE9IAAAAAAAAAAjANBgkqhkiG9w0BAQsFAAOCAgEAVULPJQg0FYag4NMF
0J6GLKZf4Jv17Sor8BKDIp2+8maYQvtM5VZM+CzfHmebKvhi/ntoItyTS2YGGIRG
3bejwBZW12TMSwILNAXeUX6V5FHmBJevdPqBXcvle0nsH1G0xUDl+epdwAvLgU8S
rlqiIldC1xoaWWX+xR5GYusop7lhFihgLxBOHL2n/QFfNPIFjrsGcIH1jXfnOZdA
/N/45CLO4IMBSm6qalZDwx4m7O2NKLXJKVuuSsWmj8BQcuV98vJsNGLX7oI4kxQF
+1buPFl4cmbuei0vhnS+EbOCjIV1pRvKfD1+jduzOJ+RRM36EqglHvGLwBnV4i5T
oLD+lF6Z/EGkSl+p4JrwmOT7wcJglaMSJBK+h2GVAq1DY77J7OFJxwrazrhoauVS
hLH1OUdU5g9gERpnlGKG7KhlUm2qA5Vz+9wc4g0QVSwj8g4CsAgSbSmYB2gj9ARl
cJabvnUvFRcmQI1FTnSvysh1QYPoF2k/9/me+85kaerLM/FLmQtvb1MjUT2Igl5T
dUWwAiSOwmGk+7mxu8FDhvG4zjAQycsoFNJ5aBiEXuvK0eMbu6n8DeB/ITZCEXb4
Ckr4jQZop7d1RBcM78o8nLVnr3lv9F+5tn3LShdRkIiVhuz4juzsJq1NT9PZC1Y3
qnad9Qa//nNxcPPoKtW2Ola+qhg=
-----END CERTIFICATE-----
certificate_fingerprint: c2522ac7fc8e0d910b664031f02d844f06d342280a85ee1c8d07c32ee17e794b
driver: lxc
driver_version: 2.0.8
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.10.0-33-generic
server: lxd
server_pid: 3625
server_version: "2.17"
storage: zfs
storage_version: 0.6.5.9-2
What does "lxc image info" show for both images?
stgraber
on 17 Sep 2017
Voila
~ # lxc image info 12b72c47c361
Fingerprint: 12b72c47c36119b1ce3e5993f07a8e3816aae771d15a42e6fab99d033cc22312
Size: 832.70MB
Architecture: x86_64
Public: no
Timestamps:
Uploaded: 2017/09/12 12:14 UTC
Expires: never
Last used: 2017/09/14 19:15 UTC
Properties:
description: Webdock Ubuntu LEMP PHP 7.1
Aliases:
Cached: yes
Auto update: enabled
Source:
Server: https://krellide.webdock.io:8443
Protocol: lxd
Alias: webdock-xenial-lemp-php71
lxc image info 989db48304a7
Fingerprint: 989db48304a7fa536fb65c133b4a7305689f1d1449d8764a6afd6fca476647ba
Size: 809.81MB
Architecture: x86_64
Public: no
Timestamps:
Uploaded: 2017/09/11 19:49 UTC
Expires: never
Last used: 2017/09/12 12:10 UTC
Properties:
description: Webdock Ubuntu LEMP PHP 7.1
Aliases:
Cached: yes
Auto update: enabled
Source:
Server: https://krellide.webdock.io:8443
Protocol: lxd
Alias: webdock-xenial-lemp-php71
Did that tell you anything useful @stgraber ?
Kramerican
on 19 Sep 2017
Well, the flags are correct at least, it's just unclear why it's not expiring them then...
stgraber
on 19 Sep 2017
Ok, so I found one obvious bug at least.
stgraber
on 19 Sep 2017
https://dl.stgraber.org/lxd-3698 (sha256: cfb4a00211a70da6743a654228f10f56b260424381173d4fa3c7c46db85ad479)
Can you start that one and see if it properly expires your image and the one that's meant to stay around will still be there?
stgraber
on 19 Sep 2017
Ok so I grabbed that file and did (listing steps here just in case I'm doing it wrong)
service lxd stop
./lxd-3698
WARN[09-20|12:26:31] CGroup memory swap accounting is disabled, swap limits will be ignored.
EROR[09-20|12:26:31] balance: Unable to set cpuset err="Failed to set LXC config: lxc.pty.max=1024" name=bigumstaging value=0,1,10,11,2,3,4,5,6,7,8,9
EROR[09-20|12:26:31] balance: Unable to set cpuset err="Failed to set LXC config: lxc.pty.max=1024" name=fossfabriker value=0,1,10,11,2,3,4,5,6,7,8,9
EROR[09-20|12:26:31] balance: Unable to set cpuset err="Failed to set LXC config: lxc.pty.max=1024" name=publify value=0,1,10,11,2,3,4,5,6,7,8,9
EROR[09-20|12:26:31] balance: Unable to set cpuset err="Failed to set LXC config: lxc.pty.max=1024" name=schmidtdelux value=0,1,10,11,2,3,4,5,6,7,8,9
left it for a minute then ctr+c out of it
Then did lxc service start and lxc image list
Result: I still see the old image hanging around. It did not expire.
Kramerican
on 20 Sep 2017
Related issues
mt-caret
路
3Comments
purell
路
4Comments
gregsifr
路
3Comments
chuegel
路
3Comments
iteco
路
3Comments