Lxd: LXD fails to start with Duplicate config row

[jrens]

Required information

• Distribution: Ubuntu
• Distribution version: 20.04.1
• The output of "lxc info" or if that fails: Fails unix.socket: connect: connection refused
  
  
  
  • Kernel version: 5.4.0-58-generic
  
  
  • LXC version: 4.9
  
  
  • LXD version: 4.9
  
  
  • Storage backend in use:
  
  

Issue description

I can't start the lxd daemon. The following snap change occurred today (about 3.5hrs ago) that caused my containers to stop:

» snap changes                                      
ID    Status  Spawn                Ready                Summary
1023  Done    today at 14:08 NZDT  today at 14:14 NZDT  Auto-refresh snap "lxd"

Error from lxd

lxd.daemon[63022]: Error: Failed to load network "lxdbr0" in project "default": Duplicate config row found for key "ipv4.address" for network ID 1
Dec 16 17:24:45 aeon lxd.daemon[62901]: => LXD failed to start

However I'm not sure how to address this issue since:

» lxc network show lxdbr0          
Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory

» lxc network edit lxdbr0
Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory

[wgrant]

This happened for me on an automatic upgrade from 4.8 to 4.9. DB state after the upgrade is thus:

sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin
sqlite> SELECT * FROM networks;
1|1|lxdbr0||1|0
sqlite> SELECT * FROM networks_config;
5|1||ipv6.address|2403:5800:a001:7b01::1/64
6|1||ipv6.nat|false
8|1||ipv6.address|none
9|1||ipv4.address|10.0.4.1/24
10|1||ipv4.nat|true

[jrens]

Similar. Removing the line didn't work. It reappeared as I restarted lxd.

2|1||ipv4.address|10.131.91.1/24
5|1||ipv4.address|10.0.4.1/24
6|1||ipv4.nat|true
7|1||ipv6.address|fd42:f0e0:674:a93f::1/64
8|1||ipv6.nat|true

[tomponline]

I'll prepare a patch file for you to run to fix this.

[tomponline]

Do you run a cluster?

[wgrant]

My machine is not a member of a cluster.

[tomponline]

@jrens the issue with your DB are these two lines:

2|1||ipv4.address|10.131.91.1/24
5|1||ipv4.address|10.0.4.1/24

It looks like you have some old config in there, which subnet is your current one?

[tomponline]

@jrens if you create a file at /var/snap/lxd/common/lxd/database/patch.global.sql containing:

DELETE FROM networks_config WHERE id = n

Where n is either 2 or 5 depending on which config item is old in your case.

And then try starting LXD.

More info here:

https://linuxcontainers.org/lxd/docs/master/database#running-custom-queries-at-lxd-daemon-startup

[tomponline]

@wgrant ah its the ipv6.address rows (IDs 5 and 8), if you remove the one that is not accurate for your setup (using the patch file above) then it should start.

[tomponline]

@jrens @wgrant have those networks existed since before LXD 4.8 was installed out of interest?

[wgrant]

Ah, thanks, patch.global.sql was the mechanism I was looking for. That's got things up and running.

This network is at least several months old. One of my other systems, which didn't have a custom subnet defined, upgraded to 4.9 apparently without incident.

[tomponline]

@wgrant thanks, I identified a way to insert duplicate config during clustered network setups in LXD 4.8, but hadn't realised there had been a way previous to that get duplicate config into the database. There is a patch in LXD 4.9 that will automatically remove duplicates that have same key and same value. But currently duplicate keys with different values are not handled automatically.

The plan is to add a unique key to the database in the next release to prevent issues like this occurring in the future (once the initial patch has highlighted any problems).

[jrens]

@tomponline That worked perfectly thanks. This machine has been running lxd since before there was a snap version. My guess would be that network was from migrating over from those days.

[goapunk]

Had the same issue, the patch file worked. Thanks for the quick fix!