Lxd: failed to apply update 11: Failed to add project_id column: FOREIGN KEY constraint failed

[cyberwillis]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04
• The output of "lxc info" or if that fails:
  
  
  
  • Kernel version: 4.4.0-135
  
  
  • LXC version: 3.5 (6decb1bfad667f23e7a5534b86df0efc096ca0a1) build from the source
  
  
  • LXD version: 3.5 (6decb1bfad667f23e7a5534b86df0efc096ca0a1) build from the source
  
  
  • Storage backend in use: zfs
  
  

Issue description

Everything was fine, I tried to build from the source normally as I do weekly.
When I tried to access the LXC I got this message:

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory

Then I tried see the daemon log in debug mode and saw this message:

failed to apply update 11: Failed to add project_id column: FOREIGN KEY constraint failed

Steps

These are the steps I did to build:

make update
make deps
make
sudo systemctl start lxd

Information to attach

sudo -E $GOPATH/bin/lxd --debug --group sudo
INFO[10-10|00:27:06] LXD 3.5 is starting in normal mode       path=/var/lib/lxd
INFO[10-10|00:27:06] Kernel uid/gid map: 
INFO[10-10|00:27:06]  - u 0 0 4294967295 
INFO[10-10|00:27:06]  - g 0 0 4294967295 
INFO[10-10|00:27:06] Configured LXD uid/gid map: 
INFO[10-10|00:27:06]  - u 0 1000000 65536 
INFO[10-10|00:27:06]  - g 0 1000000 65536 
WARN[10-10|00:27:06] CGroup memory swap accounting is disabled, swap limits will be ignored. 
DBUG[10-10|00:27:06] Failed to set NETLINK_DUMP_STRICT_CHK socket option 
DBUG[10-10|00:27:06] Running kernel does not support netnsid-based network retrieval 
DBUG[10-10|00:27:06] Running kernel does not support uevent injection 
INFO[10-10|00:27:06] Initializing local database 
INFO[10-10|00:27:06] Initializing database gateway 
INFO[10-10|00:27:06] Start database node                      address= id=1
INFO[10-10|00:27:06] Raft: Restored from snapshot 1-20209-1538402836859 
INFO[10-10|00:27:06] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}] 
INFO[10-10|00:27:06] Raft: Node at 0 [Leader] entering Leader state 
INFO[10-10|00:27:06] Dqlite: starting event loop 
DBUG[10-10|00:27:06] Dqlite: accepting connections 
INFO[10-10|00:27:06] LXD isn't socket activated 
INFO[10-10|00:27:06] Starting /dev/lxd handler: 
INFO[10-10|00:27:06]  - binding devlxd socket                 socket=/var/lib/lxd/devlxd/sock
INFO[10-10|00:27:06] REST API daemon: 
INFO[10-10|00:27:06]  - binding Unix socket                   socket=/var/lib/lxd/unix.socket
INFO[10-10|00:27:06]  - binding TCP socket                    socket=[::]:8443
INFO[10-10|00:27:06] Initializing global database 
INFO[10-10|00:27:06] Dqlite: handling new connection (fd=19) 
INFO[10-10|00:27:06] Dqlite: connected address=0 attempt=0 
INFO[10-10|00:27:06] Updating the LXD global schema. Backup made as "global.bak" 
DBUG[10-10|00:27:06] Updating global DB schema from 11 to 12 
DBUG[10-10|00:27:06] Database error: &errors.errorString{s:"failed to apply update 11: Failed to add project_id column: FOREIGN KEY constraint failed"} 
EROR[10-10|00:27:06] Failed to start the daemon: failed to open cluster database: failed to ensure schema: failed to apply update 11: Failed to add project_id column: FOREIGN KEY constraint failed 
INFO[10-10|00:27:06] Starting shutdown sequence 
INFO[10-10|00:27:06] Stopping REST API handler: 
INFO[10-10|00:27:06]  - closing socket                        socket=[::]:8443
INFO[10-10|00:27:06]  - closing socket                        socket=/var/lib/lxd/unix.socket
INFO[10-10|00:27:06] Stopping /dev/lxd handler: 
INFO[10-10|00:27:06]  - closing socket                        socket=/var/lib/lxd/devlxd/sock
INFO[10-10|00:27:06] Stop database gateway 
INFO[10-10|00:27:06] Stop raft instance 
INFO[10-10|00:27:06] Dqlite: stopping event loop 
EROR[10-10|00:27:06] Dqlite: aborting (fd=19 state=header msg=(null)) 
INFO[10-10|00:27:06] Dqlite: event loop stopped 
DBUG[10-10|00:27:06] Not unmounting temporary filesystems (containers are still running) 
INFO[10-10|00:27:06] Saving simplestreams cache 
INFO[10-10|00:27:06] Saved simplestreams cache 
Error: failed to open cluster database: failed to ensure schema: failed to apply update 11: Failed to add project_id column: FOREIGN KEY constraint failed

• [ ] Any relevant kernel output (dmesg)
• [ ] Container log (lxc info NAME --show-log)
• [ ] Container configuration (lxc config show NAME --expanded)
• [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• [ ] Output of the client with --debug
• [x] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

[stgraber]

Assigning to @freeekanayaka if I don't get to look into this before going to bed

[cyberwillis]

Thank you worked nice ! :)