Lxd: LXD v3.3: error while loading shared libraries: libdqlite.so.0: cannot open shared object file: No such file or directory

Created on 2 Aug 2018  路  9Comments  路  Source: lxc/lxd

Required information

  • Distribution: Ubuntu
  • Distribution version: 16.04
  • The output of "lxc info" or if that fails:

    • Kernel version: 4.4.0-131

    • LXC version: v3.3 (from source)

    • LXD version: v3.3 (from source)

    • Storage backend in use: zfs

1-After update lxd source and execute make deps , the following messages appeared:

Please set the following in your environment (possibly ~/.bashrc)
export CGO_CFLAGS="-I/home/willismar/go/deps/sqlite/ -I/home/willismar/go/deps/dqlite/include/"
export CGO_LDFLAGS="-L/home/willismar/go/deps/sqlite/.libs/ -L/home/willismar/go/deps/dqlite/.libs/"
export LD_LIBRARY_PATH="/home/willismar/go/deps/sqlite/.libs/:/home/willismar/go/deps/dqlite/.libs/"

2-So I put that lines on my ~/.bashrc, executed make and everything build successfully, then I executed
sudo ldconfig

3-Restarted the machine just to make sure everything was ok and for my surprise LXD wasn't initiated.
journalctl -xe showed this:

error while loading shared libraries: libdqlite.so.0: cannot open shared object file: No such file or directory

4-Tried to debug the lxd with: sudo -E $GOPATH/bin/lxd --debug --group root

error while loading shared libraries: libdqlite.so.0: cannot open shared object file: No such file or directory

5-The libraries generated by make deps wasn't listed by ldconfig -p | grep libdqlite

6-So I created a /etc/ld.so.conf.d/lxd.conf putting together the last like of LD_LIBRARY_PATH sugested:

/home/willismar/go/deps/sqlite/.libs/
/home/willismar/go/deps/dqlite/.libs/

7-Aparently worked but , kept using other version of sqlited not patched causing conflicts to read the database. sudo -E $GOPATH/bin/lxd --debug --group root

DBUG[08-01|18:26:16] Connecting to a local LXD over a Unix socket 
DBUG[08-01|18:26:16] Sending request to LXD                   etag= method=GET url=http://unix.socket/1.0
INFO[08-01|18:26:16] LXD 3.3 is starting in normal mode       path=/var/lib/lxd
INFO[08-01|18:26:16] Kernel uid/gid map: 
INFO[08-01|18:26:16]  - u 0 0 4294967295 
INFO[08-01|18:26:16]  - g 0 0 4294967295 
INFO[08-01|18:26:16] Configured LXD uid/gid map: 
INFO[08-01|18:26:16]  - u 0 100000 65536 
INFO[08-01|18:26:16]  - g 0 100000 65536 
WARN[08-01|18:26:17] CGroup memory swap accounting is disabled, swap limits will be ignored. 
INFO[08-01|18:26:17] Initializing local database 
INFO[08-01|18:26:17] Initializing database gateway 
INFO[08-01|18:26:17] Start database node                      address= id=1
*** Error in `/home/willismar/go/bin/lxd': free(): invalid next size (fast): 0x0000000002b26420 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f524c3c37e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f524c3cc37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f524c3d053c]
/usr/local/lib/libsqlite3.so.0(sqlite3_free+0x4b)[0x7f524c94be7b]
/home/willismar/go/deps/dqlite/.libs/libdqlite.so.0(dqlite_file_write+0x191)[0x7f524c71e2f1]
/home/willismar/go/bin/lxd(_cgo_a624ee52d5f0_Cfunc_dqlite_file_write+0x44)[0xdb84b4]
/home/willismar/go/bin/lxd[0x4644e0]
======= Memory map: ========
00400000-01517000 r-xp 00000000 08:01 2496892                            /home/willismar/go/bin/lxd
01717000-01718000 r--p 01117000 08:01 2496892                            /home/willismar/go/bin/lxd
01718000-0176d000 rw-p 01118000 08:01 2496892                            /home/willismar/go/bin/lxd
0176d000-0179a000 rw-p 00000000 00:00 0 
02aeb000-02b30000 rw-p 00000000 00:00 0                                  [heap]
c000000000-c000001000 rw-p 00000000 00:00 0 
c41ffe0000-c420400000 rw-p 00000000 00:00 0 
7f5234000000-7f5234021000 rw-p 00000000 00:00 0 
7f5234021000-7f5238000000 ---p 00000000 00:00 0 
7f523c000000-7f523c021000 rw-p 00000000 00:00 0 
7f523c021000-7f5240000000 ---p 00000000 00:00 0 
7f5242bd7000-7f5242bed000 r-xp 00000000 08:01 1049103                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5242bed000-7f5242dec000 ---p 00016000 08:01 1049103                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5242dec000-7f5242ded000 rw-p 00015000 08:01 1049103                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5242ded000-7f52435ed000 r--s 00000000 08:01 2231921                    /var/lib/lxd/database/global/logs.db
7f52435ed000-7f52435ee000 ---p 00000000 00:00 0 
7f52435ee000-7f5243dee000 rw-p 00000000 00:00 0 
7f5243dee000-7f5243df9000 r-xp 00000000 08:01 1049141                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5243df9000-7f5243ff8000 ---p 0000b000 08:01 1049141                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5243ff8000-7f5243ff9000 r--p 0000a000 08:01 1049141                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5243ff9000-7f5243ffa000 rw-p 0000b000 08:01 1049141                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5243ffa000-7f5244000000 rw-p 00000000 00:00 0 
7f5244000000-7f5244021000 rw-p 00000000 00:00 0 
7f5244021000-7f5248000000 ---p 00000000 00:00 0 
7f52480ff000-7f524810a000 r-xp 00000000 08:01 1049145                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f524810a000-7f5248309000 ---p 0000b000 08:01 1049145                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5248309000-7f524830a000 r--p 0000a000 08:01 1049145                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f524830a000-7f524830b000 rw-p 0000b000 08:01 1049145                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f524830b000-7f5248321000 r-xp 00000000 08:01 1049135                    /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5248321000-7f5248520000 ---p 00016000 08:01 1049135                    /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5248520000-7f5248521000 r--p 00015000 08:01 1049135                    /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5248521000-7f5248522000 rw-p 00016000 08:01 1049135                    /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5248522000-7f5248524000 rw-p 00000000 00:00 0 
7f5248524000-7f524852c000 r-xp 00000000 08:01 1049137                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f524852c000-7f524872b000 ---p 00008000 08:01 1049137                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f524872b000-7f524872c000 r--p 00007000 08:01 1049137                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f524872c000-7f524872d000 rw-p 00008000 08:01 1049137                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f524872d000-7f524872e000 ---p 00000000 00:00 0 
7f524872e000-7f5248f2e000 rw-p 00000000 00:00 0 
7f5248f2e000-7f5248f2f000 ---p 00000000 00:00 0 
7f5248f2f000-7f524972f000 rw-p 00000000 00:00 0 
7f524972f000-7f5249730000 ---p 00000000 00:00 0 
7f5249730000-7f5249f30000 rw-p 00000000 00:00 0 
7f5249f30000-7f5249f31000 ---p 00000000 00:00 0 
7f5249f31000-7f524a731000 rw-p 00000000 00:00 0 
7f524a731000-7f524a732000 ---p 00000000 00:00 0 
7f524a732000-7f524af32000 rw-p 00000000 00:00 0 
7f524af32000-7f524af39000 r-xp 00000000 08:01 1049170                    /lib/x86_64-linux-gnu/librt-2.23.so
7f524af39000-7f524b138000 ---p 00007000 08:01 1049170                    /lib/x86_64-linux-gnu/librt-2.23.so
7f524b138000-7f524b139000 r--p 00006000 08:01 1049170                    /lib/x86_64-linux-gnu/librt-2.23.so
7f524b139000-7f524b13a000 rw-p 00007000 08:01 1049170                    /lib/x86_64-linux-gnu/librt-2.23.so
7f524b13a000-7f524b1a8000 r-xp 00000000 08:01 1049157                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f524b1a8000-7f524b3a8000 ---p 0006e000 08:01 1049157                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f524b3a8000-7f524b3a9000 r--p 0006e000 08:01 1049157                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f524b3a9000-7f524b3aa000 rw-p 0006f000 08:01 1049157                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f524b3aa000-7f524b3cc000 r-xp 00000000 08:01 398305                     /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
7f524b3cc000-7f524b5cb000 ---p 00022000 08:01 398305                     /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
7f524b5cb000-7f524b5cc000 r--p 00021000 08:01 398305                     /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
7f524b5cc000-7f524b5cd000 rw-p 00022000 08:01 398305                     /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
7f524b5cd000-7f524b5d0000 r-xp 00000000 08:01 1049091                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f524b5d0000-7f524b7cf000 ---p 00003000 08:01 1049091                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f524b7cf000-7f524b7d0000 r--p 00002000 08:01 1049091                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f524b7d0000-7f524b7d1000 rw-p 00003000 08:01 1049091                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f524b7d1000-7f524b8d9000 r-xp 00000000 08:01 1049118                    /lib/x86_64-linux-gnu/libm-2.23.so
7f524b8d9000-7f524bad8000 ---p 00108000 08:01 1049118                    /lib/x86_64-linux-gnu/libm-2.23.so
7f524bad8000-7f524bad9000 r--p 00107000 08:01 1049118                    /lib/x86_64-linux-gnu/libm-2.23.so
7f524bad9000-7f524bada000 rw-p 00108000 08:01 1049118                    /lib/x86_64-linux-gnu/libm-2.23.so
7f524bada000-7f524bade000 r-xp 00000000 08:01 1049067                    /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f524bade000-7f524bcdd000 ---p 00004000 08:01 1049067                    /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f524bcdd000-7f524bcde000 r--p 00003000 08:01 1049067                    /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f524bcde000-7f524bcdf000 rw-p 00004000 08:01 1049067                    /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f524bcdf000-7f524bce3000 r-xp 00000000 08:01 1049080                    /lib/x86_64-linux-gnu/libcap.so.2.24
7f524bce3000-7f524bee3000 ---p 00004000 08:01 1049080                    /lib/x86_64-linux-gnu/libcap.so.2.24
7f524bee3000-7f524bee4000 r--p 00004000 08:01 1049080                    /lib/x86_64-linux-gnu/libcap.so.2.24
7f524bee4000-7f524bee5000 rw-p 00005000 08:01 1049080                    /lib/x86_64-linux-gnu/libcap.so.2.24
7f524bee5000-7f524bf13000 r-xp 00000000 08:01 1049173                    /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f524bf13000-7f524c113000 ---p 0002e000 08:01 1049173                    /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f524c113000-7f524c129000 r--p 0002e000 08:01 1049173                    /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f524c129000-7f524c12a000 rw-p 00044000 08:01 1049173                    /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f524c12a000-7f524c149000 r-xp 00000000 08:01 1049174                    /lib/x86_64-linux-gnu/libselinux.so.1
7f524c149000-7f524c348000 ---p 0001f000 08:01 1049174                    /lib/x86_64-linux-gnu/libselinux.so.1
7f524c348000-7f524c349000 r--p 0001e000 08:01 1049174                    /lib/x86_64-linux-gnu/libselinux.so.1
7f524c349000-7f524c34a000 rw-p 0001f000 08:01 1049174                    /lib/x86_64-linux-gnu/libselinux.so.1
7f524c34a000-7f524c34c000 rw-p 00000000 00:00 0 
7f524c34c000-7f524c50c000 r-xp 00000000 08:01 1049077                    /lib/x86_64-linux-gnu/libc-2.23.so
7f524c50c000-7f524c70c000 ---p 001c0000 08:01 1049077                    /lib/x86_64-linux-gnu/libc-2.23.so
7f524c70c000-7f524c710000 r--p 001c0000 08:01 1049077                    /lib/x86_64-linux-gnu/libc-2.23.so
7f524c710000-7f524c712000 rw-p 001c4000 08:01 1049077                    /lib/x86_64-linux-gnu/libc-2.23.so
7f524c712000-7f524c716000 rw-p 00000000 00:00 0 
7f524c716000-7f524c72f000 r-xp 00000000 08:01 2495415                    /home/willismar/go/deps/dqlite/.libs/libdqlite.so.0.0.1
7f524c72f000-7f524c92e000 ---p 00019000 08:01 2495415                    /home/willismar/go/deps/dqlite/.libs/libdqlite.so.0.0.1
7f524c92e000-7f524c92f000 r--p 00018000 08:01 2495415                    /home/willismar/go/deps/dqlite/.libs/libdqlite.so.0.0.1
7f524c92f000-7f524c930000 rw-p 00019000 08:01 2495415                    /home/willismar/go/deps/dqlite/.libs/libdqlite.so.0.0.1
7f524c930000-7f524ca2d000 r-xp 00000000 08:01 398290                     /usr/local/lib/libsqlite3.so.0.8.6
7f524ca2d000-7f524cc2c000 ---p 000fd000 08:01 398290                     /usr/local/lib/libsqlite3.so.0.8.6
7f524cc2c000-7f524cc2f000 r--p 000fc000 08:01 398290                     /usr/local/lib/libsqlite3.so.0.8.6
7f524cc2f000-7f524cc32000 rw-p 000ff000 08:01 398290                     /usr/local/lib/libsqlite3.so.0.8.6
7f524cc32000-7f524cc39000 r-xp 00000000 08:01 1049059                    /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f524cc39000-7f524ce38000 ---p 00007000 08:01 1049059                    /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f524ce38000-7f524ce39000 r--p 00006000 08:01 1049059                    /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f524ce39000-7f524ce3a000 rw-p 00007000 08:01 1049059                    /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f524ce3a000-7f524ce52000 r-xp 00000000 08:01 1049164                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f524ce52000-7f524d051000 ---p 00018000 08:01 1049164                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f524d051000-7f524d052000 r--p 00017000 08:01 1049164                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f524d052000-7f524d053000 rw-p 00018000 08:01 1049164                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f524d053000-7f524d057000 rw-p 00000000 00:00 0 
7f524d057000-7f524d059000 r-xp 00000000 08:01 1049193                    /lib/x86_64-linux-gnu/libutil-2.23.so
7f524d059000-7f524d258000 ---p 00002000 08:01 1049193                    /lib/x86_64-linux-gnu/libutil-2.23.so
7f524d258000-7f524d259000 r--p 00001000 08:01 1049193                    /lib/x86_64-linux-gnu/libutil-2.23.so
7f524d259000-7f524d25a000 rw-p 00002000 08:01 1049193                    /lib/x86_64-linux-gnu/libutil-2.23.so
7f524d25a000-7f524d327000 r-xp 00000000 08:01 397971                     /usr/local/lib/liblxc.so.1.4.0
7f524d327000-7f524d526000 ---p 000cd000 08:01 397971                     /usr/local/lib/liblxc.so.1.4.0
7f524d526000-7f524d528000 r--p 000cc000 08:01 397971                     /usr/local/lib/liblxc.so.1.4.0
7f524d528000-7f524d52b000 rw-p 000ce000 08:01 397971                     /usr/local/lib/liblxc.so.1.4.0
7f524d52b000-7f524d551000 r-xp 00000000 08:01 1049053                    /lib/x86_64-linux-gnu/ld-2.23.so
7f524d69d000-7f524d746000 rw-p 00000000 00:00 0 
7f524d74f000-7f524d750000 rw-p 00000000 00:00 0 
7f524d750000-7f524d751000 r--p 00025000 08:01 1049053                    /lib/x86_64-linux-gnu/ld-2.23.so
7f524d751000-7f524d752000 rw-p 00026000 08:01 1049053                    /lib/x86_64-linux-gnu/ld-2.23.so
7f524d752000-7f524d753000 rw-p 00000000 00:00 0 
7ffe112fb000-7ffe1131c000 rw-p 00000000 00:00 0                          [stack]
7ffe113e7000-7ffe113ea000 r--p 00000000 00:00 0                          [vvar]
7ffe113ea000-7ffe113ec000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
SIGABRT: abort
PC=0x7f524c381428 m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f524c381428
stack: frame={sp:0x7ffe113197a8, fp:0x0} stack=[0x7ffe10b1ad28,0x7ffe11319d50)
00007ffe113196a8:  666666666666660a  3030303036666666 
00007ffe113196b8:  6666666666662d30  3031303666666666 
00007ffe113196c8:  2070782d72203030  3030303030303030 
00007ffe113196d8:  302030303a303020  2020202020202020 
00007ffe113196e8:  2020202020202020  63737973765b2020 
00007ffe113196f8:  636f6c0a5d6c6c61  6c2f62696c2f6c61 
00007ffe11319708:  6f732e63786c6269  370a302e342e312e 
00007ffe11319718:  3632356434323566  323566372d303030 
00007ffe11319728:  3030303832356434  303020702d2d7220 
00007ffe11319738:  3020303030636330  3739332031303a38 
00007ffe11319748:  2020202020313739  2020202020202020 
00007ffe11319758:  2020202020202020  636f6c2f7273752f 
00007ffe11319768:  6c2f62696c2f6c61  6f732e63786c6269 
00007ffe11319778:  0000000000000064  00007ffe11319b40 
00007ffe11319788:  0000000000000064  00007ffe11319958 
00007ffe11319798:  00007ffe11319958  0000000000000002 
00007ffe113197a8: <00007f524c38302a  0000000000000020 
00007ffe113197b8:  0000000000000000  0000000000000000 
00007ffe113197c8:  0000000000000000  0000000000000000 
00007ffe113197d8:  0000000000000000  0000000000000000 
00007ffe113197e8:  0000000000000000  0000000000000000 
00007ffe113197f8:  0000000000000000  0000000000000000 
00007ffe11319808:  0000000000000000  0000000000000000 
00007ffe11319818:  0000000000000000  0000000000000000 
00007ffe11319828:  0000000000000000  3638782f62696c2f 
00007ffe11319838:  756e696c2d34365f  646c2f756e672d78 
00007ffe11319848:  6f732e33322e322d  366434323566370a 
00007ffe11319858:  66372d3030306439  3036343764343235 
00007ffe11319868:  20702d7772203030  3030303030303030 
00007ffe11319878:  302030303a303020  6434323566370a20 
00007ffe11319888:  372d303030663437  3035376434323566 
00007ffe11319898:  702d777220303030  3030303030303020 
runtime: unknown pc 0x7f524c381428
stack: frame={sp:0x7ffe113197a8, fp:0x0} stack=[0x7ffe10b1ad28,0x7ffe11319d50)
00007ffe113196a8:  666666666666660a  3030303036666666 
00007ffe113196b8:  6666666666662d30  3031303666666666 
00007ffe113196c8:  2070782d72203030  3030303030303030 
00007ffe113196d8:  302030303a303020  2020202020202020 
00007ffe113196e8:  2020202020202020  63737973765b2020 
00007ffe113196f8:  636f6c0a5d6c6c61  6c2f62696c2f6c61 
00007ffe11319708:  6f732e63786c6269  370a302e342e312e 
00007ffe11319718:  3632356434323566  323566372d303030 
00007ffe11319728:  3030303832356434  303020702d2d7220 
00007ffe11319738:  3020303030636330  3739332031303a38 
00007ffe11319748:  2020202020313739  2020202020202020 
00007ffe11319758:  2020202020202020  636f6c2f7273752f 
00007ffe11319768:  6c2f62696c2f6c61  6f732e63786c6269 
00007ffe11319778:  0000000000000064  00007ffe11319b40 
00007ffe11319788:  0000000000000064  00007ffe11319958 
00007ffe11319798:  00007ffe11319958  0000000000000002 
00007ffe113197a8: <00007f524c38302a  0000000000000020 
00007ffe113197b8:  0000000000000000  0000000000000000 
00007ffe113197c8:  0000000000000000  0000000000000000 
00007ffe113197d8:  0000000000000000  0000000000000000 
00007ffe113197e8:  0000000000000000  0000000000000000 
00007ffe113197f8:  0000000000000000  0000000000000000 
00007ffe11319808:  0000000000000000  0000000000000000 
00007ffe11319818:  0000000000000000  0000000000000000 
00007ffe11319828:  0000000000000000  3638782f62696c2f 
00007ffe11319838:  756e696c2d34365f  646c2f756e672d78 
00007ffe11319848:  6f732e33322e322d  366434323566370a 
00007ffe11319858:  66372d3030306439  3036343764343235 
00007ffe11319868:  20702d7772203030  3030303030303030 
00007ffe11319878:  302030303a303020  6434323566370a20 
00007ffe11319888:  372d303030663437  3035376434323566 
00007ffe11319898:  702d777220303030  3030303030303020 

goroutine 1 [syscall]:
runtime.cgocall(0xdb8470, 0xc420238a18, 0x8)
    /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc4202389d8 sp=0xc4202389a0 pc=0x40c6b4
github.com/CanonicalLtd/go-dqlite/internal/bindings._Cfunc_dqlite_file_write(0x2b261a8, 0x2b260c0, 0xc4202f9000, 0x1000, 0xc400000000)
    _cgo_gotypes.go:315 +0x4d fp=0xc420238a18 sp=0xc4202389d8 pc=0x7a5b7d
github.com/CanonicalLtd/go-dqlite/internal/bindings.(*Vfs).WriteFile(0x2b260e8, 0xc420273237, 0x6, 0xc4202f9000, 0x1000, 0x1000, 0x0, 0x0)
    /home/willismar/go/src/github.com/CanonicalLtd/go-dqlite/internal/bindings/vfs.go:94 +0xa7 fp=0xc420238a70 sp=0xc420238a18 pc=0x7aa857
github.com/CanonicalLtd/go-dqlite/internal/connection.Restore(0x2b260e8, 0xc420273237, 0x6, 0xc4202f9000, 0x1000, 0x1000, 0xc4202fc000, 0x3c5c0, 0x3c5c0, 0x0, ...)
    /home/willismar/go/src/github.com/CanonicalLtd/go-dqlite/internal/connection/snapshot.go:29 +0x6d fp=0xc420238ae0 sp=0xc420238a70 pc=0x8dc6ad
github.com/CanonicalLtd/go-dqlite/internal/replication.(*FSM).restoreDatabase(0xc42026dee0, 0xc4202fa000, 0x10ddf00, 0xc42028ec30, 0x0, 0x0, 0x0)
    /home/willismar/go/src/github.com/CanonicalLtd/go-dqlite/internal/replication/fsm.go:654 +0x7e8 fp=0xc420238d40 sp=0xc420238ae0 pc=0x8e0da8
github.com/CanonicalLtd/go-dqlite/internal/replication.(*FSM).Restore(0xc42026dee0, 0x10ddf00, 0xc42028ec30, 0x0, 0x0)
    /home/willismar/go/src/github.com/CanonicalLtd/go-dqlite/internal/replication/fsm.go:559 +0x29d fp=0xc420238e30 sp=0xc420238d40 pc=0x8e04ad
github.com/hashicorp/raft.(*Raft).restoreSnapshot(0xc4200ef600, 0x0, 0x0)
    /home/willismar/go/src/github.com/hashicorp/raft/api.go:530 +0x23d fp=0xc420238f40 sp=0xc420238e30 pc=0x8af33d
github.com/hashicorp/raft.NewRaft(0xc4201d8360, 0x10e13c0, 0xc42026dee0, 0x10e8840, 0xc42026dd40, 0x10e5640, 0xc42026dd40, 0x10e1640, 0xc42026de40, 0x10ebae0, ...)
    /home/willismar/go/src/github.com/hashicorp/raft/api.go:481 +0x7eb fp=0xc420239118 sp=0xc420238f40 pc=0x8ae71b
github.com/lxc/lxd/lxd/cluster.raftInstanceInit(0xc42026d080, 0xc42026dc00, 0xc4202767e0, 0x4008000000000000, 0x1, 0xc420239348, 0x712776)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/cluster/raft.go:191 +0x5c9 fp=0xc4202392e0 sp=0xc420239118 pc=0xac57c9
github.com/lxc/lxd/lxd/cluster.newRaft(0xc42026d080, 0xc4202767e0, 0x4008000000000000, 0x0, 0x0, 0xc42026da20)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/cluster/raft.go:71 +0x24d fp=0xc420239358 sp=0xc4202392e0 pc=0xac50bd
github.com/lxc/lxd/lxd/cluster.(*Gateway).init(0xc42025eea0, 0xc42026da20, 0xc420039260)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/cluster/gateway.go:448 +0x84 fp=0xc420239410 sp=0xc420239358 pc=0xabe244
github.com/lxc/lxd/lxd/cluster.NewGateway(0xc42026d080, 0xc4202767e0, 0xc4202396b8, 0x2, 0x2, 0xc420040c00, 0x22, 0xc4202394e8)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/cluster/gateway.go:57 +0x1f2 fp=0xc420239488 sp=0xc420239410 pc=0xabcf52
main.(*Daemon).init(0xc4202532b0, 0xc420239900, 0x40e5b6)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/daemon.go:424 +0x4a8 fp=0xc4202398b8 sp=0xc420239488 pc=0xc104e8
main.(*Daemon).Init(0xc4202532b0, 0xc4202532b0, 0xc420038d20)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/daemon.go:363 +0x2f fp=0xc420239910 sp=0xc4202398b8 pc=0xc0ff9f
main.(*cmdDaemon).Run(0xc4201fac80, 0xc420248000, 0xc42024e120, 0x0, 0x3, 0x0, 0x0)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/main_daemon.go:61 +0x266 fp=0xc420239b10 sp=0xc420239910 pc=0xc42ee6
main.(*cmdDaemon).Run-fm(0xc420248000, 0xc42024e120, 0x0, 0x3, 0x0, 0x0)
    /home/willismar/go/src/github.com/lxc/lxd/lxd/main_daemon.go:36 +0x52 fp=0xc420239b58 sp=0xc420239b10 pc=0xda5702
github.com/spf13/cobra.(*Command).execute(0xc420248000, 0xc4200300d0, 0x3, 0x3, 0xc420248000, 0xc4200300d0)
    /home/willismar/go/src/github.com/spf13/cobra/command.go:762 +0x468 fp=0xc420239c48 sp=0xc420239b58 pc=0xb3f028
github.com/spf13/cobra.(*Command).ExecuteC(0xc420248000, 0x0, 0xc420250500, 0xc420250500)
    /home/willismar/go/src/github.com/spf13/cobra/command.go:852 +0x30a fp=0xc420239d88 sp=0xc420239c48 pc=0xb3fa3a
github.com/spf13/cobra.(*Command).Execute(0xc420248000, 0xc420239e08, 0x1)
    /home/willismar/go/src/github.com/spf13/cobra/command.go:800 +0x2b fp=0xc420239db8 sp=0xc420239d88 pc=0xb3f70b
main.main()
    /home/willismar/go/src/github.com/lxc/lxd/lxd/main.go:160 +0xe26 fp=0xc420239f88 sp=0xc420239db8 pc=0xc411b6
runtime.main()
    /usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc420239fe0 sp=0xc420239f88 pc=0x437b82
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420239fe8 sp=0xc420239fe0 pc=0x465811

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 9 [select]:
database/sql.(*DB).connectionOpener(0xc42021bae0, 0x10e4fc0, 0xc42026e800)
    /usr/local/go/src/database/sql/sql.go:935 +0x119
created by database/sql.OpenDB
    /usr/local/go/src/database/sql/sql.go:634 +0x178

goroutine 10 [select]:
database/sql.(*DB).connectionResetter(0xc42021bae0, 0x10e4fc0, 0xc42026e800)
    /usr/local/go/src/database/sql/sql.go:948 +0x12a
created by database/sql.OpenDB
    /usr/local/go/src/database/sql/sql.go:635 +0x1ae

rax    0x0
rbx    0x64
rcx    0x7f524c381428
rdx    0x6
rdi    0xd20
rsi    0xd20
rbp    0x7ffe11319b40
rsp    0x7ffe113197a8
r8     0x8
r9     0x0
r10    0x8
r11    0x202
r12    0x64
r13    0x7ffe11319958
r14    0x7ffe11319958
r15    0x2
rip    0x7f524c381428
rflags 0x202
cs     0x33
fs     0x0
gs     0x0

7-Then by this fragment from debug, I identified that the some other libraries was interfering :
/usr/local/lib/libsqlite3.so.0(sqlite3_free+0x4b)[0x7f524c94be7b]

...
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f524c3c37e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f524c3cc37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f524c3d053c]
/usr/local/lib/libsqlite3.so.0(sqlite3_free+0x4b)[0x7f524c94be7b]
/home/willismar/go/deps/dqlite/.libs/libdqlite.so.0(dqlite_file_write+0x191)[0x7f524c71e2f1]
/home/willismar/go/bin/lxd(_cgo_a624ee52d5f0_Cfunc_dqlite_file_write+0x44)[0xdb84b4]
/home/willismar/go/bin/lxd[0x4644e0]
======= Memory map: ========
...

8-Using the command ldconfig -p | grep -E 'sqlite|dqlite' I could prove that earlier libraries was in the system:

libsqlite3.so.0 (libc6,x86-64) => /usr/local/lib/libsqlite3.so.0
libsqlite3.so.0 (libc6,x86-64) => /home/willismar/go/deps/sqlite/.libs/libsqlite3.so.0
libsqlite3.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
libsqlite3.so (libc6,x86-64) => /usr/local/lib/libsqlite3.so
libsqlite3.so (libc6,x86-64) => /home/willismar/go/deps/sqlite/.libs/libsqlite3.so
libdqlite.so.0 (libc6,x86-64) => /home/willismar/go/deps/dqlite/.libs/libdqlite.so.0
libdqlite.so (libc6,x86-64) => /home/willismar/go/deps/dqlite/.libs/libdqlite.so

8- Then I forced to uninstall the earlier libraries to see if it would work at all and it did worked. However I don't feel that putting that lined on /etc/ld.so.conf.d/lxd.conf was a good approach. Can you recommend something else to try not mess with the earlier libraries ?

  • [ ] Any relevant kernel output (dmesg)
  • [ ] Container log (lxc info NAME --show-log)
  • [ ] Container configuration (lxc config show NAME --expanded)
  • [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
  • [ ] Output of the client with --debug
  • [x] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)
picture cyberwillis

Most helpful comment

For other folks on deb-based distros wanting to build lxd from source, this PPA:

https://code.launchpad.net/~dqlite-maintainers/+archive/ubuntu/master

has builds for SQLite and dqlite. For now the PPA has only an 18.04 track, but I'll add other LTS releases and the current devel one.

picture freeekanayaka on 2 Aug 2018
🎉2 👍1

All 9 comments

You could probably just set LD_LIBRARY_PATH in the environment before starting lxd, that's effectively what we do in the Ubuntu package. I believe this takes precedence over any ldconfig and so will select your build of libsqlite and libdqlite over the system one.

stgraber picture stgraber on 2 Aug 2018
👍1

This is 1:1 my issue from today. :)

brauner picture brauner on 2 Aug 2018
👍1

Sadly my ubuntu don't accept fixing the LD_LIBRARY_PATH on /etc/environment file, I also tried in a Virtual Machine before.

So I decided to just throw everything over my library folder /usr/local/lib and the xxx.la on the parent.

Everything is working and I am happy :D

cyberwillis picture cyberwillis on 2 Aug 2018

That's one way of doing it, the other would be in your systemd unit file or init script to have the environment variable be set in there. Another solution which is what we do in the package is to wrap LXD itself, effectively have lxd.real be the binary and lxd be a shell script which sets LD_LIBRARY_PATH and then exec lxd.real.

stgraber picture stgraber on 2 Aug 2018
👍1

LD_LIBRARY_PATH on /etc/environment finally worked!

:D

cyberwillis picture cyberwillis on 2 Aug 2018

For other folks on deb-based distros wanting to build lxd from source, this PPA:

https://code.launchpad.net/~dqlite-maintainers/+archive/ubuntu/master

has builds for SQLite and dqlite. For now the PPA has only an 18.04 track, but I'll add other LTS releases and the current devel one.

freeekanayaka picture freeekanayaka on 2 Aug 2018
🎉2 👍1

I am experiencing this same issue on Debian, and the link you provided is down... What can i do? I don't understand how to set LD_LIBRARY_PATH exactly

marco-trovato picture marco-trovato on 23 Sep 2019

LD_LIBRARY_PATH must point to the directory where libdqlite.so.0 is, and the lxd process must be started with that environment variable set.

freeekanayaka picture freeekanayaka on 23 Sep 2019

Thank you freeekanayaka

marco-trovato picture marco-trovato on 23 Sep 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

sajoupa picture sajoupa  路  4Comments
srkunze picture srkunze  路  3Comments
rrva picture rrva  路  5Comments
sforteva picture sforteva  路  3Comments
jsnjack picture jsnjack  路  3Comments