Zeronet: Rebuilding content.db used alot of RAM and end up my PC not responding

[slrslr]

Step 1: Please describe your environment

• ZeroNet version: 0.6.4 r3703, Python 2.7.11
• Operating system: Linux Debian
• Tor status: not available
• Opened port: no
• Special configuration: multiuser_local

Step 2: Describe the problem:

Hello, since i deleted the data/content.db (log said that the database is malformed) when i attempt to launch ZeroNet, it begin to rebuild the database. The RAM usage of the Zeronet"s python process was growing to the point it used like 70% of the memory (4GB RAM, swappiness 10) according to htop, ps auxf.

user      7517  0.0  0.0  48064   924 pts/0    S+   16:28   0:00                  |   |   \_ grep py
user      6569  0.0  0.0  46476   572 pts/1    S+   16:18   0:00                  |   |       \_ /bin/bash ../Python/python start.py
user      6571  101 **61.9** 3664920 2504984 pts/1 R+   16:18  10:09                  |   |           \_ ../Python/python2.7 start.py
user      3854  0.0  0.0 283204  3944 ?        Sl   15:58   0:00                  \_ /usr/bin/python3 /usr/share/system-config-printer/applet.py

The debug.log shows alot of lines that seems alright, there is repetition of lines like:
[2018-11-09 *] WARNING Site:1EEZTd..7X3t data/users/content.json parse error: OSError: [Errno 2] No such file or directory: 'media/veracrypt1/ZeroNet/data/1EEZTdR6oFdeuLzryXE7ga5Y2dijmK7X3t/data/users/' in ContentManager.py line 236

Within a few minutes, computer gets trapped into constant primary/system HDD fulldisk activity (i thing because caching or such), primary/system HDD do not contain my data/ dir, only zeronet app files.), data dir is on ext. HDD (which was almost idle (according to noise) while my primary HDD was fully utilized for many minutes) and computer not responding or with like minute delay.

I had to reset my computer, it happened 2 times, then third time i let it continue hammering my HDD, it took maybe 30 minutes, then i found:
[16:58:10] - Creating UiServer....
../Python/python: line 4: 5851 Killed $DIR/python2.7 "$@"

launching ZN says "DatabaseError: database disk image is malformed" so probably incomplete db rebuild.

I sent the log to the mailbox hello@zero*.io as requested here.

[DaniellMesquita]

Maybe it is fixed when porting to Python 3
Reference: https://github.com/HelloZeroNet/ZeroNet/issues/1773

[slrslr]

The issue is still there.

Had to reset my computer two times today during rebuild of content.db
it got stuck not responding for more than 1,5 hour with full sytem/swap drive utilization.
Next time it started responding so i had opportunity to check process stats. It is using 70% of my RAM (more than 4GB)
Please check how the zeronet is handling ram during rebuild to prevent excessive RAM usage. I have hundreds of sites and more than 100GB of data on HDD. This is Linux and latest yesterday updated zeronet. The rebuild not completed successfully, as sites do not works, one or more has incomplete DBs.

$ ps auxf # 4+GB RAM used by zeronet

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
myname 4769 0.0 0.0 56560 360 pts/0 Ss 22:58 0:00 _ bash
myname 8033 83.0 77.2 6347212 4717272 pts/0 Sl+ 23:35 18:23 | _ python /home/myname/ZeroBundle/ZeroNet/zeronet.py --tor_controller 127.0.0.1:9151 --tor_proxy 127.0.0.1:9150

$ lsof -p 8033

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 8033 myname cwd DIR 8,2 4096 487526 /home/myname/ZeroBundle/ZeroNet
python 8033 myname rtd DIR 8,2 4096 2 /
python 8033 myname txt REG 8,2 3779512 270969 /usr/bin/python2.7
python 8033 myname mem-r REG 254,1 2523136 536822 /media/veracrypt2/ZeroNet/data/chart.db-shm
python 8033 myname mem REG 8,2 2585402 412965 /home/myname/ZeroBundle/lib/libcrypto.so.1.0.0
python 8033 myname mem REG 8,2 35296 275394 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
python 8033 myname mem-r REG 254,1 32768 536825 /media/veracrypt2/ZeroNet/data/content.db-shm
python 8033 myname mem REG 8,2 148376 273860 /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 1078304 276338 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
python 8033 myname mem REG 8,2 96024 273871 /usr/lib/python2.7/lib-dynload/_sqlite3.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 70536 273867 /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 170128 260103 /lib/x86_64-linux-gnu/libexpat.so.1.6.2
python 8033 myname mem REG 8,2 77408 273885 /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 29352 273865 /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 98096 415327 /usr/lib/python2.7/dist-packages/msgpack/_unpacker.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 92584 260110 /lib/x86_64-linux-gnu/libgcc_s.so.1
python 8033 myname mem REG 8,2 1566168 276353 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
python 8033 myname mem REG 8,2 78544 415326 /usr/lib/python2.7/dist-packages/msgpack/_packer.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 82624 413343 /usr/lib/python2.7/dist-packages/gevent/_semaphore.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 2711616 272561 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
python 8033 myname mem REG 8,2 442984 272562 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
python 8033 myname mem REG 8,2 110472 273872 /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 384456 413356 /usr/lib/python2.7/dist-packages/gevent/corecext.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 29672 357732 /usr/lib/python2.7/dist-packages/greenlet.x86_64-linux-gnu.so
python 8033 myname mem REG 8,2 37806080 272973 /usr/lib/locale/locale-archive
python 8033 myname mem REG 8,2 1689360 260077 /lib/x86_64-linux-gnu/libc-2.24.so
python 8033 myname mem REG 8,2 1063328 260134 /lib/x86_64-linux-gnu/libm-2.24.so
python 8033 myname mem REG 8,2 105088 260237 /lib/x86_64-linux-gnu/libz.so.1.2.8
python 8033 myname mem REG 8,2 10688 260229 /lib/x86_64-linux-gnu/libutil-2.24.so
python 8033 myname mem REG 8,2 14640 260096 /lib/x86_64-linux-gnu/libdl-2.24.so
python 8033 myname mem REG 8,2 135440 260193 /lib/x86_64-linux-gnu/libpthread-2.24.so
python 8033 myname mem REG 8,2 153288 260053 /lib/x86_64-linux-gnu/ld-2.24.so
python 8033 myname 0u CHR 136,0 0t0 3 /dev/pts/0
python 8033 myname 1u CHR 136,0 0t0 3 /dev/pts/0
python 8033 myname 2u CHR 136,0 0t0 3 /dev/pts/0
python 8033 myname 4w REG 8,2 101 394001 /home/myname/ZeroBundle/ZeroNet/log/debug.log
python 8033 myname 5wW REG 254,1 0 425435 /media/veracrypt2/ZeroNet/data/lock.pid
python 8033 myname 8u a_inode 0,13 0 9122 [eventpoll]
python 8033 myname 9u a_inode 0,13 0 9122 [eventfd]
python 8033 myname 10ur REG 254,1 159925248 522941 /media/veracrypt2/ZeroNet/data/chart.db
python 8033 myname 11u REG 254,1 327358552 536821 /media/veracrypt2/ZeroNet/data/chart.db-wal
python 8033 myname 12ur REG 254,1 2523136 536822 /media/veracrypt2/ZeroNet/data/chart.db-shm
python 8033 myname 17r CHR 1,9 0t0 1033 /dev/urandom
python 8033 myname 18ur REG 254,1 26484736 566908 /media/veracrypt2/ZeroNet/data/content.db
python 8033 myname 19u REG 254,1 10526632 279456 /media/veracrypt2/ZeroNet/data/content.db-wal
python 8033 myname 20ur REG 254,1 32768 536825 /media/veracrypt2/ZeroNet/data/content.db-shm

REGARDING PYTHON3:

$ which python3
/usr/bin/python3

$ /usr/bin/python3 zeronet.py
File "zeronet.py", line 10
print "- Starting ZeroNet..."
^
SyntaxError: Missing parentheses in call to 'print'

[HelloZeroNet]

What size of your data/content.db and -wal file?
Can you please check if the situation is the same if you use --db_mode speed ?

[slrslr]

@HelloZeroNet

data/content.db

is 27MB
i have not found any file containing "wal" in zeronet/data or zeronet/ or zeronet/* directories

--db_mode speed

yes, i just did rebuilt using db_mode = speed in zeronet.conf. it was raising memory usage up to 4GB RAM at the end which then remained. -> computer was hanging and me closing nearly all apps in order to keep it from being completely unresponsive.

Debian Linux, Zeronet Version: 0.6.4 r3821, Python 2.7.13
[GCC 6.3.0 20170516], Gevent: 1.1.2