Etcd: we have set --quota-backend-bytes=10g, but etcd stil alarm mvcc: database space exceeded

[wangjingpei]

we have set --quota-backend-bytes=10g, but etcd stil alarm mvcc: database space exceeded, while the db is stil 2.1GB
ETCDCTL_API=3 ./etcdctl --endpoints=[10.207.214.48:2379] --write-out=table endpoint status
+--------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------+------------------+---------+---------+-----------+-----------+------------+
| 10.207.214.48:2379 | 5f4ef98b11eb13b4 | 3.2.6 | 2.1 GB | false | 12 | 60576124 |
+--------------------+------------------+---------+---------+-----------+-----------+------------+

[wangjingpei]

we use the latest version

[wangjingpei]

i want to know the reasion, it seemed that the param --quota-backend-bytes is invalid

[hexfusion]

we use the latest version
| 10.207.214.48:2379 | 5f4ef98b11eb13b4 | 3.2.6 | 2.1 GB | false | 12 | 60576124 |

your etcdctl output says your server is etcd 3.2.6 not latest

i want to know the reasion, it seemed that the param --quota-backend-bytes is invalid

Duplicate of https://github.com/etcd-io/etcd/issues/10299

Please provide the etcd startup flags/config and logs so that we can reproduce.

[wangjingpei]

the etcd startup flags/config is following:
./etcd --max-snapshots=0 --max-wals=0 --quota-backend-bytes=6442450944 --auto-compaction-retention=1 --config-file ./conf.yml
and following is conf.yml

name: dcs-etcd-1
data-dir: /home/work/p2p-server/etcd/data
listen-client-urls: http://10.207.214.48:2379,http://127.0.0.1:2379
advertise-client-urls: http://10.207.214.48:2379,http://127.0.0.1:2379
listen-peer-urls: http://10.207.214.48:2380
initial-advertise-peer-urls: http://10.207.214.48:2380
initial-cluster: dcs-etcd-1=http://10.207.214.48:2380,dcs-etcd-2=http://10.207.201.20:2380,dcs-etcd-3=http://10.207.201.58:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new

and the log is following:

2018-12-10 17:08:09.440707 I | etcdmain: Loading server configuration from "./conf.yml"
2018-12-10 17:08:09.441407 I | etcdmain: etcd Version: 3.2.6
2018-12-10 17:08:09.441421 I | etcdmain: Git SHA: 9d43462
2018-12-10 17:08:09.441426 I | etcdmain: Go Version: go1.8.3
2018-12-10 17:08:09.441431 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-10 17:08:09.441436 I | etcdmain: setting maximum number of CPUs to 16, total number of available CPUs is 16
2018-12-10 17:08:09.441551 I | embed: listening for peers on http://10.207.214.48:2380
2018-12-10 17:08:09.441614 I | embed: listening for client requests on 10.207.214.48:2379
2018-12-10 17:08:09.441652 I | embed: listening for client requests on 127.0.0.1:2379
2018-12-10 17:08:09.444067 I | etcdserver: name = dcs-etcd-1
2018-12-10 17:08:09.444095 I | etcdserver: data dir = /home/work/p2p-server/etcd/data
2018-12-10 17:08:09.444103 I | etcdserver: member dir = /home/work/p2p-server/etcd/data/member
2018-12-10 17:08:09.444108 I | etcdserver: heartbeat = 100ms
2018-12-10 17:08:09.444113 I | etcdserver: election = 1000ms
2018-12-10 17:08:09.444117 I | etcdserver: snapshot count = 100000
2018-12-10 17:08:09.444130 I | etcdserver: advertise client URLs = http://10.207.214.48:2379,http://127.0.0.1:2379
2018-12-10 17:08:09.444136 I | etcdserver: initial advertise peer URLs = http://10.207.214.48:2380
2018-12-10 17:08:09.444150 I | etcdserver: initial cluster = dcs-etcd-1=http://10.207.214.48:2380,dcs-etcd-2=http://10.207.201.20:2380,dcs-etcd-3=http://10.207.201.58:2380
2018-12-10 17:08:09.445741 I | etcdserver: starting member 5f4ef98b11eb13b4 in cluster 7fa2441bb5598b37
2018-12-10 17:08:09.445786 I | raft: 5f4ef98b11eb13b4 became follower at term 0
2018-12-10 17:08:09.445801 I | raft: newRaft 5f4ef98b11eb13b4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018-12-10 17:08:09.445807 I | raft: 5f4ef98b11eb13b4 became follower at term 1
2018-12-10 17:08:09.447396 W | auth: simple token is not cryptographically signed
2018-12-10 17:08:09.447902 I | rafthttp: starting peer 38c82c62e3986978...
2018-12-10 17:08:09.447938 I | rafthttp: started HTTP pipelining with peer 38c82c62e3986978
2018-12-10 17:08:09.448579 I | rafthttp: started streaming with peer 38c82c62e3986978 (writer)
2018-12-10 17:08:09.448609 I | rafthttp: started streaming with peer 38c82c62e3986978 (writer)
2018-12-10 17:08:09.450606 I | rafthttp: started peer 38c82c62e3986978
2018-12-10 17:08:09.450631 I | rafthttp: added peer 38c82c62e3986978
2018-12-10 17:08:09.450644 I | rafthttp: started streaming with peer 38c82c62e3986978 (stream MsgApp v2 reader)
2018-12-10 17:08:09.450695 I | rafthttp: started streaming with peer 38c82c62e3986978 (stream Message reader)
2018-12-10 17:08:09.450730 I | rafthttp: starting peer acdfbe3388900a7c...
2018-12-10 17:08:09.450766 I | rafthttp: started HTTP pipelining with peer acdfbe3388900a7c
2018-12-10 17:08:09.450971 I | rafthttp: started streaming with peer acdfbe3388900a7c (writer)
2018-12-10 17:08:09.451274 I | rafthttp: started streaming with peer acdfbe3388900a7c (writer)
2018-12-10 17:08:09.452629 I | rafthttp: started peer acdfbe3388900a7c
2018-12-10 17:08:09.452655 I | rafthttp: added peer acdfbe3388900a7c
2018-12-10 17:08:09.452698 I | rafthttp: started streaming with peer acdfbe3388900a7c (stream MsgApp v2 reader)
2018-12-10 17:08:09.452712 I | rafthttp: started streaming with peer acdfbe3388900a7c (stream Message reader)
2018-12-10 17:08:09.452774 I | etcdserver: starting server... [version: 3.2.6, cluster version: to_be_decided]
2018-12-10 17:08:09.453221 I | etcdserver/membership: added member 38c82c62e3986978 [http://10.207.201.58:2380] to cluster 7fa2441bb5598b37
2018-12-10 17:08:09.453340 I | etcdserver/membership: added member 5f4ef98b11eb13b4 [http://10.207.214.48:2380] to cluster 7fa2441bb5598b37
2018-12-10 17:08:09.453368 I | rafthttp: peer acdfbe3388900a7c became active
2018-12-10 17:08:09.453381 I | rafthttp: established a TCP streaming connection with peer acdfbe3388900a7c (stream MsgApp v2 reader)
2018-12-10 17:08:09.453408 I | etcdserver/membership: added member acdfbe3388900a7c [http://10.207.201.20:2380] to cluster 7fa2441bb5598b37
2018-12-10 17:08:09.453524 I | rafthttp: established a TCP streaming connection with peer acdfbe3388900a7c (stream Message reader)
2018-12-10 17:08:09.507581 I | rafthttp: established a TCP streaming connection with peer acdfbe3388900a7c (stream Message writer)
2018-12-10 17:08:09.507635 I | rafthttp: established a TCP streaming connection with peer acdfbe3388900a7c (stream MsgApp v2 writer)
2018-12-10 17:08:10.046002 I | raft: 5f4ef98b11eb13b4 is starting a new election at term 1
2018-12-10 17:08:10.046079 I | raft: 5f4ef98b11eb13b4 became candidate at term 2
2018-12-10 17:08:10.046098 I | raft: 5f4ef98b11eb13b4 received MsgVoteResp from 5f4ef98b11eb13b4 at term 2
2018-12-10 17:08:10.046109 I | raft: 5f4ef98b11eb13b4 [logterm: 1, index: 3] sent MsgVote request to 38c82c62e3986978 at term 2
2018-12-10 17:08:10.046118 I | raft: 5f4ef98b11eb13b4 [logterm: 1, index: 3] sent MsgVote request to acdfbe3388900a7c at term 2
2018-12-10 17:08:10.185941 I | raft: 5f4ef98b11eb13b4 [term: 2] received a MsgVote message with higher term from acdfbe3388900a7c [term: 5]
2018-12-10 17:08:10.185971 I | raft: 5f4ef98b11eb13b4 became follower at term 5
2018-12-10 17:08:10.185983 I | raft: 5f4ef98b11eb13b4 [logterm: 1, index: 3, vote: 0] cast MsgVote for acdfbe3388900a7c [logterm: 1, index: 3] at term 5
2018-12-10 17:08:10.186544 I | raft: raft.node: 5f4ef98b11eb13b4 elected leader acdfbe3388900a7c at term 5
2018-12-10 17:08:10.187701 I | etcdserver: published {Name:dcs-etcd-1 ClientURLs:[http://10.207.214.48:2379 http://127.0.0.1:2379]} to cluster 7fa2441bb5598b37
2018-12-10 17:08:10.187746 I | embed: ready to serve client requests
2018-12-10 17:08:10.187865 I | embed: ready to serve client requests
2018-12-10 17:08:10.188062 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2018-12-10 17:08:10.188138 N | embed: serving insecure client requests on 10.207.214.48:2379, this is strongly discouraged!
2018-12-10 17:08:10.191534 N | etcdserver/membership: set the initial cluster version to 3.0
2018-12-10 17:08:10.191604 I | etcdserver/api: enabled capabilities for version 3.0

[wangjingpei]

we have got the answer, thanks very much!!!
it seems that we must put the following param in conf.yml while in clouster inveronment， if not , the param is invalid, but there is no notify and this seems very inconvenient
--max-snapshots=0 --max-wals=0 --quota-backend-bytes=6442450944 --auto-compaction-retention=1

[hexfusion]

it seems that we must put the following param in conf.yml while in clouster inveronment， if not , the param is invalid, but there is no notify and this seems very inconvenient

The assumption is that when using a config file all configurations would be localized to that config. While I see your point we do document this. Regardless I am glad you have resolved the problem!

ref: op-guide: configuration-flags