Badger: High memory consumption

[unknwon]

Hi, I know it's a common and vague title, if some tuning options could easily help with it, would be very much appreciated!

What version of Go are you using (go version)?

$ go version
go version go1.13.4 darwin/amd64

What version of Badger are you using?

v2.0.0

Does this issue reproduce with the latest master?

Yes

What are the hardware specifications of the machine (RAM, OS, Disk)?

macOS 10.15.1, 16 GB, Flash Storage

What did you do?

Open DB (badger.Open):
https://github.com/unknwon/go-import-server/blob/5d9ecfda58a254d1bd401cc04a6554ea58efed80/main.go#L163-L166

    db, err := badger.Open(badger.DefaultOptions(path))
    if err != nil {
        return nil, nil, fmt.Errorf("open: %v", err)
    }

Read data (just 4 keys in my case) (db.View):
https://github.com/unknwon/go-import-server/blob/5d9ecfda58a254d1bd401cc04a6554ea58efed80/stats.go#L54-L94

func (s *stats) loadFromDB(db *badger.DB) error {
    return db.View(func(tx *badger.Txn) error {
        iter := tx.NewIterator(badger.DefaultIteratorOptions)
        defer iter.Close()

        for iter.Rewind(); iter.Valid(); iter.Next() {
            item := iter.Item()
            k := item.Key()
            err := item.Value(func(v []byte) (err error) {
                ks := string(k)
                if ks == "view_total" {
                    s.totalView, _ = strconv.ParseInt(string(v), 10, 64)
                    return nil
                } else if ks == "get_total" {
                    s.totalGet, _ = strconv.ParseInt(string(v), 10, 64)
                    return nil
                }

                if strings.HasPrefix(ks, "view_") {
                    importPath := strings.TrimPrefix(ks, "view_")
                    pkgView, _ := strconv.ParseInt(string(v), 10, 64)
                    s.pkgsView[importPath] = &pkgView
                    return nil
                }

                if strings.HasPrefix(ks, "get_") {
                    importPath := strings.TrimPrefix(ks, "get_")
                    pkgGet, _ := strconv.ParseInt(string(v), 10, 64)
                    s.pkgsGet[importPath] = &pkgGet
                    return nil
                }

                return nil
            })
            if err != nil {
                return err
            }
        }
        return nil
    })
}

What did you expect to see?

Low memory consumption like 20-40 MB

What did you see instead?

Over 400MB+

---

Please let me know if I could help in any means.

[jarifibrahim]

@unknwon Badger uses a 1 GB cache by default to keep blocks in memory. The ristretto thing is because of the cache. You can reduce the cache size by setting maxCacheSize option. If you don't need a cache, I suggest you set the maxCacheSize to 10 (you can't have a value less than 10).

[unknwon]

Thanks @jarifibrahim! It's dropped to 88MB which is more acceptable for my use case :)