Badger: Confusion regarding RunValueLogGC

[jackmiller334]

Hi guys. I'm running badger version 1.5.3 and I'm hitting some confusion about how the garbage collection works with badger.

The aim of the following test code is to try and test the garbage collection.

To summarise what it does:

• Creates and deletes a bunch of key/value stores in Badger.
• Checks to see if there are any keys stored after insertion and removal
• Produce a number of vlog files at 10mb each. (From my understanding, garbage collection only works when there are more than 1 vlog files)
• Run garbage collection (I expect that vlog files are removed at this point)

package main

import (
    "crypto/rand"
    "fmt"

    "github.com/dgraph-io/badger"
)

func main() {
    directory := "./db"

    bopts := badger.DefaultOptions
    bopts.Dir = directory
    bopts.ValueDir = directory
    bopts.ValueLogFileSize = 10485760 // 10 mb
    db, err := badger.Open(bopts)
    if err != nil {
        panic(err)
    }
    defer db.Close()

    k := make([]byte, 10)
    v := make([]byte, 50000)
    if _, err := rand.Read(k); err != nil {
        panic(err)
    }
    for i := 0; i < 1000; i++ {
        err := db.Update(func(txn *badger.Txn) error {
            return txn.Set(k, v)
        })
        if err != nil {
            panic(err)
        }
        err = db.Update(func(txn *badger.Txn) error {
            return txn.Delete(k)
        })
        if err != nil {
            panic(err)
        }
    }
    err = db.View(func(txn *badger.Txn) error {
        opts := badger.DefaultIteratorOptions
        opts.PrefetchSize = 10
        it := txn.NewIterator(opts)
        defer it.Close()
        for it.Rewind(); it.Valid(); it.Next() {
            item := it.Item()
            k := item.Key()
            v, err = item.Value()
            if err != nil {
                return err
            }
            fmt.Printf("key=%s, value=%s\n", k, v)
        }
        return nil
    })

    if err := db.RunValueLogGC(0.1); err == nil {
        fmt.Println("success")
    } else {
        fmt.Println(err)
    }
}

Here is the output from running the above code:

$ go run main.go
Value log GC attempt didn't result in any cleanup

After inspecting my db directory, I can see the following files

   10M 28 Jan 21:47 000000.vlog
   10M 28 Jan 21:47 000001.vlog
  186B 28 Jan 21:47 000002.sst
   10M 28 Jan 21:47 000002.vlog
   10M 28 Jan 21:47 000003.vlog
  7.7M 28 Jan 21:47 000004.vlog
   48B 28 Jan 21:47 MANIFEST

I expected these files to either be considerably smaller or removed. I was hoping that I could get some insight as to why this is happening and if what I thought would occur is even to be expected at all.

Thanks for your time!

[manishrjain]

There's only one SST file. The values would only be GCed once the there have been some compactions.

[jackmiller334]

Thanks for the reply @manishrjain. May I ask what the SST files are for and are they required at all for garbage collection?

Thanks

[manishrjain]

SST files constitute the LSM tree. Only when keys are discarded or become invalid in the SST files, are they removed from the value logs.

[jackmiller334]

Sorry to re-open this, but we're having issues with vlogs growing to max size and never being cleaned up, even when there are no keys in badger.

Our application is a long running web server with badger as acting as our database, with lots of concurrent writes and reads to and from badger.

Many items have a TTL. The core issue is that badger isn't cleaning up any vlog files containing values which have expired due to the TTL.

I've lowered maxTableSize in the test code above to try and force some compactions I'm still getting Value log GC attempt didn't result in any cleanup

Is there something I am missing?

Thanks

[yanchenghust]

Sorry to re-open this, but we're having issues with vlogs growing to max size and never being cleaned up, even when there are no keys in badger.

Our application is a long running web server with badger as acting as our database, with lots of concurrent writes and reads to and from badger.

Many items have a TTL. The core issue is that badger isn't cleaning up any vlog files containing values which have expired due to the TTL.

I've lowered maxTableSize in the test code above to try and force some compactions I'm still getting Value log GC attempt didn't result in any cleanup

Is there something I am missing?

Thanks

I have the same issue. Have you fixed this problem?

[sana-jawad]

u fixed this problem

I am also seeing similar issue. Any insights would be appreciated.

[jarifibrahim]

Hey @sana-jawad @yanchenghust do you guys also have keys with TTL? Maybe there's an issue with how GC handles TTL.

@sana-jawad @yanchenghust what version of badger are you using? TTLs had some issue with GC https://github.com/dgraph-io/badger/issues/974 but this is fixed in Badger v1.6 (we are yet to release v1.6.1 which has the patch) and v2.0.