Badger: windows LOCK exists, program can't restart

[zjb0807]

In the windows environment, if the program is killed and the LOCK file is not deleted, the restart program will be wrongly reported

Cannot create pid lock file "C:\\Users\\User\\Desktop\\chain.exe\\datadir\\LOCK".  Another process is using this Badger database: open C:\Users\User\Desktop\chain.exe\datadir\LOCK: The file exists.

[manishrjain]

You can manually delete the lock file, and restart. Let me know if that doesn't for some reason. Or, if you need some other resolution.

[manishrjain]

If you can point us to how LevelDB is doing this, or others are doing this -- we can implement it. But, not sure what we can change now...

Also, if you can tell us the steps that you took to reproduce this, that'd be great.

[zjb0807]

leveldb used syscall.CreateFile to create LOCK file，badger used acquireDirectoryLock,but the code like this:
f, err := os.OpenFile(absLockFilePath, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0666)
os.O_EXCL flag caused panic, Shall we delete it？

[manishrjain]

       O_EXCL Ensure that this call creates  the  file:
              if  this flag is specified in conjunction
              with  O_CREAT,   and   pathname   already
              exists,  then open() fails with the error
              EEXIST.

We need this flag to ensure that this process is the only one operating in the directory. This flag provides exclusivity.

[zjb0807]

package main

import (
    "log"
    "time"

    "github.com/dgraph-io/badger"
)

func main() {
    opts := badger.DefaultOptions
    opts.Dir = "./badger"
    opts.ValueDir = "./badger"
    db, err := badger.Open(opts)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    for i := 1; i < 1000; i++ {
        time.Sleep(1 * time.Second)
        log.Println(i)
    }
}

then i start the program, in another terminal, i start the program again:

[chain33@localhost chain33]$ ./main 
2018/04/20 03:14:02 Cannot acquire directory lock on "./badger".  Another process is using this Badger database.: resource temporarily unavailable

then, i have deleted the LOCK file, try again:

[chain33@localhost chain33]$ ./main 
2018/04/20 03:16:13 Cannot acquire directory lock on "./badger".  Another process is using this Badger database.: resource temporarily unavailable

the unix.Flock function lock the directory, so it is normal to restart when the program is panic and LOCK file exists on linux.

But, i have tested goleveldb:

package main

import (
    "log"
    "time"

    "github.com/syndtr/goleveldb/leveldb"
)

func main() {
    db, err := leveldb.OpenFile("./leveldb", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    for i := 1; i < 1000; i++ {
        time.Sleep(1 * time.Second)
        log.Println(i)
    }
}

then i start the program, in another terminal, i start the program again:

[chain33@localhost chain33]$ ./level
2018/04/20 06:10:33 resource temporarily unavailable

then, i have deleted the LOCK file, try again:

[chain33@localhost chain33]$ rm leveldb/LOCK
[chain33@localhost chain33]$ ./level
2018/04/20 06:11:36 1
2018/04/20 06:11:37 2
2018/04/20 06:11:38 3

so, goleveldb used LOCK file to ensure that this process is the only one operating.

but on windows system, when i delete LOCK file, there is an error for file is busy. So we can used goleveldb newFileLock instead of badger acquireDirectoryLock just dir_windows.go.

[CapacitorSet]

This is causing issues in another software that uses Badger. @manishrjain Would you accept a PR to fix this?

[manishrjain]

Sure. Will have a look at it over the next week.

[nordinvannes]

Any follow up on this issue? Thanks in advance!

[manishrjain]

If someone wants to help, ping me via Slack or Discuss.dgraph.io. I'd like to fix this issue, but I'm not a Windows guy.

[CapacitorSet]

@manishrjain did you review #470?

[manishrjain]

Hey @CapacitorSet , Thanks for the PR, first of all. The main issue I have with it is that it brings in a new dependency. Ideally, I'd like to see if we can just fix up the locking system directly in our own code base.

Could you potentially look at how others, like BoltDB, solve this problem? And then rationalize what Badger is doing wrong?

[CapacitorSet]

I'll ping you on Slack.

[PaluMacil]

This is pretty difficult to work with, since on Windows I get "Value log truncate required to run DB. This might result in data loss." after deleting the lock file. I don't understand what I can do at the moment short of deleting the database if I get a panic and can't close the database on exit. Is this still being discussed? It seems like this is pretty severe, but I don't have any relevant experience to assist, sadly.

[manishrjain]

The way mmap works on Windows, you'd always have this, if the process crashed. For windows, you can set the Truncate flag to true always. Or, not use mmap for value log or for LSM tree, that can be set via Options.

[CapacitorSet]

If you want to contribute and have a Windows machine/VM you could test #470. It's not fundamental, but additional feedback is welcome.

[PaluMacil]

Both of those topics (mmap and truncate) don't appear to be discussed in the readme, so I'm not sure what either mean I would be doing, but I'd love to test #470 and maybe make a PR for docs when I learn more (I'd need to get employer permission to sign the CLA, but they have been pretty accommodating of my requests for permission so far).

[baumgarte]

just tried dgraph and ran into the same problem - do you have suggestions how to terminate dgraph processes friendly?

[PaluMacil]

@baumgarte Since my visit to this thread, I've added a couple things to help this. I'm just scribbling out a personal website in my free time and work in C# by day, but the ideas might help.

Make sure you close your database. I do that in my automated tests even though I also delete the database afterwards (shrug). If you close the application with an interrupt (ctrl-c or a signal from another process like a supervisor) you will need to handle that in order to get to your database close call. If you catch the interrupt signal, you can close the database gracefully so that there isn't unwritten data to truncate (lose).

    stop := make(chan os.Signal)
    signal.Notify(stop, os.Interrupt)
    go func() {
        log.Println("Now serving on port", appModule.Port)
        log.Println(srv.ListenAndServe())
    }()
    <-stop

    log.Printf("shutting down ...\n")

    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatal(err)
    }
    appModule.Db.Close()
    log.Printf("Badger: database stopped\n")

When I open a database, I call retry if it fails from presence of a lock file:

func New(dir string) (*Db, error) {
    registerGobs()
    opts := badger.DefaultOptions
    opts.Dir = dir
    opts.ValueDir = dir
    bgr, err := badger.Open(opts)
    if err != nil {
        if strings.Contains(err.Error(), "LOCK") {
            log.Println("database locked, probably due to improper shutdown")
            if db, err := retry(dir, opts); err == nil {
                log.Println("database unlocked, value log truncated")
                db.WithProviders()
                return db, nil
            }
            log.Println("could not unlock database:", err)

        }
        return nil, err
    }
    database := &Db{bgr: bgr}
    database.WithProviders()
    return database, nil
}

Retry first deletes the LOCK and then truncates the data to only the last good written:

func retry(dir string, originalOpts badger.Options) (*Db, error) {
    lockPath := filepath.Join(dir, "LOCK")
    if err := os.Remove(lockPath); err != nil {
        return nil, fmt.Errorf(`removing "LOCK": %s`, err)
    }
    retryOpts := originalOpts
    retryOpts.Truncate = true
    bgr, err := badger.Open(retryOpts)
    return &Db{bgr: bgr}, err
}

It's a new project that supports only login via Google oauth and will sport an account page and such later this weekend, but if you want to look, it's at https://github.com/PaluMacil/dwn