Cosmos-sdk: ERROR: resource temporarily unavailable for all keys operations using CLI after REST server is started

[kwunyeung]

After starting REST server with gaiacli advanced rest-server, all operations with gaiacli keys will give this error.

ERROR: resource temporarily unavailable

Stop the REST server and those operations work using CLI again. This doesn't happen for other modules.

[ebuchman]

Thanks for reporting.

Do you have a real use case for needing this? If you're running the REST server already, can't you use those endpoints?

I think we might just not support this for now unless there's a very compelling reason to.

[ebuchman]

At least we should change the error message to indicate what the problem is. We can for sure do that :)

[davekaj]

I was getting the exact same error when implementing a go levelDB. leveldb will return "resource temporarily unavailable" if you have two routines trying to access the db. If this is the error it can be fixed by inserting db.Close() at the end of the functions accessing the db.

not sure if the REST server is even using leveldb , but I thought I'd add it since I saw the exact same error

[fkbenjamin]

Can confirm that this error still exists and that while running the REST-server it will return:

ERROR: resource temporarily unavailable

However the problem is, that the REST endpoint doesn't seem to work either. When using it like /keys/create?name=cosmos&password=cosmos
it will return

Key create not found

So you can't use the REST endpoints to create keys while running the REST server. It also seems that all (or at least most) REST endpoints that use POST are not working correctly.

[alexanderbez]

@fkbenjamin these are all the key endpoints I'm aware of (I'm also aware some of the docs are invalid/outdated):

func RegisterRoutes(r *mux.Router) {
    r.HandleFunc("/keys", QueryKeysRequestHandler).Methods("GET")
    r.HandleFunc("/keys", AddNewKeyRequestHandler).Methods("POST")
    r.HandleFunc("/keys/seed", SeedRequestHandler).Methods("GET")
    r.HandleFunc("/keys/{name}", GetKeyRequestHandler).Methods("GET")
    r.HandleFunc("/keys/{name}", UpdateKeyRequestHandler).Methods("PUT")
    r.HandleFunc("/keys/{name}", DeleteKeyRequestHandler).Methods("DELETE")
}

I was successfully able to create a key on the latest develop (POST @ /keys)

[fkbenjamin]

@alexanderbez Thx for that info! Was able to create a key using your info...

I was looking at the docs for
v0.24.0 (here) and develop (here). Both show wrong information.

[cwgoes]

This behavior is currently expected, since the open database is locked - if this is a problem for clients, I guess we can close the DB after every operation, the speed penalty doesn't matter much.

[honeycrypto]

The same error occurs if we try to run 2 parallel commands in CLI. I have few bash scripts for monitoring and they conflicts with each other due to "ERROR: resource temporarily unavailable"

[ackratos]

This behavior is currently expected, since the open database is locked - if this is a problem for clients, I guess we can close the DB after every operation, the speed penalty doesn't matter much.

IMHO, we can expose the file opening option for level db in tendermint/libs/db/go_level_db.go

func NewGoLevelDB(name string, dir string) (*GoLevelDB, error) {
    dbPath := filepath.Join(dir, name+".db")
    db, err := leveldb.OpenFile(dbPath, nil)
    if err != nil {
        return nil, err
    }
    database := &GoLevelDB{
        db: db,
    }
    return database, nil
}

so that the caller in cosmos can pass Options.ReadOnly = true in for some subcommand that only read key db (i.e. if we only want start gaiacli to put transactions rather than add new keys to generate high volumn of transactions as pressure test):

// initialize a keybase based on the configuration
func GetKeyBaseFromDir(rootDir string) (keys.Keybase, error) {
    if keybase == nil {
        db, err := dbm.NewGoLevelDB(KeyDBName, filepath.Join(rootDir, "keys"))
        if err != nil {
            return nil, err
        }
        keybase = client.GetKeyBase(db)
    }
    return keybase, nil
}

@kwunyeung
Our workaround to this is set "--home" to different gaiacli home dir for each gaiacli process we run.

But this still blocks us pressure testing in large scale. Do you have any suggestions or can point doc to me how tendermint / cosmos do the performance test to see the highest transaction per seconds can be achieved? Thanks a lot!

[jackzampolin]

@ackratos This is a good fix! Would you feel comfortable putting up a PR with a fix?

[ackratos]

@ackratos This is a good fix! Would you feel comfortable putting up a PR with a fix?

Thanks, I'd like to raise a PR for fixing today

[jackzampolin]

@ackratos Did you get a chance to write that code 🙏 Would love to get this merged and fix this issue!

[ackratos]

@ackratos Did you get a chance to write that code 🙏 Would love to get this merged and fix this issue!

blocked a bit yesterday for daily job... Will working on it now, sorry

[jackzampolin]

@ackratos Totally know how that goes! Thank you for contributing!