Etcd: ETCD lock example

[sanamsarath]

Can someone provide the exact use case of lock feature in ETCD.

How can I use the lock id generated after lock request, should I have to include that id in the atomic operations like put, get or delete?? also want to know if lock applies to the entire key value store or if a single key can be locked and updates on other keys can be performed in parallel??

Thanks.

[wenjiaswe]

@jingyih

[jingyih]

Etcd lock service API is documented here.

And there is an example of using etcdctl lock command.

To answer your question, I do not think a lock id should be associated with a specific etcd key, or an etcd command. My understanding is that etcd lock is a service that can be used by users(or other systems) for protecting access to whatever resource they wanted to protect (not necessarily any resource in etcd database), something like:
1. acquire an etcd lock
2. do something (again, not necessarily related to etcd database)
3. unlock the same etcd lock

After a closer look at our current etcdctl lock command, I did not find a way to create a demo flow I mentioned above. The current behavior of etcdctl lock is that it tries to acquire the lock, blocks itself if the lock is already locked by some other user. If lock is acquired, upon hitting the Ctrl-C, it unlocks the acquired lock. I did not find a way to add operations between locking and unlocking.

@gyuho Do you have any comment on this topic?

/cc @wenjiaswe

[gyuho]

@sanamsarath @jingyih @wenjiaswe You can create a session with Go client and simulate concurrency as follows:

ss1, err := concurrency.NewSession(cli, concurrency.WithContext(ctx))
if err != nil {
    log.Fatal(err)
}
defer ss1.Close()
mu1 := concurrency.NewMutex(ss1, "/my-lock/")
fmt.Println("Lock 1")
fmt.Println(mu1.Lock(context.Background()))
fmt.Println("Lock 2")

doCancel := func() {
    fmt.Println("cancel 1")
    cancel()
    fmt.Println("cancel 2")
    mu1.Unlock(context.Background())
}

go func() {
    time.Sleep(5 * time.Second)
    doCancel()
}()

ss2, err := concurrency.NewSession(cli, concurrency.WithContext(ctx))
if err != nil {
    log.Fatal(err)
}
defer ss2.Close()
mu2 := concurrency.NewMutex(ss2, "/my-lock/")
fmt.Println("Lock 3")
fmt.Println(mu2.Lock(context.Background()))
fmt.Println("Lock 4")

Lock 1
<nil>
Lock 2
Lock 3
cancel 1
cancel 2
<nil>
Lock 4

[sanamsarath]

@jingyih @gyuho Thanks for the explanation and example.

Just want to be clear about the following, as I am trying to implement the c++ client library for etcd

mu1 := concurrency.NewMutex(ss1, "/my-lock/") - > here you are acquiring a lock.

mu1.Lock(context.Background()) -> does this method use the already acquired lock id to do some
operations on etcd server?? what does background function does??

mu1.Unlock(context.Background()) -> And this method sends the unlock request with lock id??

[sanamsarath]

@gyuho

lease generated for a lock is never expiring,

lock acquired using command line:

[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lock mylock --ttl=20
mylock/52dd65ee79d88077

lease timetolive is not terminating:

[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(19s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(16s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(14s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(19s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(18s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(16s)ive 52dd65ee79d88077

[hexfusion]

lease generated for a lock is never expiring,

@sakshamsharma etcd --version etcdctl version ?

[hexfusion]

I guess it is possible if new leader is elected during this test TTL could be reset. Can you reproduce this?

[jingyih]

maybe related to this
https://github.com/kubernetes/kubernetes/issues/65497

[sanamsarath]

@hexfusion version 3.3 and I have only 1 etcd node.

[hexfusion]

@sanamsarath 3.3.x? please output etcd --version so you're saying it is reproducible?

[sanamsarath]

@hexfusion sorry about that, i was lazy.

etcd Version: 3.3.2+git
Git SHA: 227c3e5
Go Version: go1.10.3
Go OS/Arch: linux/amd64

yes it is reproducile.

[jingyih]

@sanamsarath
I think this is the intended behavior. Since you are still holding the lock, the lease gets auto renewed. If you kill the process that is holding the lock, or if there is network partition between this machine you are on and the etcd server, the lease will expire. The API doc mentioned this behavior:

On success, it will return a unique key that exists so long as the lock is held by the caller.

The lock is held until Unlock is called on the key or the lease associate with the owner expires.

We might need to do better documentation.

[sanamsarath]

@jingyih Thanks.