Openwhisk: Round-robin HA controller can lead to concurrent modications errors

[devbv]

Since Nginx does proxy in Round-Robin way, some tests which were designed to run sequentially in one controller are failed.

Let me explain one example, travis of latest commit(0737826c06c73a904c49977910d103d1eed79174) failed the test "Wsk Package should allow updating a package". Because the result of package update was 409, not 200

See the code below

it should "allow updating a package" in withAssetCleaner(wskprops) { (wp, assetHelper) =>
    val name = "simplepackagetoupdate"
    assetHelper.withCleaner(wsk.pkg, name) { (pkg, _) =>
      pkg.create(name, params1)
      pkg.create(name, params2, update = true)
    }
  }

It might runs in this way

1. delete pkg when withCleaner starts -> controller0 delete package if it exists
2. pkg.create(name, params1) -> controller1 put package with rev A
3. pkg.create(name, params1) -> controller0 put package with rev B
4. delete pkg when withCleaner ends -> controller1 del package with rev A

Package is finally stored with rev B, but controller1 try to delete it with rev A, and then CouchDB response last operation as a conflict(409)
It could be success, if 'cacheInvalidation' msg arrived between 3 & 4. But we can't guarantee that.

I found that many tests were failed by similar scenario while running Travis.

Any good idea?

tarvis log
controller0_logs.log
controller1_logs.log

[rabbah]

Thanks @devbv for the cross link - I think having a sticky session to route CRUD operations to the same controller will help alleviate this issue vs round robin spraying the operations between controllers. cc @cbickel

May classify this or #3021 as duplicates.
Also related to https://github.com/apache/incubator-openwhisk/issues/1815.

[Ahmed-elsayed-mahmoud]

@rabbah @devbv Does round-robin handle load balancing between controllers and requests passing to them? Could you tell me where round-robin is implemented?
Thanks!

[rabbah]

It’s done at the router (nginx).

[Ahmed-elsayed-mahmoud]

which class ? in controller module?

[rabbah]

See https://github.com/apache/incubator-openwhisk/pull/3266 for the nginx change.

[rabbah]

Other solution suggested in https://github.com/apache/incubator-openwhisk/issues/1815#issuecomment-277844752