Openwhisk: Use non-blocking UUID generation in controller

[vvraskin]

While doing profiling of controller we came across the fact that UUID generation is a blocking operation, e.g.

    synchronized public void nextBytes(byte[] bytes) {
        secureRandomSpi.engineNextBytes(bytes);
    }

We need to think of alternatives to the current approach considering the fact that we should enable several active controllers in the near future, e.g. both cross-thread/cross-controller unique UUIDs.

cc @mhenke1 @markusthoemmes @rabbah @cbickel

[markusthoemmes]

Very good find, there will definitly be contention going on when having multiple cores and thus largely parallel requests. Good find!

One thing to keep in mind as well is that activation ids need to be unique in the domain of one namespace only, not necessarily globally which might reduce the need for "secure" randomization.

[dgrove-oss]

One possibility could be to have a cache of precomputed UUIDs that is populated by a background thread. That should allow the latency to be hidden.

[bwmcadams]

It looks like from Java 8 onwards you can provide ?custom? SecureRandom instances which aren't blocking. Found a similar github issue discussing this.

Usefulness of this depends on whether OpenWhisk supports Java 7 of course…

But I know I've seen third party implementations of UUID that aren't blocking. There's also the possibility of doing something like MongoDB Object IDs do with a custom UUID. Mongo OIDs were designed so that in a sharded (cluster) environment OIDs could be generated without cross-cluster / instance collision by combining time, a machine identifier (IIRC the Java driver uses the mac address for it?), a PID, and a counter (often implemented as a random). Within a single set of controllers the actor ID could be used for uniqueness-ness.

[markusthoemmes]

@rabbah and I have been debating a fix for this. Providing an alternative SecureRandom instance could help (especially if it uses the non-blocking /dev/urandom vs /dev/random), but there would still be the problem of thread contention due to the synchronized nature of getBytes.

A naive implementation I did to at least get the issue out of the way for now is:

new java.util.UUID(ThreadLocalRandom.current.nextLong, ThreadLocalRandom.current.nextLong)

The randomness is not cryptographically secure, yes. We are namespacing our ActivationIds though, so the domain of collision is significantly smaller than having them globally namespaced. I'm not too much an expert in that area though to judge wether the randomness provided by ThreadLocalRandom is sufficient. It for sure does fix the contention and blocking problem though.

[markusthoemmes]

Making up my mind, using ThreadLocalRandom seems like a bad idea. How does implementing ThreadLocal[SecureRandom] sound? As our dispatcher uses a fixed number of threads, the performance hit of generating a new SecureRandom instance should be negligible.

[dgrove-oss]

Do we know if there is any synchronization around the reads from /dev/random or /dev/urandom within the guts of the SecureRandom implementation? If there is, we might not buy that much from making ThreadLocal instances.

[markusthoemmes]

@dgrove-oss fair point, here are a few profiling results. I think I arrived at a solution for us:

Testcode

def main(args: Array[String]): Unit = {
  (1 to 100000000).par.foreach { _ =>
    val uuid = randomUUID()
  }
}

Parallel generation of 100000000 random UUIDs.

Baseline: java.util.UUID.randomUUID()

(It goes on until about 2:50 minutes)

Switching from /dev/random (blocking) to /dev/urandom (non-blocking)

Much better already, but still a lot of thread contention.

Switching to ThreadLocal[SecureRandom]

There we go!

So it looks like the innerts of SecureRandom do not contain any more contention which is great.

[62mkv]

Hi @markusthoemmes may I ask which tool/mode did you use to produce such pretty thread pictures?