Caffeine: Suggestion: Propose Caffeine to Solr 6 folks.

[guidomedina]

Hi Ben,

I have migrated one of clients caches from Guava to Caffeine (after migrated so much legacy code to Java 8, etc) and now post-updating other things like Solr 5 that will be migrated to Solr 6 which minimal requirements are Java 8, they still use Guava 14.0.1

That said, you could initiate the talks with Solr folks and convince them that Caffeine is the way to go due to three reasons, two of them major:

• Caffeine jar is smaller.
• Caffeine cache space is around half in allocation?
• Caffeine cache speed is at least 10x faster?

Also you are the right person to do that due to your Guava contribution background, etc.

[ben-manes]

Thanks for the supportive suggestion!

Please see SOLR-8241 where I supplied a patch for evaluation. My understanding is that Solr uses LRU by default and has an LFU option. There are custom multiple cache implementations and Guava's is sparingly used. The search traces I have support @elyograg's observation that frequency has a significantly higher impact than recency, so there is an opportunity to for a significant improvement to the hit rates.

It is a bit more difficult to make migration proposals as a library author. While I can show advantages, the changes don't resolve a user's problem and rightfully developers are weary of breaking something for theoretical improvements. @elyograg has been supportive of the idea and is currently using Caffeine in a project for work, hoping to gain confidence & familiarity to move that ticket forward. I think if we had real-world data that might help answer the open questions (does it improve performance? does it improve the hit rate? is the cache a current bottleneck?). For now progress depends on how much time @elyograg can steal away, which can be really difficult.

[guidomedina]

Nice !!!

I can only see good things coming out of this, my client uses lots of caches extensively and Solr to do lots of map searching, etc, millions of results and have the philosophy of: Give 31g heap sizes to JVM as the services are memory hunger due to heavy use of caches, that to tackle a design problem where distribution of services was not possible _-which I will gradually add with the aid of Akka or Vert.x-_ hence caches are essential.

Solr have been of great aid so reduction of time and space is also of a great importance.

[ben-manes]

Perhaps then you'd be interested in experimenting with the patch to quantify the impact? Its a few months old so some updates are needed and I wasn't familiar with running Solr. (Our search needs are modest and there is only 1 other engineer, so I started with Algolia). Alternatively, if we captured a cache access trace then we could run it through the simulator to compare the hit rates.

I suspect improvements to the hit rate will have a greater impact than to concurrency. It might allow for reducing the heap size, lower miss penalties (e.g. disk access), and discard low value entries sooner.

At some point I'd like to show a proof-of-concept of Caffeine as an off-heap cache using Apache Mnemonic. I think that could be an attractive option for large heaps, especially since it avoids the serialization overhead.

[guidomedina]

Patching it is possible, I only have to find the time and wait till my client moves to Solr 6.

I just had to change the default executor to a SAME_THREAD_EXECUTOR approach to emulate Guava's behavior; some applications use ThreadLocal to store I/O resources like DB connections so the cache was leaking connections.

I can expect something similar might happen with Solr in terms of I/O resources so that's something to be aware of, so you probably want to start in Solr with same thread approach, investigate further and then remove it if not needed.

Edit: I meant it for CacheLoader or _-unlikely but probable-_ RemovalListener

[ben-manes]

I/O resources can definitely be tricky. It is very tempting to use a cache as an object pool, but the resource constraints are slightly different. You'd still have a race with Guava or a same thread executor, but I guess its less pronounced under load. In both libraries the removal listener can still be called by a different thread. Guava evicts to a removal notification queue which all threads process (so the remover might not do the work). I think whatever leak you ran into was already present and may still be.

Usually one should prefer having the cached data be fully materialized and using the I/O resource during the load operation. Then the function fetches using the connection pool (db, http), does the work, releases it, and multiple clients can access the cached result concurrently.

So you might want to revisit the code and see if the assumptions are actually being held. It sounds like you might want to play with an object pool like Stormpot or Vibur.

[guidomedina]

Legacy code, DB connections are cached at ThreadLocal and released at the end of the whole process so if you use a CacheLoader inside this application, you re-use the existing connection or get a new one, but you don't release it there hence cannot load on a different thread _-unless you release there but then you would have to change so much legacy code-_

My personal preference for JDBC pool is HikariCP

[ben-manes]

Oh, the awful Session-per-request pattern Hibernate liked to promote? Shudder.

[guidomedina]

Yes, now you know my pain and why I cannot use CacheLoader on a different thread, but not all is bad, that's the legacy part including a custom DB object mapping.

But in the other hand it has been a successful business, good team, the code keeps growing and there are good patterns in it, but no time for re-write, business too big and too many things to take into account so it just keep moving, mixing the old with the new, etc.

Imagine PostgreSQL + Riak + Solr, my next step is to add micro-services and slowly move some services out of the main application.

[ben-manes]

I've done my fair share of migrating legacy systems. I've always been surprised how hostile most developers are towards services, events, etc. Too many war stories of truly awful home grown frameworks...

Right now I'm finishing a rewrite of Django/Celery to Java, as small enough to make the change before its too late. I've never had auto-clustered frameworks work out well in practice, so using a stack I've had success with in the past. That's jsonschema2pojo, JAX-RS services, jOOQ, retrofit service clients (returning CompletableFuture), Guice, and RabbitMQ (for events & work queues). I'm hoping to setup Kubernetes to use DNS for service discovery by end-of-year, but for now its calls to localhost given the priorities. Surprisingly fewer LOC than Python, while much faster and easier to develop on. I generally prefer using stacks that are easy to migrate away from, so http + dns + json is my preferred starting point. Then its easy to switch frameworks or, if scaled, revisit the protocol.

[guidomedina]

Asynchronous design in general requires some mindset and most developers indeed are hostile to it, it has always taken me some time to convince people of queues and fire-and-forget approach, you might enjoy Vert.x, take a look at it, it is very simple, easy to work with and I think it scales very well.

[ben-manes]

Closing since the request is submitted and we're waiting for @elyograg to tackle it.

[ben-manes]

After a lot of patience, our patch was picked up anew, released in Solr 8.3, and the prior implementations will be removed in 9.0.

[jbduncan]

Congratulations @ben-manes! 👏