Hazelcast: IMap could support "faster" bulk operations setAll, putAllAsync and setAllAsync

[zeratul021]

Feature request from a SO question http://stackoverflow.com/q/30344207/303559.

It would be nice for IMap to support "fast" bulk operation like setAll() and a even "faster" non-blocking putAllAsync() or even setAllAsync() bulk operations.

The former would be beneficial for use-cases where user needs to insert/update lot of generated/loaded data and is not interested in previous values.
The asynchronous methods would help in case user doesn't want to wait for the operation to finish because the cluster will have time to synchronize.

[bwzhang2011]

+1

[pedrotiagopontes]

+1

[nick7ikin]

+1
setAll() is really needed

[mmedenjak]

putAllAsync is now available as private API and we might promote it to public API in a future release - https://github.com/hazelcast/hazelcast/issues/16449. Promoting this into a public API method is a simple PR so feel free to send it.

[cangencer]

how would setAll be different than putAll? putAll already returns void. Am I missing something?

[mmedenjak]

Interesting, I was expecting putAll to return a map with the old values. Seems setAll and putAll are the same.

[TomaszGaweda]

@mmedenjak JavaDoc for putAll states that MapLoader will be invoked. So despite void return value, it still does read-through, while setAll would not do read-through

[pertsodian]

I can give this a go to implement setAll and setAllAsync

[mmedenjak]

@pertsodian seems it's not needed. We have already exposed com.hazelcast.map.IMap#putAllAsync and it has no return value so it's effectively equal to setAllAsync. As for setAll, the user can simply block on the returned future to simulate that method.

Closing this issue as the newly exposed method covers all use cases. Feel free to comment if you disagree and if you would like to have different flavours of methods.

[pertsodian]

@mmedenjak com.hazelcast.map.impl.operation.PutAllOperation#putToRecordStore will call recordStore#put if there are any map listener registered. This is to load the previous value to honor the contract of EntryListener in which EntryEvent#oldValue is the existing value.

setAll and setAllSync, if implemented, should NOT trigger MapLoader to fetch previous values at all. And like set, they will break contract of EntryListener
I think these are still useful. Let me know what you think.

[mmedenjak]

Hm, yes, interesting, reopening. Should we still invoke MapStore#storeAll in case IMap.setAllAsync is called? I would prefer that to keep the map store in-sync with the cache but if IMap.set already doesn't call the map store, I'd prefer to align it with that one.

In any case, yes, I think we're going to need one more PR, probably simple, but with clear javadoc defining the difference.

[pertsodian]

Yes, set will invoke MapStore#store to update the underlying map store. And setAll / setAllSync should behave the same way. However, they will not trigger MapStore#load if the key is not in-memory. This will results in EntryEvent with null oldValue (as if it is added) even though the key is available in map store, as opposed to put methods.

I will give this a go. But likely will have to change hazelcast-client-protocol to add MapSetAllCodec as well. So that will be 2 PRs if I understand it correctly.

[mmedenjak]

Yes, that will be two PRs, right.

[Holmistr]

@pertsodian @mmedenjak I see the PRs are merged. Is the issue done now? BTW awesome work! @pertsodian

[RiotR1cket]

Also can someone please clarify which HZ versions this will be going into? Will it be backported to 3.12.x?

[Holmistr]

@RiotR1cket So far this is going only to 4.1+ (@mmedenjak I have just updated the milestone).

Why do you ask @RiotR1cket ? Can't you upgrade to 4.x?

[RiotR1cket]

It's a really nice optimization to the default behavior of setAll. So for apps that are still on 3.x, if this was backported, they'd get this optimization for free by just bumping the minor or patch version, instead of doing the full major version migration. It also _seems_ like a change that would be easily portable to the 3.x codeline.

[mmedenjak]

Yes, closing. Unfortunately this can only be introduced in a minor release as the change introduces serialization changes and those changes conflict with out patch-level guarantee. The guarantee says that you should be able to have a cluster which consists of members with same minor but different patch level versions (e.g. 3.12.1, 3.12.3., 3.12.5) and everything should function normally.
We allow introducing serialization changes in minor releases because these are guarded by special rolling-upgrade checks.
And finally, I don't believe any 3.x minor release is planned so the next minor release is definitely going to be 4.1.