Orleans: Indexing grains

[philbe]

We just published an indexing mechanism for Orleans in OrleansContrib, here. You can read about it in our research paper and associated slides. This issue is to provide a forum for discussing the feature.

[richorama]

In this issue @mdashti suggests the changes required in Orleans to support indexing are relatively modest.

Perhaps we could cherry pick these commits, and then deliver indexing as an additional package?

[sergeybykov]

I don't believe it's that simple. I think we need to refactor the code to move away from inheritance (IndexableGrain), probably to facets.

[galvesribeiro]

That is what we were discussing on that issue, @sergeybykov

[lmagyar]

I have 2 quick questions (I can't build the repo, ~50 errors, I'm just browsing the source):

• I see only GetActiveGrains method but no GetGrains - Should we use GetIndex??? As I see QueryIndexedGrainsNode is not used at all like QueryActiveGrainsNode in IndexFactory.
• Is it supported/planned to change the indexes? I mean changing the index attributes or adding new properties to be indexed or adding indexing to already existing grains?

[mdashti]

@lmagyar Can you please report the build issues and your system spec (including OS and VS version)? There shouldn't be any build issues.

I see only GetActiveGrains method but no GetGrains - Should we use GetIndex??? As I see QueryIndexedGrainsNode is not used at all like QueryActiveGrainsNode in IndexFactory.

That's right. The query functionality is not implemented for non-active grains, yet. You can use GetIndex for now.

Is it supported/planned to change the indexes? I mean changing the index attributes or adding new properties to be indexed or adding indexing to already existing grains?

For active grains, it's already implemented. You can add/remove indexes and it will work as soon as you ship the next version of your code. However, currently, for the initialized indexes (i.e., I-Indexes), it requires app developer to load and save all the initialized grains once for indexes to refresh. Adding the support for automatically applying the changes in I-Indexes is on the TODO list.

[KSemenenko]

@philbe @sergeybykov Is there any news? when will it https://github.com/OrleansContrib/OrleansV2.Fork.Indexing be available?

[sergeybykov]

@KSemenenko See latest update by @philbe in #5281. We don't have a concrete timeline commitment yet, as we are juggling other priorities. In the meantime, any help with what @philbe and @TedHartMS are doing would very much appreciated.

[philbe]

@KSemenenko Sorry for the delay. We thought we'd have the new API available as a PR by now, but ran into a problem. It's very close. Hopefully, @TedHartMS will be able to post it next week.

[TedHartMS]

@KSemenenko @Arshia001 @jamiemitchellconsultants The Indexing Facet PR is posted. Documentation is at src/Orleans.Indexing/IndexingFacet.md; let me know if you have any questions or feedback. Thanks

[KSemenenko]

@TedHartMS looks good

[KSemenenko]

@Arshia001 @jamiemitchellconsultants What do you think?

[Arshia001]

@KSemenenko looks good to me.

[KSemenenko]

@philbe @TedHartMS @sergeybykov What will we do next?

[sergeybykov]

We started reviewing the changes on Orleans runtime that were made in this fork. The ultimate goal is to bring them to the core codebase, so that indexing could become a clean add-on with only NuGet level dependency on the core. The initial impression is that some of those changes will need to be refactored or maybe even redesigned.

[KSemenenko]

Any updates?

[andreujuanc]

This is amazing.
Will love to have it ready for production.
Is this the "current" fork? https://github.com/OrleansContrib/OrleansV2.Fork.Indexing

[sergeybykov]

@KSemenenko After a more thorough examination, the changes to the core runtime are substantial and sometimes controversial. Because they are unlikely to get merged to the main repo quickly. We do have a branch where the runtime commits are cleanly separated from the indexing functionality. We are reviewing it to see is we should force-push it to master of https://github.com/OrleansContrib/OrleansV2.Fork.Indexing to have a straigh line commit history with two sets of changes.

@andreujuanc Yes, this is the current version of indexing.

[KSemenenko]

We look forward to this feature 👍

[KSemenenko]

Any updates?

[sergeybykov]

We published last week to https://github.com/OrleansContrib/Orleans.Indexing a version of indexing that is an add-on to Orleans, and not a fork anymore. The runtime changes that were needed for that are in Orleans master and in recent nightly builds on https://dotnet.myget.org/gallery/orleans-ci. They will be included in 3.0.0-beta1 soon.

[KSemenenko]

It's is great news!

[RehanSaeed]

Congrats on all the hard work paying off!

[KSemenenko]

Any news/updates? We are still looking forward to this feature.

[Arshia001]

@KSemenenko funny, I was thinking about asking the same thing yesterday.

[sergeybykov]

At the moment, we are not spending any time on https://github.com/OrleansContrib/Orleans.Indexing (due to other priorities). We are open to contributions.

[dannykay1710]

Any news on this?

[sergeybykov]

No news.