Orleans: [Question/Proposal] Are there LINQ bindings for Orleans

[AesaKamar]

I really love Apache Spark's concept of an RDD. It would be nice to have a set of LINQ extensions which allow you to similarly perform distributed operations on collections.

Is something like this implemented?
If not, how can I get started contributing.

[sergeybykov]

In general, one can easily orchestrate a distributed computation with Orleans by partitioning it or its stages across a pool of grains. There's no built-in integration with LINQ. If you could elaborate on the desired support for it, that would help evaluate feasibility of such integration.

[galvesribeiro]

@AesaKamar do you want distributed queries on grains with aggregated results (i.e. some sort of map-reduce) ?

[AesaKamar]

Let me know if this example makes clear the intention:

I have a collection of objects, say they're of type _Fruit_
[Apple, Orange, Pineapple, Pear]

I would like to map the operation _Fruit.ExtractJuice()_ on each of the objects in the collection, so I end up with a new collection of _FruitJuice_ objects.
It would be nice to have some sort of generic collection that could execute the function.

[Apple, Orange, Pineapple, Pear].MapOrleans(x => x.ExtractJuice() )
And yield the result,
[AppleJuice, OrangeJuice, PineappleJuice, PearJuice]

Thanks

[AesaKamar]

@galvesribeiro Yes. Essentially distributed operations on collections.

[galvesribeiro]

Being very simplistic, you could have a collection of Grains, when each grain being a fruit on this case then you can fan out execution of one method on that grain type... Something like this:

var extractJuiceTasks = new List<Task<Juice>>();
extractJuiceTasks.Add(GrainFactory.GetGrain<IFruitGrain>("Apple").ExtractJuice());
extractJuiceTasks.Add(GrainFactory.GetGrain<IFruitGrain>("Orange").ExtractJuice());
extractJuiceTasks.Add(GrainFactory.GetGrain<IFruitGrain>("Pineapple").ExtractJuice());
extractJuiceTasks.Add(GrainFactory.GetGrain<IFruitGrain>("Pear").ExtractJuice());

await Task.WhenAll(extractJuiceTasks );

That would fanout the execution of extractJuiceTasks to multiple silos and you will get back the Task<T> after all are completed. If you have enough silos or enough cores they will run theoretically in parallel and after it complete, you can just get the results from all tasks.

That is not the best solution for collection operation but that would work for your case.

We don't have today a "distributed collection" in Orleans but IIRC, someone was working on that and @sergeybykov can confirm that...

[sergeybykov]

Of course, if you already have a collection of fruit grain, e.g. List<IFruitGrain>, then you can iterate over it with a foreach or another loop construct and a make ExtractJuice() call on each of them. This can be further generalized with helper/extension methods.

The problem with LINQ though is that it doesn't integrate nicely with async operations returning Tasks.

[dVakulen]

https://github.com/OrleansContrib/Orleans.Containers - "Distributed storage and processing of objects in Orleans".

[AesaKamar]

@dVakulen That's extraordinarily useful, thanks!

And @sergeybykov, thanks for entertaining my questions.

[galvesribeiro]

Wow! The name is really misleading @dVakulen I never saw that! Looks nice! I wonder why we don`t bring it in.

[Liversage]

Perhaps I'm stating the obvious here but as "LINQ" is in the title you can do what @galvesribeiro suggests in a "LINQ way" like this:

C# // The input that you get from somewhere else. var fruits = new[] { "Apple", "Orange", "Pineapple", "Pear" }; var extractJuiceTasks = fruits .Select(fruit => GrainFactory.GetGrain<IFruitGrain>(fruit).ExtractJuice()); await Tasks.WhenAll(extractJuiceTasks);

Normally, when you talk about "LINQ bindings" it means using IQueryable<T> with expressions trees (not lambdas) to transfer your query from C# code to whatever back-end you are using (like a database). I don't see how that can be applied to Orleans.

[hiyelbaz]

IQueryable -> odata string -> IQueryable ?

[sergeybykov]

Seems like there's nothing to do here. Closing it.