Orleans: Question: Is there a stream example?

[w14525555]

Hi,

I am working on my master thesis about the fault tolerance techniques in Actor Based Stream Processing Frameworks. I have read through the documentation about stream and before starting programming, a code sample(even a hello world program) about stream can definitely helps me a lot. The only example I can find is the quick start example in the documentation.

I am new to stream processing so I hope to get some help here. Is there any code sample about stream?

Thanks a lot!

[mehmetakbulut]

@lmagyar 's presentation deck (here) has a detailed section on streams with examples. Take a look at slides 76-106.

If you want to see some functional usage, the unit tests could also be good to delve in.

[w14525555]

@mehmetakbulut Thanks for your suggestion!

[xiazen]

if we are to add a streaming sample. what kind of sample will be good for new streaming users in your view? persistent stream or sms should be demonstrated ?

[w14525555]

@xiazen In my opinion, a fairly simple stream project with Simple Message Stream Provider is a good start point for new streaming users. In fact, Azure Queue Stream provider intrigues me a lot and it will be fantastic if there is a example with AQS which illustrates some advanced features.

[xiazen]

I recently upgraded one of @centur 's sample use orleans 2.0. It is a chat room app where the client can join, leave the chat room and also post messages to the chat room. The sample I touched is in https://github.com/xiazen/altnet-streams-demo/tree/update-to-2.0 . Feel free to take a look to see if that's a good candidate for SMS streaming sample. @centur if we like your sample, can we revise it and put it in our sample folder? no copy-right violation intended :)

For AQ stream, the difference between it and SMS is, when the producer produce to a stream, it doesn't deliver to the consumers directly, but instead send the messages to a azure queue. On the producer side, AQ Stream Provider enqueues events directly into Azure Queue. On the consumer side, AQ Stream Provider manages a set of pulling agents that pull events from a set of Azure Queues and deliver them to application code that consumes them. So it has advanced feature like queue balancing to balance out message delivering process across the cluster. It is also more reliable , since messages are now persisted in azure queue, and also there's retry logic built in if message failed to deliver to consumers. it is hard to show those advanced feature in a sample, since the public interface isn't very different from SMS. Maybe some docs and code samples would be more helpful? since docs can be very explicit on what these advanced features are ? And what is AQS pros and cons?

[centur]

@xiazen yes, I'm easy - it was a demo for alt.net presentation and in general it's quite basic - if it's what you need - use it, there is no copyright attached ( I can add a particular license to that repo if it's needed).

[w14525555]

@xiazen @centur Thanks! The chatroom should be an appropriate example for SMS streaming(Maybe need some tweaks).
Besides, one question about this sample: Instead of using _subscribe_ and _unsubscribe_, _LeaveChannel_ and _JoinChannel_ are used in the chatroom. In the documentation, we are recommended to use the observer and observable to send and receive messages. Could you please explain why ?

Thanks a lot for explaining the differences and the details of AQ stream. I agree with the idea of some docs and code samples, which should be more helpful than merely a sample. Maybe you can focus on questions like why the AQ stream is more reliable(Message Delivery Guarantees and failure recovery)? how it works? pros and cons?(as you said)

[centur]

@w14525555 Umm, I'm not sure I understand the question. JoinChannel and LeaveChannel are just method names, that can be invoked from external caller to request grain to "join chatroom" (in a form of "subscribing to stream") or leave chatroom (in a form of "unsubscribing from stream"). You can call it in any meaningful way you want, it's your interface and you know better the context of using it.

"Observer" and "Observable", in my opinion, are common conceptual models that are familiar to many developers. In other words it's a "shared glossary" to explain how code is functioning, not how your code and methods must be named.
It's the same as if anyone talks about "Object Factory" - it's a known object creation pattern but this doesn't mean that you must name everything ObjectFactory, in your code you will have "ConnectionFactory" or "ComInstanceFactory" or "ExternalServiceFactory" and you preserve only "Factory" suffix, but it's just because naming something "...-Factory" helps your fellow developers to understand your code faster, by bringing correct mental models.

If there is a link to docs which says - it must be named that way - can you please link it here - keen to read and understand the context of it.

[w14525555]

@centur Thanks for your explanation! Sorry for not making the question clear enough. In fact, I didn't worry about names. I thought we should implement interface like _IGrainObserver_ to receive messages. Also, we can use class which implements _ObserverSubscriptionManager_ to subscribe and unsubscribe observers. And use _notify_ to send messages. You can look at this page: ObserverSubscriptionManager

[w14525555]

@xiazen I also find another provider EventHubProvider, which seems even powerful than the AQ steam. The EventHub also support checkpoint but AQ stream does not has implementation of check pointer(right?). Therefore, maybe adding docs and code samples for EventHubProvider is a better choice? It is even not mentioned in the document here.

[xiazen]

Yes you are right, only EventHub supports check pointer now. We are updating our docs now to reflect Orleans 2.0 latest state. @jason-bragg is updating the streaming docs. I will make him explain the pros and cons and other nuance of different stream provider in Orleans . Thanks for the suggestion, stay tuned for the new docs .

[jason-bragg]

Current doc update work involves just getting the docs accurate to 2.0. Not much new will be added in this pass. As far as stream provider behaviors, we do need more documentation in that area, but if you need information that is not in the docs, issues with targeted questions would get you answers faster and would be useful for a future streaming FAQ.

Fast, recoverable, distributed stream processing is a very complex subject and is not a solved problem. The docs will most likely only ever cover the subject in a surface manner.

Specific to fault tolerant behaviors, I'd suggest taking some time to familiarize yourself with ImplicitSubscritionRecoverableStreamTestRunner and the related tests that use it. This is a general test harness we use to test the resilience and recoverability of Orleans persistent streams. Many of the faults injected into the stream processing grains during those tests cover cases derived directly from bugs that have manifested in services using high throughput persistent streams which must be resilient to transient errors.

[w14525555]

@jason-bragg Make sense. Since I have read a lot of code, the docs are not urgent or prerequisite for me. Just hope to get more stream code samples.

Thanks for your suggestion on the specific unit tests runner. There are plenty of Unit Tests and it is time-consuming to navigate.

[xiazen]

I submitted a pr to add a simple chatroom app in #4388 . This sample is based on the sample writted by @centur, but revised with changes.

[xiazen]

close this since we added a chatroom sample , and also updated the streaming docs