Artillery: [Question] How to monitor messages arriving on socket.io channel?

[gboysko]

I understand how flows can be written to emit socket.io messages and wait (for some indeterminate amount of time) for a response message, but how would we write flows that can receive arbitrary (real-time) messages sent from other virtual users?

I can envision some type of monitor function that receives messages for a given flow and can verify that they are correct. Perhaps this monitoring can occur during a "think" phase.

I don't see anything in the documentation that addresses this type of real-time message validation. Is that consistent with the vision for Artillery?

[hassy]

You're right @gboysko, there's nothing like it in Artillery yet, but it's definitely something worth adding! What is your exact use case?

One way to do it would be to define a hook (like the ones in the HTTP engine) - something like onMessage at the scenario level. Is that like what you're thinking of with monitors?

[gboysko]

Yeah, that's pretty close to what I'm thinking. My use case is what you'd expect in many chat/collaboration services: new messages arrive as users interact and I'd like to ensure that the messages received are consistent with the scenario/flow. I'd love to be able to "fail" the flow if an unexpected message was received (or an expected message contained fields indicating a failure condition).

I'm not sure how we would indicate that a flow remains in a "listening" state (other than re-purposing think).

[hassy]

Cool, this is definitely a needed feature (not just for Socket.io either, but also for WebSockets and any other async protocol we want to support like MQTT or XMPP).

As the first experiment we could go ahead with something like this:

1. Support defining an onMessage function that would be invoked for every incoming message, which you could use to implement a custom counter for example. This would be completely async.

scenarios:
  - name: "A Socket.io scenario"
    onMessage: "myOnMessageHandler"
    flow:
      - emit: "something" # flow definition here

1. Add a listen command to flows. The function specified in a listen command will receive every incoming message from that point onwards until it gives control back to the flow.

scenarios:
  - name: "A Socket.io scenario"
    flow:
      - emit: "something" # flow definition here
      - listen: "myHandlerFunction"
      - emit: "somethingElse"

I can give it a go some time in the next few weeks, but it's up for grabs if someone wants to give it a go and open a PR before then.

[gboysko]

I'm new to Artillery and the code base, but may give it a try. I like the second option better as it reflects my own use case. If I did, I think it would be helpful to clearly understand the semantics.

For example, I would suspect that the invoked handler function (myHandlerFunction) can convey failures encountered, right? And that the "listening" phase of the flow would block the following emit steps from proceeding.

[hassy]

@gboysko yep, in the HTTP engine handler functions can do one of two things:

1. Emit an error event which will log the error (in the shell output / JSON log) but won't abort the scenario.
2. Invoke the callback with an error object - that will log the error and abort the scenario.

As to whether the emits following a listen would be blocked until the handler function returns - I think that both makes intuitive sense _and_ I am not sure how the opposite could be expressed cleanly in a flow spec. :)

[hassy]

@gboysko any luck? (if you've had the time) feel free to open a WIP PR, I'm happy to help out!

[gboysko]

Thanks, @hassy. I unfortunately, have not yet had the time. I am perhaps able to work on a different enhancement request (if you deem it worthy of pursuing [#171]). For me, this issue (#170) is more of a longer term concern, whereas #171 is somewhat nearer term.

[hassy]

@gboysko yes, #171 is certainly something very useful!

[hassy]

For when we're implementing this feature - we now have the means to attach a handler to any incoming message in Socket.io now:

https://github.com/shoreditch-ops/artillery-core/pull/192/commits/ed21d66a87f736e49ec7543a1f571fe744a5936a#diff-4c8bbeda468bf6bfdd048bd21ee2d037R248

[alvatech]

@hassy @gboysko Is anyone working on this feature? I need this feature but I'm new to the Artillery code base. Any pointer on how this can be implemented?

[ruchi-shakya]

@hassy hi! I am looking if you have tried to implement Listen/Receive Events feature on Artillery. This feature is something I need in my load testing where I am trying to Emit and Listen on like 50+ users.

Any work around from your side would be really appreciated. Thanks!!