st2 execution list on large database of executions results in "Message length is larger than server max message size"

[trstruth]

SUMMARY

As part of a stress test of a HA deployment of stackstorm, I requested 4000 mistral workflow executions. The workflows successfully completed, however st2 execution list results in ERROR: 500 Server Error: Internal Server Error MESSAGE: Internal Server Error for url: http://st2api:9101/executions/?limit=50&exclude_attributes=result%2Ctrigger_instance&sort_desc=True&parent=null

After confirming that all of our st2 components and external systems were operational, I began trying other ways of requesting the execution data. I confirmed that st2 execution list --action=core.local returned successfully, then used st2 execution list --timestamp-gt 2018-10-02T15:00:00.000Z to retrieve a subset of the executions. The timestamp also allowed me to view some of the 4000 executions that I had requested.

I tailed the st2api logs and fired st2 execution list which resulted in the trace https://pastebin.com/crbhkmm5

My intuition is that the client which interfaces with mongo is not declaring its max message size correctly. Unfortunately, since st2 execution list is a curl that is similar to the request that populates the st2web homepage, we are also unable to view meaningful execution data from our webui.

After digging through the trace, and inspecting the mongoengine queryset docs, I see a way to set the batch_size here. Since I lack familiarity with mongodb and the internals of how stackstorm interfaces with it, this is about as far as I was able to get.

ISSUE TYPE

• Bug Report

STACKSTORM VERSION

Paste the output of st2 --version:
st2 2.9.0, on Python 2.7.6

OS / ENVIRONMENT / INSTALL METHOD

We have adapted our stackstorm deployment from the 1ppc st2-docker containers described in the st2-docker repo. We use ansible to deploy all stackstorm containers, and have them interface with SaaS services on Azure, and our own RabbitMQ deployment. It's important to note that we are using the MongoDB API for Azure Cosmos DB, as described here

STEPS TO REPRODUCE

On a 1ppc installation of stackstorm with ~4000 executions, st2 execution list

EXPECTED RESULTS

A table of the last 50 executions

ACTUAL RESULTS

ERROR: 500 Server Error: Internal Server Error
MESSAGE: Internal Server Error for url: http://st2api:9101/executions/?limit=50&exclude_attributes=result%2Ctrigger_instance&sort_desc=True&parent=null

[LindsayHill]

I have an ST2 system with 8,488 executions, and st2 execution list works as expected.

Looks like Cosmos does not yet fully support MongoDB 3.4, which ST2 is tested against.

What happens if you run your test against a MongoDB 3.4 system?

[trstruth]

@LindsayHill Thanks for the fast response, I'll do some more investigation against 3.4

[Kami]

On a related note - we are releasing v2.9.1 soon which will utilize new "include_attributes" functionality and might help with that, at least on the WebUI side (WebUI will only retrieve fields it needs and this will result in less data being retrieved from the database and returned to the client).

New request from WebUI to st2api will look something like that - curl http://st2api:9101/executions/?limit=50&parent=null&include_attributes=id,status,start_timestamp,action.ref,action.name,action.runner_type,action.parameters,parameters,rule.ref,trigger.type,context.user.

So you can see if that will help in your specific scenario by directly hitting the API with curl.

[trstruth]

We ended up switching back to MongoDB and the problem disappeared. Looks like a Cosmos issue like @LindsayHill suggested. Feel free to close this issue.

[LindsayHill]

@trstruth that's a shame about Cosmos - it would be good to know that it could be used in place of Mongo. Hopefully will work in the near future as Cosmos gets more updates.