Artillery: Bug: Artillery won't exit once the test is done

[hassy]

Affected versions: tested on 1.5.0-16 and 1.5.0-17

Debug log: http://hastebin.com/sasidexamo.vbs

Reported by @sjb933.

[hassy]

I've not been able to replicate with v1.5.0-16 and a simple test script that sends a bunch of requests to an endpoint that ends up timing out.

@sjb933 Is there any other output on the terminal after "All phases launched" is printed? There should be more output (either printing the number of pending request/scenarios or saying that all scenarios have completed). Does the spinner still run?

Also could you try running Artillery directly just to make sure that it's Artillery that doesn't exit rather than the shell script wrapping it?

[gboysko]

Hey @hassy, how did you get the debug log? Is this automatically generated? I am encountering a case where the workload is done, but Artillery keeps running indefinitely.

[hassy]

@gboysko debug info will be printed if you run DEBUG=artillery:runner,runner,http,socketio artillery run .... Which version of Artillery and Node.js are you on?

[gboysko]

We are using the latest (1.5.0-16) with Node 6.x. I'll make sure to enable that so that we can understand the hanging condition.

[sjb933]

Hi @hassy apologies I have not been able to provide more information. This issue arose during a work project and we're now beyond the point where I am able to test/reproduce. I hope that @gboysko can provide additional information that will help.

Thanks for the hard work!

[gboysko]

FWIW, we are unable to reproduce the hanging condition at this time. However, we are armed with the DEBUG settings the next time it happens!

[hassy]

@gboysko if you see it again please do share the logs. :-)

[joeldodson]

Hi @hassy, @gboysko, @sjb933,

I'm seeing the same behavior. The run finishes and artillery continues to run until I ctrl-c it.

If I run 60 seconds at 10 virtual users artillery will generally stop. If I run 20 virtual users, it does not stop.

I'm using versions:

[artillery] $ artillery --version
1.5.0-17
[artillery] $ node --version
v6.9.1

I ran with command:

DEBUG=artillery:runner,runner,http,socketio artillery run add_token.json

but the debug output didn't look very useful. It showed the HTTP requests and a line for each runner:

runner picking scenario 0 (Add token) weight = 1 +47ms

and after all the requests had finished, I get the all phases launched and ever 10 seconds the same report with NaN.

\ runner All phases launched +22ms

Report for the previous 10s @ 2016-12-07T01:08:11.643Z
Scenarios launched: 0
Scenarios completed: 0
Requests completed: 0
RPS sent: NaN
Request latency:
min: NaN
max: NaN
median: NaN
p95: NaN
p99: NaN
Scenario duration:
min: NaN
max: NaN
median: NaN
p95: NaN
p99: NaN

Here's my JSON:
{
"config": {
"target": "http://127.0.0.1:8000",
"phases": [
{
"duration": 60,
"arrivalRate": 20,
"name": "Stress"
}
],
"payload": {
"path": "name_and_token.csv",
"fields": [
"name",
"token"
],
"order": "sequence"
}
},
"scenarios": [
{
"name": "Add token",
"flow": [
{
"post": {
"url": "/api/v1/register",
"json": {
"name": "{{ name }}",
"token": "{{ token }}",
"os": "testing",
"app_name": "com.example",
"production": false,
"overwrite": true
}
}
}
]
}
]
}

I can reproduce this consistently. Unfortunately it's on a service I'm writing for work. I have a personal/generic node/express app I use to play around to learn. I'll try and reproduce the issue on that and send you the github link.

thanks,
Joel

[joeldodson]

here's the output from

[artillery] $ DEBUG=artillery:runner,runner,http,socketio artillery run add_token.json > debugoutput.txt 2>&1

debugoutput.txt

[hassy]

@joeldodson very useful, thank you! I'm able to reproduce the issue with the info you provided. 👍

[hassy]

So the issue seems to be caused by a bug in Node.js v6.8.1 and above, quite possibly the bug that caused #204:

Bug report: https://github.com/nodejs/node/issues/9756
PR: https://github.com/nodejs/node/pull/9759

I haven't isolated the code in Artillery itself that triggers the bug yet, but the test case will hang reliably on Node.js 6.8.1 and above, and work fine on 6.8.0 or 4.x.

@gboysko @sjb933 @joeldodson The solution is to use Node.js =<6.8.0 until the bug in Node is fixed.

EDIT: Confirmed the bug in Node.js is responsible by running the test case with a Node.js binary with the patch applied.

[gboysko]

Great job, @hassy. I can't be certain, but I feel that I've seen the hanging condition in Node 4.x as well. However, at the time, it was also an older version of artillery.

If we see the hanging condition again, we will resort to Node.js v6.8.0.

[joeldodson]

Thanks @hassy. I'm running artillery under node 6.7 now (already had that installed) and it's working great. I'll stay with that for the time being.
Thanks for the artillery project, it's very useful.

[hassy]

I have rewritten the code inside Artillery that triggered the bug. I think we should be OK even on Node.js 6.8.1+ now.

[hassy]

v1.5.0-19 with the fixes is out now - https://github.com/shoreditch-ops/artillery/releases/tag/1.5.0-19

[joeldodson]

thanks @hassy! it's working great.