Newman: Feature request: Run test in parallel

[passsy]

Sadly the webservice I'm testing has some very slow calls. My test suite runs 30min+ when I wait for responses. When I'd be able to run test in parallel it would be much faster! Running folders in parallel would be enough.

[a85]

You can do this with multiple process instances?

[czardoz]

Yep, that would be one solution. Another solution is to use Newman as a library in a simple script. We have an example for parallel collection runs here: https://github.com/postmanlabs/newman/blob/896a7b61955ee59bc8bfcfa144e7c5aa3639194e/examples/parallel-collection-runs.js

All you need to change in this script is the options, where you would want to specify different options for each run (each containing a different folder name)

[passsy]

Is one of both solutions possible to publish combined results (lint, json)?

[czardoz]

Yes, but both involve writing some aggregation logic. For the first approach, you can use a JSON reporter, and aggregate results from the reports when all the parallel runs are complete.
For the second approach, it's a little easier, since you can aggregate results as they are available (i.e, _during_ the run)

[czardoz]

@passsy To add to my comment above, you can use the result.run object to find additional information about failures, successes etc :smile:

[simonmbcgdv]

@czardoz i'm running this script with 10 parallelCollectionRuns on newman 3.3.0 on linux and notice the following behaviour:

1) it's getting noticeably slower over the course of about 1h towards the tail end. I've run out of memory twice and increased the node old memory accordingly. I suspect it may be leaking memory?

2) While running postman scripts using parallelCollectionRuns, we have the old {{ unsubstituted variable }} sent to the server during execution, for a small fraction of requests only. This never happens when we run the script single threaded from the CLI using newman directly, where everything is substituted correctly. It looks like a side effect of the parallelCollectionRun.

anybody else experience this? If you need specs / further details please let me know.

Cheers,

Simon

[shamasis]

Just to add my two cents - Newman has this behaviour of growing memory usage owing to the amount of details it currently needs to hold on in memory. Down the line, we need to possibly run Newman in a mode where it flushes out results to disk and not store in memory. Until then, parallel or no parallel, long running and repeated requests will have this issue.

[sumit16986]

Please provide the parallel run feature to the newman as this is the most useful feature when some has the some huge amount of test cases........

[gas421]

@shamasis please consider this feature into the roadmap of Newman. I would like to point out the parallel execution will be really useful in the context of be able to speedup tests and have faster feedbacks (NOT for load testing purpose).
Thanks!

[flash286]

I'm voting for parallel run and it's gonna be fantastic useful.
Until newman implementing this, i did small tool written on golang around it which can help to run it in parallel mode and get aggregated stats. But it still very experimental.
https://github.com/flash286/postman-load-testing

[gas421]

Hello, thanks @flash286 for your message even if I have a little concern: I think the overall idea and vision of Postman is not to be also a Load Test tool, for this you have many other different tools and option. I see Postman as an API functional validation tool.
My message was focused more on the integration with Postman Newman and Jenkins (or TFS...). My point was to have a possibility to run in parallel the API functional validation tests step (i.e. at the end of a build sequence) in order to speed up the execution of the whole build.

Nothing to do with load testing.

Thanks!

[shamasis]

This is being tracked in main postman-app-support tracker. This feature is needed, but not exactly in the way we are thinking of. We would need to think more on this. 🤔

[gas421]

@shamasis can you post the link to this "main postman-app-support tracker"?
Is there a way to follow and get updates about this topic?
How are you going to proceed?
Thanks

[duddu]

Please review this PR and post your thoughts if you're still interested in concurrency: https://github.com/postmanlabs/postman-runtime/pull/867

[SarathkumarRajendran]

Please let me know how to clear the newman runner instance memory once the run is completed. Currently, i'm updating the environment variables from nodejs script to environment file and then passing it to newman runner. This is working for the first newman run instance.

But when again update environment file with new values and then calling newman run 2nd time, it's still running with old environment values (NOT taking newly updated values) , whereas when we open the environment file, could see that values are really updated inside it.

Can anyone please let me know the solution for this - or how to clear the neman run instance memory whenever the run is completed, such that it read the environment file with same name again but with updated values.

[danielbraus]

@czardoz, what would I add to this script to specify an environment, a data file, and a certain number of iterations to run?

https://github.com/postmanlabs/newman/blob/896a7b61955ee59bc8bfcfa144e7c5aa3639194e/examples/parallel-collection-runs.js