Newman: Run one or few tests from collection

[igladun]

Hello,
It would be great to run certain tests or test group from a collection, not the whole collection.
Can we expect to have such feature?

[czardoz]

@igladun, you can run a specific folder inside a collection with the -f option. You can group the requests you want to run in a folder and then run it.. Does that work for you?

[igladun]

Hello @czardoz ,
We would like to run tests (requests) by specifying their name in options.

[czardoz]

@igladun Test names are not unique, so you can have a collection that has multiple tests with the same name.

[bgaillard]

Hi, I think we have the same needs as @igladun, in our case all our collections are organized the same way (i.e one folder per ws "prefix").

For example to test users (Web service http://myserver.com/users and http://myserver.com/users/:id) we have a folder users :

• GET user 1
• GET all without pagination
• POST test user 1
• POST test user 2

Because POST operations modify our testing database it can make fail other tests in our collection after. So in our case we would like to disable only specific requests from the tests.

I think 2 features would be useful for us :

• What's described here https://github.com/postmanlabs/newman/issues/231
• A disable button associated to the Tests tab (or the whole request) in Postman, this would force a specific request to not being executed / tested (implicitly not executed in Newman)

For the second point perhaps having only a disable button would be too specific, the other solution i'm thinking about would be to be able to add "generic tags" attached to request. Then we would be able to include or exlude request depending on their tags at command line.

newman --exclude-having-tags tag1,tag2,tag3 --include-having-tags tag4 --exclude-wihout-tests

Those tags could be used not only in newman but in other applications if needed (for exemple as a view filter in Postman...)

What do you think about that ?

[czardoz]

That's an awesome idea. Tags are on the roadmap for sure. I'll keep you guys updated :)

[bgaillard]

Hi @czardoz, thanks for your quick response :+1:

[igladun]

Thanks @czardoz !

[bgaillard]

Hi, I think their is one more feature which would be useful to filter tests to run, for now we can only select one folder while running tests.

For example :

newman --url https://www.getpostman.com/collections/XXXX --environment $WORKSPACE/postman/staging.postman_environment --folder brands --testReportFile junit-output-brands.xml

In our case we have lots of folders and run our tests on Jenkins, we do not want to take into account all folders. So we have several newman calls.

newman ... --folder folder1
newman ... --folder folder2
newman ... --folder folder3
...
newman ... --folder folderX

It would be great to update the folder [folder-name] option to a folders [folder-names...] option :

newman ... --folders folder1,folder2,folder3,...,folderX ...

What's you opinion about this feature ?

[czardoz]

The tags would make this rather redundant I think, but I'll keep this issue open until we take a call :+1:

Oh, and pull requests are always welcome :)

[a85]

@czardoz Is there something in v2 that'll help with this?

[czardoz]

No, we will have tags, but there's nothing that'll help with running multiple folders particularly. Personally, I don't think we can do anything specific in the collection itself to enable support for multiple folders. It would have to be a part of the library that is using the collection.. in this case, Newman.

[contactprad]

already feeling the pain about not able to select folders. This issue is blocker for me. we should have a way to include folders to run

[shamasis]

@czardoz - why can we not have multiple --folder switches?

[darrylkuhn]

+1 for multiple folder switches. In our case we have a folder of "test fixtures" that does some boiler plate stuff like get an access token, and fetch a list of valid values, etc... same as you might have in your typical unit testing implementation. Then once those fixtures are run we run the specific tests we're interested in. Having the ability to specify multiple folders would make it easy to leverage this common fixture pattern

[qarasha]

+1 for multiple folders. It's a feature that we need as well 👍

[rapidroamer]

+1

[kunagpal]

This is not a part of our current roadmap, but has been marked as a feature request and added to our internal tracker. We'll update this thread when any progress occurs on this. :)

[czardoz]

@kunagpal We should put it on our roadmap, and track it here, so everyone stays updated :)

[a-b]

Do we have any progress on tags?

Maybe it's faster and easier to implement filtering with regex on package level?

newman run ... --include '.*\[safe\]'

this will run tests ending with [safe] inline tag in all folders, like Verify feature [safe]

newman run ... --exclude '.*\[dev\]'

this will run all tests excluding ending with [dev]

Ideally I'd love to be able to provide multiple tags like

newman run ... --include 'safe' --include 'approved' --exlude 'under_development' --exclude 'broken'

[smaudet]

Postman is quite the horrid mess as is...I was hoping for a quick win with newman but it appears it is equally hobbled...

I think it would be just faster to write a small tool that rescues tests from postman's grip and runs them independently.

[mtaylorSF]

@czardoz Any update on this feature? When is it slated for release in your roadmap? Thanks

[darrylkuhn]

Hey all - while we all wait for this feature this may be of use. We have a collection that we use for integration testing with ~650 calls broken into many sub-folders. We have a "Fixtures" folder to do all the set up that each of the subsequent folders rely on. So in our set up we needed to be able to run "Fixtures" and then another folder. Using the simple bash script below we were able to achieve this. Hope this helps someone!

https://gist.github.com/darrylkuhn/28b83432c5b89c3872baf9455c210061

[kunagpal]

Would an option similar to --folder like --request do?

[arjunshukla]

Hi @czardoz @kunagpal any updates on this issue?
--request will work, but we are looking to execute this as a node module.
The intent for us is not to run in terminal but from node app.

[kunagpal]

@arjunshulka, Yes, the request option is intended to be available in both forms. There's no ETA on this, but this thread will be updated whenever any conclusion is reached.

[ejerskov]

@kunagpal
Any update on this soon?

[vikt0rs]

@kunagpal any updates, please?

[shamasis]

I have put some comment on #231.

For this issue, we need to discuss one of the following approaches:

1. Multiple folder switches
2. Regex match in folder switches
3. Also, request switches (with regex matches)

The thing to also look into is how Postman App and Monitors would handle this. And how much of confusion will it create if say, person-a shares collection with person-b expecting to run them with a few switches on.

[codenirvana]

@igladun @bgaillard @contactprad @darrylkuhn
Ability to select multiple folders has been added in Newman v4.1.0.

In CLI --folder option can be used multiple times and, for Library array of folder name/id can be passed to folder property.

[mohit-sentieo]

+1 for running request by name or tag. Along with friendly ids we could do request with Ids.

[emersonf]

Can someone at Postman chime in on why this issue has been closed without implementing a --request flag?

Those of us using Postman to manually test and hit our APIs have put hundreds of hours into creating requests and curating our collections. The folder structures we've come up with are meant for human consumption, not Newman. The ability to specify a single request would allow us to write shell scripts like:

1. Execute login request R1.
2. Execute data upload request R2, 3 times.
3. Execute data upload request R3, once for each CSV line.
4. Execute search request R4.

There's only one folder structure, but there are countless sequences of requests we may want to run, so folder-orchestrated executions won't solve this. Being able to pick out the request to execute and pass state between them is necessary to help us leverage the work we've put into Postman from scripts.

[paulmwatson]

+1 emersconf, we are a paying user of Postman and would appreciate being able to run specific requests with newman.

[shamasis]

As part of the bigger objective, this (selective run) will be implemented as part of collection runner upgrades while we work towards a "workflow" runner where one can mix and match anything. This is scheduled mid term. Will show up on public roadmap soon: https://trello.com/b/4N7PnHAz/postman-roadmap-for-developers

[Aypahyo]

@shamasis I inspected the roadmap - could not find it. Is this already part of newman or is this still in the future? I am looking for as many XUnit-Pattern features as I can get since we have layered setup / teardown requirements. Do you need someone to write up some bdd user stories / requirements? I could spend a few days helping to draft something based on my excessive NUnit testing experience if that helps in any way.

[chuck-wood]

Can someone at Postman chime in on why this issue has been closed without implementing a --request flag?

Those of us using Postman to manually test and hit our APIs have put hundreds of hours into creating requests and curating our collections. The folder structures we've come up are meant for human consumption, not Newman. The ability to specify a single request would allow us to write shell scripts as follows:

1. Execute login request R1.
2. Execute data upload request R2, 3 times.
3. Execute data upload request R3, once for each CSV line.
4. Execute search request R4.

There's only one folder structure, but there are countless sequences of requests we may want to run, so folder-orchestrated executions won't solve this. Being able to pick out the request to execute and pass state between them is necessary to help us leverage the work we've put into Postman from scripts.

This is a fantastic way to put it. There's a disconnect between the UI and the automation, and so far the only way to deal with it is to butcher the human experience to satisfy the automation.

[MartinDevillers]

Can someone at Postman chime in on why this issue has been closed without implementing a --request flag?

Those of us using Postman to manually test and hit our APIs have put hundreds of hours into creating requests and curating our collections. The folder structures we've come up with are meant for human consumption, not Newman. The ability to specify a single request would allow us to write shell scripts like:

1. Execute login request R1.
2. Execute data upload request R2, 3 times.
3. Execute data upload request R3, once for each CSV line.
4. Execute search request R4.

There's only one folder structure, but there are countless sequences of requests we may want to run, so folder-orchestrated executions won't solve this. Being able to pick out the request to execute and pass state between them is necessary to help us leverage the work we've put into Postman from scripts.

My teams are in the exact same situation. We're building a complex ecosystem of micro-services that we test using Postman. Our collections are modeled around business flows and contain series of requests, some of which are optional depending on our specific use case. We already have a naming convention in place to order, tag and label our requests like TSL-01-Authenticate, TSL-02-Perform-Preflight, TSL-02a-OPT-Warm-Auto-Pilot, TSL-03-Elect-Auto-Pilot-Network, TSL-04-Signal-Driver, etc.

Without a --request flag, there's no way for us to use these collections in Newman. I think the best option I have now is to create Newman specific collections with flows that are specifically tailored for Newman-- but that means duplicating requests as there is no way to reuse requests across collections (which is actually another feature request that's been around for a while).

It'd be great if I could target specific requests by name, preferably using a RegEx pattern. Or, another great feature would be if I could "tag" requests and use that to select requests for Newman.