newman.run() not pulling in the environment option

[danonorato]

It appears newman.run() parses a exported Postman environment JSON file w/out error but fails to use the environment at run-time.

here is the console output from a very simple test.js file as well as running the same collection and environment via newman CLI(which works as expected). I've attached the newman_pos.js file along with the environment and collection files.

The newman_poc.js script prints out the summary.collection. you can see the environment is []. You will also notice in the console output the InvalidURL error as the {{url}} env var is not being pulled in from the environment.

```/src/postman danonorato $ node newman_poc.js
running a collection...
Running 1 request(s) and 1 iteration(s)
newman

google_test

→ http://{{url}}
GET http://{{url}} [errored]
2⠄ ReferenceError in test-script
collection run completed.
PostmanCollection {
description: PostmanPropertyDescription { content: '', type: 'text/plain' },
_: { postman_id: 'e61faac5-348d-4550-da1c-de37af2870b3' },
id: 'e61faac5-348d-4550-da1c-de37af2870b3',
name: 'google_test',
disabled: undefined,
items:
PostmanPropertyList {
members: [ [Object] ],
reference: { 'd4148304-7da8-4c04-b263-7d9545f1b353': [Object] },
Type: [Function],
_postman_propertyIndexKey: 'id',
_postman_propertyIndexCaseInsensitive: false },
events:
PostmanEventList {
members: [],
reference: {},
Type: { [Function: PostmanEvent] _postman_propertyName: 'Event' },
_postman_propertyIndexKey: 'id',
_postman_propertyIndexCaseInsensitive: false },
variables:
PostmanVariableList {
_environments: [],
members: [],
reference: {},
Type:
{ [Function: PostmanVariable]
_postman_propertyName: 'Variable',
types: [Object],
isVariable: [Function] },
_postman_propertyIndexKey: 'id',
_postman_propertyIndexCaseInsensitive: false },
version: undefined }

┌─────────────────────────┬──────────┬──────────┐
│ │ executed │ failed │
├─────────────────────────┼──────────┼──────────┤
│ iterations │ 1 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ requests │ 1 │ 1 │
├─────────────────────────┼──────────┼──────────┤
│ test-scripts │ 1 │ 1 │
├─────────────────────────┼──────────┼──────────┤
│ prerequest-scripts │ 0 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ assertions │ 0 │ 0 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 518ms │
├───────────────────────────────────────────────┤
│ total data received: 0B (approx) │
├───────────────────────────────────────────────┤
│ average response time: 0ms │
└───────────────────────────────────────────────┘

# failure detail

1⠄ Error Invalid URI "http:///%7B%7Burl%7D%7D"
at request
inside "http://{{url}}" of "google_test"

2⠄ ReferenceError responseCode is not defined
at test-script:5:31
inside "http://{{url}}" of "google_test"```

Now from the command line
```~/src/postman danonorato $ newman run -e google.postman_environment.json google_test.postman_collection.json
newman

google_test

→ http://{{url}}
GET http://www.google.com [200 OK, 5.31KB, 131ms]
✓ Status code is 200

┌─────────────────────────┬──────────┬──────────┐
│ │ executed │ failed │
├─────────────────────────┼──────────┼──────────┤
│ iterations │ 1 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ requests │ 1 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ test-scripts │ 1 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ prerequest-scripts │ 0 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ assertions │ 1 │ 0 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 691ms │
├───────────────────────────────────────────────┤
│ total data received: 4.7KB (approx) │
├───────────────────────────────────────────────┤
│ average response time: 131ms │
└───────────────────────────────────────────────┘```

I had to change the extensions to txt to attach.
newman_poc.txt -> newman_poc.js
google.postman_environment.txt -> google.postman_environment.json
google_test.postman_collection.txt -> google_test.postman_collection.json

newman_poc.txt
google.postman_environment.txt
google_test.postman_collection.txt

-> google_test.postman_collection.json

[kunagpal]

@danonorato This had been identified and fixed as a bug in #614, and will be rolled out in the next Newman release.

[danonorato]

great, any idea when that release may come out? Also, will newman ever support automatically printing out the details of failed asserts? IE. looking for responseCode 200 but got 503. Currently we only see that a assert failed, but not why it failed. This is normally something any test framework includes. ie.. rspec, jasmine, junit. etc... Seeing the reason for failure out of the gate is very helpful prior to debugging. best regards.

[kunagpal]

@danonorato The release will be out before the end of this week, we'll update this thread right after the release. As for the failed assertion details, there is a distinct section at the bottom of each collection run's CLI output, which aids debugging.

[danonorato]

yes, I've see the section, but its truly insufficient, unless i'm missing the missing the detail. For instance, if I expect the responseBody to === "Foobar". the output only prints that the Foobar assert failed, but it does not tell be why. IE. expect responseBody == Foobar actual = BarFoo

1⠄ AssertionFailure Status code is 2002
at assertion:1 in test-script
inside "http://192.168.99.100:9000/tasks" of "dano"

2⠄ AssertionFailure Response Body is FOOBAR
at assertion:2 in test-script
inside "http://192.168.99.100:9000/tasks" of "dano"

[kunagpal]

@danonorato Interesting. We'll take a closer look at your suggestions, thanks. :+1:

[kunagpal]

@danonorato This bug should have been fixed in Newman v3.1.1, please check it out and get back to us if your issue persists. Thanks!

[rodrigogs]

Sorry to post in an old issue, but this is still happening very often in 3.4.3 :(

[shamasis]

@kunagpal can you verify and reopen if needed?

[rodrigogs]

It's kinda hard to verify. The problem happens from time to time when I change my Postman tests and export them to a JSON that I use to run tests in my application.
Although all tests are running perfectly in Postman, some(just some) of them fail in newman due to URL variables are not being parsed in some cases.

Ex.: info: GET /api/v1/location/%7B%7Blocation_id%7D%7D 500 21.920 ms - 228

Again, the strange part is that those tests are passing when runnin on Postman.

http://pastebin.com/Rk7LzM6Z

[bijoej]

Happens for me as well trying to run the test from Newman.
The test runs fine from Postman, but fails from Newman saying "ReferenceError describe is not defined".
But I have got all latest version updated in my machine.
Any solutions?

[kunagpal]

@rodrigogs @bijoej Could you create separate issues for this? We'll take a look.

[bijoej]

sure kunal

Created...
https://github.com/postmanlabs/newman/issues/1128

[seekingua]

reproduce...

Invalid URI
"http:///%7B%7Bat_e_hostname%7D%7D/da%…
at request

newman version: 3.7.5

[kunagpal]

@SeekerResource It looks like you're trying to reference the variable called at_e_hostname, which is missing. Could you do the following to confirm that you're using the environment correctly?

1. Update Newman to v3.8.0.
2. Ensure that the newman run ... command includes an option to load the environment file, like so: newman run collection.json -e env.json. If you're using Newman as a library, ensure that the options object passed to newman.run contains a valid environment option, like so:

newman.run({
    collection: <>, // this can be a relative path to the collection, or the collection JSON itself
    environment: <> // similar to the above
}, function (err, summary) {
    // Do something with the err and summary here
});

1. In the test/pre-request scripts for the erroring request, please add console.log(environment); and verify that the resultant value is indeed valid.

If all of the above conditions are satisfied and you're still experiencing issues with variable resolution, please open a new issue at https://github.com/postmanlabs/newman/issues/new and we'll be more than happy to help you out. Thanks! 😄

[seekingua]

@kunagpal

this's my command to run newman:

newman run ${file_api} -e ${file_env} -e ${file_standard} -g ${path_global_test} --color

but i used two env files, both have environment variables with different name, the variable named 'at_e_hostname' is in file ${file_env}, files look like this:

${file_env}

{
  "id": "02fc487e-e0ce-2324-6118-41e9ee4cfa73",
  "name": "local",
  "values": [
    {
      "enabled": true,
      "key": "at_e_hostname",
      "value": "localhost:16600",
      "type": "text"
    },
...

${file_standard}

{
  "_": {
    "postman_variable_scope": "environment",
    "postman_exported_at": "2017-08-03T07:51:08.310Z",
    "postman_exported_using": "Postman/5.1.2"
  },
  "id": "0d56fd02-073e-5f43-a97a-1f27d7c815de",
  "name": "sandbox",
  "values": [
    {
      "type": "any",
      "value": "xxxx",
      "key": "4146ec82a0f0a638db9293a0c2039e6b"
    },
...

is this the reason?

[kunagpal]

Yes, Newman does not support multiple -e flags as of now, so only the last specified file will be considered.

[seekingua]

@kunagpal thx, and do we have a plan to support multiple -e flags. this is out case:

1、run newman to get result and write them into environment and export to new env files;
2、run newman again whenever using same env file and new env file from step 1, to compare api result with result that already in environment, if same to, then test passed, or failed

[kunagpal]

@SeekerResource

1. You can use the --export-environment new-env.json option to place the current environment into and specified file of your choice.

1. It looks like you want to check that responses are consistent across multiple requests. You can do this in the test script as follows:

var expectedResponse = {foo: 'bar'};

tests['Response is valid'] = _.isEqual(expectedResponse, JSON.parse(responseBody));

In the snippet above, responseBody is the response string, and _ is the inbuilt Lodash v3.10.1 object, documented here: lodash.com/docs/3.10.1

[seekingua]

@kunagpal this doesn't work for me. because if you changed some environment value in request, pre-req. or tests, when you export to new-env.json, they will take different values. and you will get different response if using as env when you run newman again. so i don't want to change value for variable from import env file when export to new env file, just only insert some new variable.

[nobe0716]

I found it!

I used

newman --environment {{ENV_PATH}} {{COLLECTION_PATH}}

in old version, but it does not work in latest version.

So I swap arguments like

newman {{COLLECTION_PATH}} --environment {{ENV_PATH}}

then, it works!

I wonder why the order of args affects the behavior of newman :(

[deepakpathania-zz]

@nobe0716 Hey this has been discussed in detail in #1336
Please go through the comments to have a better idea.

[gituser222]

Hello, my requirement is to run postman scripts in a CI environment using bamboo. We have a repo for the collection.json, however as my environment file has some sensitive data like client_id, secret_id, username, password etc.. I cant put my environment file in repo. In this case how can i run my collection in Newman .

[shamasis]

Hey there 😊 - you can provide global variables using Newman command line. Checkout readme to pass Newman variables via command line. If the secrets are available as environment variables inside your CI, you can simply pass it while calling newman --global-var “secret=$ENVVAR” style.

Also, this is not a good place to be spotted by someone to answer this query - this is a closed issue of a different nature and it is a “how to” question that goes into community.getpostman.com 😊

[pavelsmajda]

This issue is still present in Newman 4.5.7. When I try to get token then Postman process it correctly but Newman don't. I have token url set as environment variable.

Newman output (using environment variable):
GET TOKEN
POST [errored]
runtime:extenstions~request: request url is empty
2⠄ JSONError in test-script

Newman output (without using environment variable):
GET TOKEN
POST {{token_url}} [errored]
Invalid URI "http:///%7B%7Btoken_url%7D%7D"
2⠄ JSONError in test-script