Newman: What is the difference between AssertionError and AssertionFailure

[prateeksarda]

1. Newman Version (can be found via newman -v): latest
2. OS details (type, version, and architecture): Linux
3. Are you using Newman as a library, or via the CLI? NodeJS library
4. Did you encounter this recently, or has this bug always been there: recently
5. Expected behaviour: (See point 5)
   With bail set as true, while command (a) stops the execution of collection as it fails
   but command (b) continues and gracefully stops the execution.

a) pm.expect(1).to.be.equal(2)
// AssertionError is thrown here

b) pm.test('MyTest', () => {
pm.expect(1).to.be.equal(2)
})
// AssertionFailure and AssertionError is thrown here

1. Command / script used to run Newman:
   newman.run({ collection: <collectionname> })

2. Sample collection, and auxiliary files (minus the sensitive details):

{
    "info": {
        "_postman_id": "f0de4cb5-0714-4264-b446-16253c3f4420",
        "name": "Sample Collection",
        "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
    },
    "item": [
        {
            "name": "Sample Request",
            "event": [
                {
                    "listen": "test",
                    "script": {
                        "id": "fcae8c4d-4a1b-4a60-b922-b8b8d1bef6da",
                        "type": "text/javascript",
                        "exec": [
                            "pm.expect(1, 'Outside Test : Expect').to.be.equal(2)",
                            "",
                            "pm.test('Test', () => {",
                            "   pm.expect(1, 'Inside Test : Expect').to.be.equal(2)",
                            "})"
                        ]
                    }
                },
                {
                    "listen": "prerequest",
                    "script": {
                        "id": "ad8f76e9-e215-43b4-bf3d-76d969b45b30",
                        "type": "text/javascript",
                        "exec": [
                            ""
                        ]
                    }
                }
            ],
            "request": {
                "method": "GET",
                "header": [],
                "body": {},
                "url": {
                    "raw": "http://postman-echo.com/get?foo=bar",
                    "protocol": "http",
                    "host": [
                        "postman-echo",
                        "com"
                    ],
                    "path": [
                        "get"
                    ],
                    "query": [
                        {
                            "key": "foo",
                            "value": "bar"
                        }
                    ]
                }
            },
            "response": []
        }
    ]
}

1. Screenshots (if applicable):

[kunagpal]

@prateeksarda Apologies for the delay here.

AssertionError
pm.expect assertions (as well as other assertions in general) work by throwing an error. When a pm.expect statement is present outside the pm.test wrapper, an assertion failure will cause an error to be thrown, causing a script error. This is what would cause your collection run to exit prematurely (if the bail flag has been set).

AssertionFailure
However, pm.expect statements inside the pm.test enclosure will merely cause that singular test to fail if there is an assertion error. While this too can cause your collection run to terminate, the termination happens more gracefully due to a test failure, and not an arbitrary error being thrown. In the second case, the script containing a failed test will continue to completion.

[prateeksarda]

@kunagpal Thanks Kunal that explains it well.