Newman: Add documentation about what is passed to event listeners for each event.

[timhuff]

I spent about 45 minutes tracking down that, in .on('request', (err, obj)=>{}), that obj has the type signature

{
  response: Response,
  request: Request,
  item: Item,
  cookies: CookieList,
  history: any
}

where each of those types comes from the postman-collection library. This is not obvious at all.

(didn't bother figuring out what history was)

[shamasis]

Yep. We did write down parts of it in the postman-runtime repo.

@codenirvana @mohitranka lets set aside some time to consolidate it here.

@timhuff if you'd like to help, we are more than happy to partner up.

[timhuff]

@shamasis Sorry I missed your comment but I came back to share this with everyone. I'm glad you also find this attention worthy.

This was generated by me tapping the event emitter and programmatically generating a rough sketch. Then I had to dive deeper into a few aspects of it (such as wtf an Execution is. not sure if that's properly defined somewhere). There are still a few properties that are of type Object that might be able to be improved but it's only a handful.

Oh, this also makes the assumption that every .emit is of the form .emit(name, error, data)

import { NewmanRunSummary } from 'newman';
import {
  Collection,
  Cookie,
  CookieList,
  Event as PostmanEvent,
  Item,
  Request,
  Response,
  Script,
  VariableScope,
} from 'postman-collection';

type Execution = {
  event: PostmanEvent,
  script: Script,
  result: {
    id: string,
    target: string,
    data: Object,
    cookies: Array<Cookie>,
    _variables: VariableScope,
    environment: VariableScope,
    collectionVariables: VariableScope,
    globals: VariableScope,
    request: Request,
    return: { nextRequest: string, async: boolean },
    response: Response
  }
}

  on(eventName: 'start', listener: (err?: Error) => void): EventEmitter
  on(eventName: 'beforeIteration', listener: (err?: Error) => void): EventEmitter
  on(eventName: 'beforeItem', listener: (err?: Error, event?: {
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforePrerequest', listener: (err?: Error, event?: {
    events: Array<PostmanEvent>,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforeScript', listener: (err?: Error, event?: {
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforePrerequestScript', listener: (err?: Error, event?: {
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'script', listener: (err?: Error, event?: {
    execution: Execution,
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'prerequestScript', listener: (err?: Error, event?: {
    execution: Execution,
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'prerequest', listener: (err?: Error, event?: {
    executions: Array<Execution>,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforeRequest', listener: (err?: Error, event?: {
    request: Request,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'request', listener: (err?: Error, event?: {
    response: Response,
    request: Request,
    item: Item,
    cookies: CookieList,
    history: Object
  }) => void): EventEmitter
  on(eventName: 'beforeTest', listener: (err?: Error, event?: {
    events: Array<PostmanEvent>,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforeScript', listener: (err?: Error, event?: {
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforeTestScript', listener: (err?: Error, event?: {
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'script', listener: (err?: Error, event?: {
    execution: Execution,
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'testScript', listener: (err?: Error, event?: {
    execution: Execution,
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforeScript', listener: (err?: Error, event?: {
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'beforeTestScript', listener: (err?: Error, event?: {
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'console', listener: (err?: Error, event?: {
    level: String,
    messages: Array<string>
  }) => void): EventEmitter
  on(eventName: 'script', listener: (err?: Error, event?: {
    execution: Execution,
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'testScript', listener: (err?: Error, event?: {
    execution: Execution,
    script: Script,
    event: PostmanEvent,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'test', listener: (err?: Error, event?: {
    executions: Array<Execution>,
    item: Item
  }) => void): EventEmitter
  on(eventName: 'item', listener: (err?: Error, event?: {
    item: Item
  }) => void): EventEmitter
  on(eventName: 'iteration', listener: (err?: Error) => void): EventEmitter
  on(eventName: 'beforeDone', listener: (err?: Error, event?: {
    summary: NewmanRunSummary
  }) => void): EventEmitter
  on(eventName: 'done', listener: (err?: Error, event?: {
    collection: Collection,
    environment: VariableScope,
    globals: VariableScope,
    run: Object
  }) => void): EventEmitter
  on(eventName: 'exception', listener: (err?: Error, event?: {
    error: { type: string, name: string, message: string },
  }) => void): EventEmitter

[timhuff]

@shamasis if you've already seen my last comment, refresh the page. Had a few things I forgot to change for the public.

[timhuff]

@shamasis also, while I've got your guys' attention, how feasible would it be to provide a property in the event object that is a function that permits you to modify the runtime variables? It'd be great if I could inject information into the runtime based on the information I receive in the event listener. I'd be willing to make this change, provided that my pull request would be considered =P

@codenirvana @mohitranka

[shamasis]

@timhuff while I parse your comments in detail, here's a quick reply.

Almost all the names constructors you see in the event should be documented here https://www.postmanlabs.com/postman-collection/ since they're mostly collection SDK object instances.

Regarding being able to modify the variables on event, it may already possible (not through anything documented) but by getting access to the runtime runner instance. That also seems to be a more architecturally right way of doing this than using functions over events to mutate the producer data as a consumer.

[timhuff]

@shamasis Thanks for getting back. I came to realize that all but a small number were postman-collection classes. But, being someone who just started using newman, I was completely lost. I didn't even realize postman-collection was a thing.

You're totally right about the runner instance. I didn't want to flood this thread but while working with the event listeners, I found a method that was... it was something like "updateVariables" or something. It was kind of deep in one of the objects but it was there. Ultimately, I think a lot of this comes down to documentation / typescript typings. As an outsider, all the docs tell me is "this is when this event fires". Which helps.... but it makes it hard to figure out how I'm to accomplish my task when I don't know what's available to me.

All that said, I'm using this tool in a pretty unconventional way (basically, I'm using it as a headless browser against a horror of a 3rd party JSF server). So perhaps these are only really issues for me because I'm the edge case.

---

footnote: this server is so poorly designed that I ended up needing to put a proxy between postman and the target so that I could transform their responses into something that's not a nightmare to figure out in postman.... like status code 200 XML responses with <redirect url="..."></redirect> tags...

[shamasis]

Oh. I feel your pain.

We have a tonne of objects and a lot of deep embedded features and even have documentation. But it's disconnected and scattered all over. With a bit of help from everyone, I think we can make it easier for everyone. (And PS - no use case is far off. I'm glad newman is helping you.)

@mohitranka @codenirvana Let's introduce a document with reference to descriptions of these event parameters. We don't need to target full completion but if we add one entry then I'm sure all of us will come together and complete it.

[timhuff]

@shamasis Let me know if I can lend a hand. I'm unfamiliar with the codebase but if you have some grunt work, I'd be happy to contribute. This is a great tool you guys have here.

[shamasis]

Thank you for your generous offer. It is definitely our responsibility but we wouldn't have been able to create what we did without the love and input from everyone.

We will document a few events, iterate on it and leave it open for everyone to complete.

[Cronus1007]

@shamasis Shall I work upon this issue.

[AswinCG2002]

Is this issue fixed? If not can u tell me how to get working on this issue? @shamasis

[shamasis]

I think we have a branch or just about to commit one that has the structure of one event. Goal is to work together and complete the remaining.

@mohitranka when we have that ready, can you update us here?

[mohitranka]

@timhuff You can follow the event listener information here.

[timhuff]

@mohitranka Thanks for the heads up. Apologies for the late reply (swamped with work). This documentation just so happens to be exactly what I need right now, though. Thank you to everyone who helped make my nitpick a thing of the past 😄 . Keep up the good work 👍