Chromedp: Expose events via callbacks

[m4p1e]

there are many events in the CDP page , network, i want to set a function of one of them, what should i do? pls

[m4p1e]

and the other issue the chrome process cant shutdown when using headless mode ,cuz when closed all pages cant shutdown chrome in headless mode

[AndreaJegher]

Hi, I found a way to have a callback for each event without changing the code base.
Since it has been a month since you have posted, have you found a better solution?
Cheers!

package main

import (
    "context"
    "log"
    "encoding/json"
    "fmt"
    "strings"
    "github.com/chromedp/cdproto"
    "github.com/chromedp/cdproto/network"
    "github.com/chromedp/cdproto/page"
    "github.com/chromedp/cdproto/cdp"
    "github.com/chromedp/chromedp"
    "github.com/chromedp/chromedp/runner"
)

var (
    msgChann = make(chan cdproto.Message)
    ctxt context.Context
)

func devToolHandler(s string, is ...interface{}) {
    /*
     Uncomment the following line to have a log of the events
     log.Printf(s, is...)
    */
    /*
     We need this to be on a separate gorutine
     otherwise we block the browser and we don't receive messages
    */
    go func () {

    for _, elem := range is {
            var msg cdproto.Message
            // The CDP messages are sent as strings so we need to convert them back
            json.Unmarshal([]byte(fmt.Sprintf("%s", elem)), &msg)
            msgChann <- msg
        }
    }()
}

func main() {

    // create context
    ctxt, cancel := context.WithCancel(context.Background())
    defer cancel()

    // create chrome instance
    c, err := chromedp.New(ctxt,
        chromedp.WithRunnerOptions(
            runner.Flag("disable-infobars", true)),
            chromedp.WithLog(devToolHandler))
    if err != nil {
        log.Fatal(err)
    }

    err = c.Run(ctxt, network.Enable())
    if err != nil {
        log.Println(err)
    }

    c.Run(ctxt, chromedp.ActionFunc(func(_ context.Context, h cdp.Executor) error {
        // This is in an action function because the Executor is needed in some method
        go func() {
            for {
                /*
                Right now I have no guaranties (did not run enough tests) that the messages are evaluated
                in the same order they are received
                */
                msg := <- msgChann

                switch msg.Method.String() {
                case "Page.frameScheduledNavigation":
                    var schednavevent page.EventFrameScheduledNavigation
                    json.Unmarshal(msg.Params, &schednavevent)
                    fmt.Println(schednavevent.URL)

                case "Network.requestWillBeSent":
                    var reqWillSend network.EventRequestWillBeSent
                    json.Unmarshal(msg.Params, &reqWillSend)
                    fmt.Println(reqWillSend.Request.URL)

                case "Network.responseReceived":
                    var respevent network.EventResponseReceived
                    // json.Unmarshal(msg.Params, &respevent)
                    // This contains a bunch of data, like the response headers
                    fmt.Println(respevent.Response)
                    /*
                    Uncomment the following lines if you want to print the response body
                    rbp := network.GetResponseBody(respevent.RequestID)
                    b, e := rbp.Do(ctxt, h)
                    if e != nil {
                        fmt.Println(e)
                        continue
                    }
                    fmt.Printf("%s\n", b)
                    */

                default:
                    continue
                }
            }
        }()
        return nil
    }))


    err = c.Run(ctxt, chromedp.Navigate("https://duckduckgo.com"))
    if err != nil {
        log.Fatal(err)
    }

    /*
    Since chromedp.Navigate does not wait for the page to be fully loaded
    we wait manually, there may be a better and more reliable way to do this
    */
    state := "notloaded"
    for {
        script := `document.readyState`
        err = c.Run(ctxt, chromedp.EvaluateAsDevTools(script, &state))
        if err != nil {
            log.Println(err)
        }
        if strings.Compare(state, "complete") == 0 {
            break
        }
    }

    // shutdown chrome
    err = c.Shutdown(ctxt)
    if err != nil {
        log.Fatal(err)
    }

    // wait for chrome to finish the shutdown
    err = c.Wait()
    if err != nil {
        log.Fatal(err)
    }
}

[m4p1e]

good way！ the logger way , yes i motified the handler.go ，i let the handler can deal any msg from devtools that i want ， but thanks to hear a good way from u！ cheer！

[pmurley]

I would love to hear from @kenshaw what his view on the right way to do this is, and whether we could add something like this to the repo so we could handle arbitrary events from any domain. Maybe there is already is a way to do this built in (other than the two ways above) that we have just missed?

[Northern-Lights]

I have an open PR here addressing the issue of being able to collect events:

https://github.com/chromedp/chromedp/pull/259/files

It allows you to supply a completely optional channel for events when you create the new CDP, meaning it isn't required for those who don't want to use it and won't affect code that doesn't use it. On the channel, you can then receive events which tell you the target (tab, etc.) they come from, and then you can of course access the cdproto.Message. You can filter which ones you want or don't want from there.

I'd love to get your feedback.

[tj]

+1 for events, definitely need those! Spent a while hunting around for them, thought they were in the API somewhere

[ArcticZero]

+1 on this. Could really use a way to fetch the request response after Navigate(), so I can do things like, say, determine whether a page loaded successfully or not (via HTTP status code) before attempting to parse the DOM.

[mvdan]

There's a bunch of somewhat similar issues and PRs, all converging around events. We're currently designing v2 of this module, and we'll have a go at this feature. For example, I think something like #251 could be elegant. We can use this issue to track progress.

[Northern-Lights]

@mvdan thanks for the updates, and I'm glad to see that this feature request has garnered attention. I disagree that #251 is an elegant solution because it creates an awkward

type EventFunc func(ev interface{})

definition and then exposes an

OnEvent(hook EventFunc)

method on the TargetHandler, which I believe is usually wrapped by the cdp.Executor interface, e.g. when obtaining it through

func (c *CDP) GetHandlerByIndex(i int) cdp.Executor

An elegant solution would enable everything to be setup from the initial instantiation of the CDP object, e.g. through one of the options as opposed to having to go through lower levels and performing type casts each time a new target handler is created.

I hope this is considered during the design of v2.

[mvdan]

Yes, you're right - note that I tried to say "something like X could be elegant" :)

The new API will be vastly different when it comes to starting and stopping Chrome processes, so it's hard to tell what the events API could look like. All I meant to say is that it would be nice if the API overhead was as small as what's shown in that PR.

[pmurley]

I wanted to check progress on this in the refactor. I definitely need arbitrary event callbacks for my use case. Is this a feature that will be arriving soon, or am I better off forking and implementing? Would it be something you would consider accepting as a pull request if I implement?

As always, thanks for your help!

[kenshaw]

Either in the initial v0.2.0 tag or shortly after in the v0.2.x, there will be a solid, "idiomatic Go" way of handling events. I have prototyped 3 different APIs over the last 2 years, but have not been satisfied with any of them. I am very close to finalizing what I feel is the right path forward, but it takes time. It'll be done when it's done.

[pivolan]

some changes in devToolHandler:

var msgChann chan cdproto.Message

func  devToolHandler(s string, is ...interface{}) {
    /*
     Uncomment the following line to have a log of the events
     log.Printf(s, is...)
    */
    /*
     We need this to be on a separate gorutine
     otherwise we block the browser and we don't receive messages
    */
    go func() {
        for _, elem := range is {
            var msg cdproto.Message
            var msgIn struct {
                SessionId string `json:"sessionId"`
                Message string `json:"message"`
            }
            var msgLast cdproto.Message
            // The CDP messages are sent as strings so we need to convert them back
            err := json.Unmarshal([]byte(fmt.Sprintf("%s", elem)), &msg)
            if err != nil {
                continue
            }
            err = json.Unmarshal(msg.Params, &msgIn)
            if err != nil {
                continue
            }
            err = json.Unmarshal([]byte(msgIn.Message), &msgLast)
            msgChann <- msgLast
        }
    }()
}

And Init:

ctx, cancel = chromedp.NewContext(ctx, chromedp.WithDebugf(devToolHandler), chromedp.WithErrorf(devToolHandler), chromedp.WithLogf(devToolHandler))

to enable network logs:
err = c.Run(ctxt, network.Enable())

[pmurley]

@kenshaw Understood, thanks. Just so I know what to plan for: Do you anticipate that, once this solution is in place, chromedp will allow subscription to all events across all domains of the DevTools protocol, or will it be limited to a subset?

[kenshaw]

@pmurley yes.

[pmurley]

Thanks! So exciting - can't wait. If there's anything I can do to help out, let me know - I'm paused on my project until this feature arrives.

[mvdan]

We've added the initial, low-level API for this - it allows listening for target or browser events. It's also possible to listen for target events from multiple tabs, as it is context-based (layered).

For now, it's not possible to stop the listener, other than stopping the target or browser entirely. We'll add that possibility as well.

Here's a quick example:

ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()

chromedp.ListenTarget(ctx, func(v interface{}) error {
        fmt.Printf("%#v\n", v)
        return nil
})

if err := chromedp.Run(ctx,
        chromedp.Navigate("https://github.com/chromedp/chromedp/issues"),
        chromedp.WaitVisible("#start-of-content", chromedp.ByID),
); err != nil {
        t.Fatal(err)
}

You should then see a bunch of target events being printed:

&page.EventFrameStartedLoading{FrameID:"8616C4A0D594F7189BA4B7B7D470E69F"}
&runtime.EventExecutionContextDestroyed{ExecutionContextID:1}
&runtime.EventExecutionContextsCleared{}
&page.EventFrameNavigated{Frame:(*cdp.Frame)(0xc003dda000)}
&runtime.EventExecutionContextCreated{Context:(*runtime.ExecutionContextDescription)(0xc000032180)}
&dom.EventDocumentUpdated{}
&dom.EventChildNodeCountUpdated{NodeID:4, ChildNodeCount:39}

If you want to receive events from domains which aren't enabled by default, such as Network, you can do something like chromedp.Run(ctx, network.Enable()) before running the actions that would trigger events.

[mvdan]

In the future, we'll probably add higher-level APIs like "wait for this event", but you should still be able to implement that with the Listen API and a bit of code. The higher-level APIs would just make some common operations simpler.

[mvdan]

Just a quick clarification - these callback functions are called synchronously, so do not run any blocking code as part of them. This includes running further chromedp actions or commands, as that will essentially deadlock the target handler. The handler can't run your action because it is stuck handling an event, the event can't finish being handled until your callback finishes, and the callback won't finish until your action is run.

[mvdan]

The event callbacks are still synchronous in master after d5a3de5952e6a7ba0758843e35d38c3fe26fb01d, but a small change was made - it's now possible to make certain event callbacks run actions while the target handler is blocked on a wait function. For example, see TestCloseDialog, which successfully closes a dialog when its "opening" event is received.

You should still avoid running actions and blocking operations in callbacks whenever necessary, as it's easy to run into deadlocks, but in the case of alerts the entire tab gets locked up until the alert is closed.

[mvdan]

All should be done; I added an example above. See https://godoc.org/github.com/chromedp/chromedp#example-ListenTarget--ConsoleLog.

If anyone has any issues, please open a new issue on the tracker. We're starting to use the new Listen APIs for our tests and deployments, so as far as we can tell it works.

[matthewcummings]

This feature is excellent, thank you!

[mvdan]

For the sake of updating all the people in this thread, I implemented cancellation of listeners in 67e8409e37d4549c581e0357e040f0456a7091ef. This allows waiting for one specific event only, without leaking the listener. It's also now possible to add a listener to an existing browser or target (tab).

These two features are useful for the common use case of catching one specific event that is expected after one runs an action. For example, clicking a button that should open a new tab. I haven't written many examples for this stuff yet, but there are plenty of tests in master you can look at.

[matthewcummings]

Perfect, I need to do exactly this, I have some code which needs to wait for and parse a specific network.EventResponseReceived event.

[tutley]

Perfect, I need to do exactly this, I have some code which needs to wait for and parse a specific network.EventResponseReceived event.

Hi Matthew, did you ever figure out how to parse the network.EventResponseReceived event? I'd like to grab the first one of those on a page load and print the response code. (200, 401, 302, etc)

[matthewcummings]

Hi @tutley my code is in a private repo but I can give you the relevant snippets. Note that my code is basically a modified version of what @AndreaJegher is doing (see above).

[matthewcummings]

My listener function and goroutine:

func listenForNetworkEvent(ctx context.Context){
    chromedp.ListenTarget(ctx, func(ev interface{}) {
        switch ev := ev.(type) {
        case *network.EventResponseReceived:
            if ev.Type == "XHR" {
                resp := ev.Response
                if strings.Contains(resp.URL, "REDACTED" ) {
                    go func() {
                        if val, ok := resp.RequestHeaders["Cookie"]; ok {
                            responseChan <- *resp
                        }
                    }()
                }
            }
        }
    })  
}

This is where I wait for the specific response that I need (this snippet is from another function with only the relevant bits posted here):

    timer := time.NewTimer(time.Duration(BrowserTimeout) * time.Second)
    defer timer.Stop()

    for {
        select {
        case resp := <-responseChan:
            if val, ok := resp.RequestHeaders["Cookie"]; ok {
                if strings.Contains(fmt.Sprintf("%v",val), "REDACTED") {
                    return &resp, nil // RETURN here to break from both the select and for blocks
                }
            }
        case <-timer.C:
            log.Error(fmt.Sprintf("We timed out after %d seconds of waiting to capture the browser headers/cookies.", s.SyncConfig.rowserTimeout))
            // RETURN here iff we didn't find a matching network.Response
            return nil, err
        }
    }

I'm sure there's a more elegant way to do this but the above code works for me.

[slealq]

Hi!

I can't seem to get what should be the right way to get the ResponseBody of an http request.

So far:

`

func target_listener(ev interface{}) {

if ev, ok := ev.(*network.EventResponseReceived); ok {
    fmt.Println("event received:")
    fmt.Println(ev.Type)
    fmt.Println(ev.Response)

}

// print response body
c := chromedp.FromContext(ctx)
rbp := network.GetResponseBody(ev.RequestID)
body, err := rbp.Do(cdp.WithExecutor(ctx, c.Target))
if err != nil {
    fmt.Println(err)
}
if err = ioutil.WriteFile(ev.RequestID.String(), body, 0644); err != nil {
    log.Fatal(err)
}               
if err == nil {
    fmt.Printf("%s\n", body)
}   

}

`

BUT, cleary I don't have access to the contenxt in this callback. I tried another method but it says
2019/12/23 21:16:58 context deadline exceeded

I guess it has something to do with a deadlock...

Any help?

[marco-m]

See #543