Chromedp: How to get fulle page source html code?

[liudanking]

Currently, I am using cdp.OuterHTML("html", &domHtml). But it is not perfect.

Any ideas?

[kenshaw]

Try chromedp/cdp/dom.GetOuterHTML with a NodeID of 0:

html, err := dom.GetOuterHTML().WithNodeID(cdptypes.NodeID(0)).Do(ctxt, c)

[kenshaw]

Sorry, I wasn't really thinking when I wrote the response above. Apologies -- you need to listen for the network events and capture the network and loader. It's on my TODO list to make this easier.

[liudanking]

@kenshaw Cool.

[kenshaw]

An update to the package will include a simple API call for this.

[slav123]

@kenshaw so... what is the proper way to get generated and altered (via JS) code ? I also was playing with chromedp.OuterHTML(html, res, chromedp.ByQuery), but definetly it's not that. Just need get to final DOM generated after everything is load. Sth. like web inspector view in real chrome.

[tmathews]

What's the status on this? There are two issues that show how to grab the full page html, but as far as I have tried neither are up to date or work.

[tmathews]

Figured it out.

func scrapIt(url string, str *string) chrome.Tasks {
    return chrome.Tasks{
        chrome.Navigate(url),
        chrome.Sleep(2000 * time.Millisecond),
        chrome.ActionFunc(func(ctx context.Context, h cdp.Executor) error {
            node, err := dom.GetDocument().Do(ctx, h)
            if err != nil {
                return err
            }
            *str, err = dom.GetOuterHTML().WithNodeID(node.NodeID).Do(ctx, h)
            return err
        }),
    }
}

[nifflin]

An update to the package will include a simple API call for this.

Is the simple API working now?

[nifflin]

Figured it out.

func scrapIt(url string, str *string) chrome.Tasks {
  return chrome.Tasks{
      chrome.Navigate(url),
      chrome.Sleep(2000 * time.Millisecond),
      chrome.ActionFunc(func(ctx context.Context, h cdp.Executor) error {
          node, err := dom.GetDocument().Do(ctx, h)
          if err != nil {
              return err
          }
          *str, err = dom.GetOuterHTML().WithNodeID(node.NodeID).Do(ctx, h)
          return err
      }),
  }
}

I tested it, this function doesn't work.

[ta1bbty]

@smcduck you probably need to wait till the whole DOM is loaded.
Since it's not still possible using chromedp ( @kenshaw ), I am using this as a quickfix.

package main

import (
    "fmt"
    "log"
    "os/exec"
)

func main() {
    out, err := exec.Command("sh", "-c", "/root/tools/chromium-latest-linux/latest/chrome --headless --no-sandbox --dump-dom http://url_here").Output()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s\n", out)
}

I am using Chromium 73.0.3671.0

[mvdan]

We're working on an API redesign, so there's no simpler way to do this at the moment. We hope to have something ready soon, but I don't want to promise any ETAs.

[lansana]

@mvdan Just to confirm, the examples above that attempt to use chromedp to render an HTML page, wait for the DOM to load and return the HTML result, do not work.

Your team is also currently working on an API redesign that will solve these problems.

And no estimated ETA.

Is that right?

[mvdan]

The scrapIt example above uses a sleep, so it's bound to be somewhat unstable.

Yes, that is correct about the v2 redesign.

[mvdan]

In the lastest chromedp master, Navigate plus dom.GetOuterHTML should work with no sleeps at all, because the navigate action waits for the page to complete loading via the frameStoppedLoading event. This includes waiting for the page's JS code to finish running.

Of course, if the page asynchronously loads extra HTML elements later, those won't be covered. You can always use combinations of functions like WaitReady with xpath selectors to wait for certain nodes to appear.

Alternatively, like Ken mentioned, you can grab the original HTML code by capturing the network response via an event. See my response in https://github.com/chromedp/chromedp/issues/105#issuecomment-490155758.

[Xeoncross]

FYI, after the update the code seems to work fine for me when using the new API.

func scrapIt(url string, str *string) chromedp.Tasks {
    return chromedp.Tasks{
        chromedp.Navigate(url),
        chromedp.ActionFunc(func(ctx context.Context) error {
            node, err := dom.GetDocument().Do(ctx)
            if err != nil {
                return err
            }
            *str, err = dom.GetOuterHTML().WithNodeID(node.NodeID).Do(ctx)
            return err
        }),
    }
}

[joshcano]

There's a race condition going on.
Sometimes the dom.GetDocument().Do(ctx) pulls in the correct node.

This works 20% of the time, on master#3e490a6

Can we reopen this issue?

package main                                                                       

import (                                                                           
  "context"                                                                        
  "fmt"                                                                            

  "github.com/chromedp/chromedp"                                                   
  "github.com/chromedp/cdproto/dom"                                                
)                                                                                  

func main() {                                                                      
  ctx, cancel := chromedp.NewContext(context.Background())                         
  defer cancel()                                                                   

  var res string                                                                   

  err := chromedp.Run(ctx,                                                         
    chromedp.Navigate(`http://example.com`),                                       
    chromedp.ActionFunc(func(ctx context.Context) error {                          
      node, err := dom.GetDocument().Do(ctx)                                       
      if err != nil {                                                              
        return err                                                                 
      }                                                                            
      res, err = dom.GetOuterHTML().WithNodeID(node.NodeID).Do(ctx)                
      return err                                                                   
    }),                                                                            
  )                                                                                

  if err != nil {                                                                  
    fmt.Println(err)                                                               
  }                                                                                

  fmt.Println(res)                                                                 
} 

[mvdan]

Sorry, but 3e490a6 doesn't seem to be a valid commit. If you encounter some form of race, it's easiest if you file a separate issue.

Also see ExampleRetrieveHTML, #370, and #42. I don't think reopening this issue is a good idea right now, because the original question is too broad, and the two issues above already tackle specific features.

[mvdan]

Hmm, it looks like GitHub can indeed resolve that commit. My local git couldn't:

$ git show 3e490a6
fatal: ambiguous argument '3e490a6': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
$ git show 3e490a6 -- .
fatal: bad revision '3e490a6'

My best guess is that seven characters isn't enough these days.

Edit: I was on an old master clone, so ignore all that.

[YoTM]

Hmm... And how to get full visible text from html page?