Chromedp: Child nodes are not loaded using Nodes()

[orlangure]

What versions are you running?

github.com/chromedp/chromedp v0.2.1-0.20190430114454-d15a83b92825
go version go1.12.4 darwin/amd64

What did you do?

I'm trying to find table rows on a webpage served by a test server (code below). For every row found, I'm trying to see its contents and access its child nodes:

package main

import (
    "context"
    "github.com/chromedp/cdproto/cdp"
    "github.com/chromedp/chromedp"
    "log"
    "time"
)

func main() {
    ctx, cancel := chromedp.NewContext(
        context.Background(),
        chromedp.WithLogf(log.Printf),
    )
    defer cancel()

    ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

    url := "http://localhost:81/"
    err := chromedp.Run(
        ctx,
        chromedp.Navigate(url),
        chromedp.WaitVisible("table"),
    )
    if err != nil {
        log.Fatal(err)
    }

    var rows []*cdp.Node
    err = chromedp.Run(
        ctx,
        chromedp.Nodes(
            "tr",
            &rows,
            chromedp.ByQueryAll,
        ),
    )
    if err != nil {
        log.Fatal(err)
    }
    log.Println("Found rows", len(rows))

    for _, row := range rows {
        log.Println("value", row.NodeValue, "type", row.NodeType.String(), "childcount", row.ChildNodeCount)
        for i, c := range row.Children {
            log.Println(i, c.NodeValue, c.ChildNodeCount)
        }
    }
}

Test server code:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/html; charset=utf-8")
        fmt.Fprintf(w, "<table><tr><td>1</td><td>2</td></tr><tr><td>3</td><td>4</td></tr></table>")
    })

    http.ListenAndServe(":81", nil)
}

What did you expect to see?

Found rows 2
value  type Element childcount 2
...some output for each child td...
value  type Element childcount 2
...some output for each child td...

What did you see instead?

No actual child nodes, while ChildNodeCount field says otherwise:

Found rows 2
value  type Element childcount 2
value  type Element childcount 2

I believe I'm misusing the API, and child nodes are not loaded automatically. I tried looking into CommandRequestChildNodes, but coultnd't figure it out.

[rnd]

i'm not really sure if this behaviour is intended or not. But accessing child nodes from chromedp.Nodes is not safe, because chromedp doesn't watch changes on returned nodes.

If you want to get the context from all the td elements, what you can do is to find the number of rows of the table, and get the text based on the number of the rows.

```package main

import (
"context"
"fmt"
"log"
"time"

"github.com/chromedp/cdproto/cdp"
"github.com/chromedp/chromedp"

)

func main() {
ctx, cancel := chromedp.NewContext(
context.Background(),
chromedp.WithLogf(log.Printf),
)
defer cancel()

ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
defer cancel()

url := "http://localhost:8000/"
err := chromedp.Run(
    ctx,
    chromedp.Navigate(url),
    chromedp.WaitVisible("table"),
)
if err != nil {
    log.Fatal(err)
}

var rows []*cdp.Node
err = chromedp.Run(
    ctx,
    chromedp.Nodes(
        "tr",
        &rows,
        chromedp.ByQueryAll,
    ),
)
if err != nil {
    log.Fatal(err)
}
log.Println("Found rows", len(rows))

// loop through all the td nodes
const sel = `/html/body/table/tbody/tr[%d]/td[%d]`
var firstCol, secondCol string
for i := 1; i <= len(rows); i++ {
    if err := chromedp.Run(ctx,
        chromedp.Text(fmt.Sprintf(sel, i, 1), &firstCol),
        chromedp.Text(fmt.Sprintf(sel, i, 2), &secondCol),
    ); err != nil {
        log.Fatal(err)
    }

    fmt.Printf("got first column value = %s, from row %d\n", firstCol, i)
    fmt.Printf("got second column value = %s, from row %d\n", secondCol, i)
}

}
```

[orlangure]

Thanks @rnd, it helped.
Should we add such example to the docs? I can take this task.

[rnd]

awesome, i don't think it's necessary to put this on the docs, because there are many ways to get this stuff done
but if you want you can add more example on https://github.com/chromedp/examples.