Chromedp: Iterating Text Action over all nodes matched by selector

[t-blake]

I understand that the Text Action returns visible text only from the first node matching the selector. This behaviour is unfortunately shared by all convenience functions in query.go.

// Text retrieves the visible text of the first node matching the selector.
func Text(sel interface{}, text *string, opts ...QueryOption) Action {
    if text == nil {
        panic("text cannot be nil")
    }

    return QueryAfter(sel, func(ctxt context.Context, h cdp.Handler, nodes ...*cdp.Node) error {
        if len(nodes) < 1 {
            return fmt.Errorf("selector `%s` did not return any nodes", sel)
        }

        return EvaluateAsDevTools(fmt.Sprintf(textJS, nodes[0].FullXPath()), text).Do(ctxt, h)
    }, opts...)
}

Is there an efficient and/or idiomatic way to iterate the Text Action (and other convenience functions defined in query.go) over all nodes matched by the selector?

I think that what is required is to iterate from nodes[0] to len(nodes), but I am not sure how to implement this (apologies, I am quite new to Go).

Finally, many thanks for your excellent work on Chromedp!

[kenshaw]

As you can see, this is simply a small wrapper around executing javascript. My intention with all of the high-level actions in chromedp is that they will target the first, single node. We might consider adding more high-level actions that have different style names that target groups of returned nodes, or a general select option. I haven't really fomented my thoughts on how I would like to do this. As this isn't going to be changed in the short term, I would suggest just writing an action with the appropriate Javascript for returning the relevant text nodes you're concerned about.

For the record, my hesitancy to add more options/configurable logic to the high-level actions is to avoid reinventing a DSL in Go through func's. Many of the requests for changes to the API have been to add more indirection / logic to high-level actions that would be configurable via something like options / common logic -- ie, something like chromedp.If(). The problem with that is that you just end up reinventing a general programming language, and add an additional level of obfuscation/confusion over the intention.

[t-blake]

@kenshaw: Thank you for your response. I do understand your reticence in adding an additional layer of high level logic. However, for what its worth, my opinion is that if cdp is to be considered as a drop in replacement for Selenium (as you implied at the Singapore Gophercon), additional high level functions would facilitate uptake. Scraping data from all nodes matched by the selector would appear to be fairly standard in any scraping workflow.

Having said that, the solution to the issue was simple. I include the code that I used for future reference.

func jsGetText(sel string) (js string) {
    const funcJS = `function getText(sel) {
                var text = [];
                var elements = document.body.querySelectorAll(sel);

                for(var i = 0; i < elements.length; i++) {
                    var current = elements[i];
                    if(current.children.length === 0 && current.textContent.replace(/ |\n/g,'') !== '') {
                    // Check the element has no children && that it is not empty
                        text.push(current.textContent + ',');
                    }
                }
                return text
             };`

    invokeFuncJS := `var a = getText('` + sel + `'); a;`
    return strings.Join([]string{funcJS, invokeFuncJS}, " ")
}

func Scrape(url, readySel, textSel string, text *[]string) cdp.Tasks {
    jsText := jsGetText(textSel)

    return cdp.Tasks {
        cdp.Navigate(url),
        cdp.WaitReady(readySel, cdp.ByID),
        cdp.Evaluate(jsText, &text),
    }
}

[stemsin]

Is it possible to add some more details to the sample, please? If I want to go through a page and on each section with a specific selector get some texts/attributes, how do I do this?

Something similar to what is done here: http://go-colly.org/docs/examples/basic/
Cannot use colly as I need my target site to be rendered first and chromedp looks fantastic for this purpose.