Chromedp: Error when passing headless flags

[tarikguney]

I think this is what it is called "headless". However, when I run the following command, I get exceptions:

I have used the code from here https://github.com/knq/chromedp/blob/master/examples/simple/main.go but with the following difference:

c, err := cdp.New(ctxt, cdp.WithRunnerOptions(runner.Flag("headless", true),
    runner.Flag("disable-gpu", true), runner.Flag("--remote-debugging-port", "9222")))

instead of

c, err := cdp.New(ctxt, cdp.WithLog(log.Printf))

Exceptions

Why is this exception happening? Does chromedp not work without opening a browser window?

Environmental Settings:

1. Windows 10 Enterprise
2. Google Chrome Version 61.0.3163.100 (Official Build) (64-bit)
3. go version go1.9 windows/amd64

[yogesh-desai]

Even, I am facing the same problem. For me, the code works fine if I don't use headless option, but if I use it then the same kind of errors I am getting. My code is here. Line no. 227
@kenshaw: What might be the cause of this? Please help/ guide if I am doing something wrong or missed something somewhere. Thank you.

[tarikguney]

@yogesh-desai I was experimenting with simple-headless-chrome NodeJS package and it works perfectly with headless mode. Even though we closed all of the chrome instances including the background processes of chrome, it didn't start any visible chrome instance. At first, I thought it was a problem with the Chrome itself, but after seeing that library, I think it is a problem with the library itself.

In the original code I used above, perhaps --remote-debugging-port flag does not make sense with a headless browser since it requires a running instance.. However, I tried with the following code and still no luck:

c, err := cdp.New(ctxt, cdp.WithRunnerOptions(runner.Flag("headless", true),
        runner.Flag("disable-gpu", true)))

Since I am using Windows, so I also tried without disable-gpu flag and nope, no luck!

c, err := cdp.New(ctxt, cdp.WithRunnerOptions(runner.Flag("headless", true)))

It still looks like a problem with the library.

[kenshaw]

I don't believe headless mode on Windows is fully supported yet by the Chromium code base. I'll try some things and see if there is an issue. For what it's worth, if you specify --headless, you must also specify --remote-debugging-port. If the Windows Chromium code now fully supports headless mode, then I imagine the problem is simply that you are not passing the right combination of command line flags.

[kenshaw]

The issue here is that the --headless flag is not being provided, as well as all the other necessary variables to launch Chrome properly in headless mode. If you are using WithRunnerOptions then you need to specify everything, including the path to the Chrome executable:

Please note that the reason for this is that it would not be possible to distinguish between headless_shell and standard Chrome. We should probably add a option for launching regular Chrome with the --headless flag, since that is now supported out of the box on all three platforms. When chromedp/runner was originally implemented, only headless_shell was available, and there was no out of the box support on Windows, OSX, or Linux for running regular Chrome headless-ly. I'll make a point of updating the runner package to better support this workflow -- a PR would be greatly appreciated.

In the interim, something like this should probably work for you:

      path := `C:\Program Files (x86)\Google\Chrome\Application\chrome.exe`
      if runtime.GOOS != "windows" {
          path = "/usr/bin/google-chrome"
      }   

      c, err := cdp.New(ctxt, cdp.WithRunnerOptions(
          runner.Headless(path, 9222),
          runner.Flag("headless", true),
          runner.Flag("disable-gpu", true),
          runner.Flag("no-first-run", true),
          runner.Flag("no-default-browser-check", true),
      ), cdp.WithLog(log.Printf))
      if err != nil {
          log.Fatal(err)
      }

[tarikguney]

@kenshaw I think it does. I tested simple-headless-chrome NodeJS library on Windows, and it worked perfectly. I have tested it with --headless flag as well as --remote-debugging-port arguments with no luck. However, the incorrect argument combination may be the problem.

[kenshaw]

Note that the runner.Headless option only specifies the path to the executable, and the --remote-debugging-port flag.

[tarikguney]

@kenshaw Thanks for the suggestion. I will try that code. I also appreciate the background of the reason regarding why the library is working the way it is currently.

[kenshaw]

If I remember correctly, the reason I did not specify all those other options in runner.Headless was that headless_shell would fail to start when passed command line options that it did not recognize. The runner.Headless option was really meant to invoke/start headless_shell and not regular Chrome.

That's also why the comment on that option indicates "default settings for running the headless_shell executable".

[yogesh-desai]

@kenshaw : Thank you for your reply. I could understand your explanation. But in my case, the solution doesn't work. I want to open the chrome in "disable web security" mode. When I run with non-Headless; the program works. But if I do run it in headless mode then it just hangs. Please check and help me.
I tried following setup for headless which hangs after some time.
path variable is calculated as you have mentioned in your answer. I have tried cdpr.ExecPath(path), cdpr.Headless(path, 9222) also, but no luck.

c, err := cdp.New(ctxt, cdp.WithRunnerOptions(                                                                                                                                                          
                cdpr.Path(path),                                                                                                                                                                                
                cdpr.Flag("headless", true),                                                                                                                                                                    
                cdpr.Flag("disable-web-security", true),                                                                                                                                                        
                cdpr.Flag("no-first-run", true),                                                                                                                                                                
                cdpr.Flag("no-default-browser-check", true),                                                                                                                                                    
                cdpr.Flag("disable-gpu", true),                                                                                                                                                                 
        ), cdp.WithLog(log.Printf))                

[tarikguney]

@kenshaw Aren’t you supposed to add --remote-debugging-port=9222 as well?

[yogesh-desai]

@atarikguney : I have tried it as well but no luck for me. :(
My task is to open chrome in headless mode then navigate to a web page with --disable-web-security mode on and scrap few things from iframes present on the page. Chrome with UI is working but somehow --headless is not working in my case.

[yogesh-desai]

@atarikguney @kenshaw An update: I have tried following code which worked. Actually, I was using cdp.Shutdown() & cdp.Wait() to shut down and wait until chrome shutdowns. I commented it out and found that code works well.

      path := getOS()                                                                                                                                                                                            
        c, err := cdp.New(ctxt, cdp.WithRunnerOptions(                                                                                                                                                             
                cdpr.Headless(path, 9222),                                                                                                                                                                         
                cdpr.Flag("headless", true),                                                                                                                                                                       
                cdpr.Flag("disable-web-security", true),                                                                                                                                                           
                cdpr.Flag("no-first-run", true),                                                                                                                                                                   
                cdpr.Flag("no-default-browser-check", true),                                                                                                                                                       
                cdpr.Flag("disable-gpu", true),                                                                                                                                                                    
        ), cdp.WithLog(log.Printf))                                 


       err = c.Run(ctxt, isPresent(url, &buf1))                                                                                                                                                                   
        check(err, "Error in Run method of cdp")                                                                                                                                                                   

Problem: Though, it does not wait for the specified time cdp.Sleep() and hence failed to achieve expected work. On the other hand, chrome with GUI is working as per expectations. Please help me to solve the issue. What should I use to make headless setup explicitly wait for given time period?
Following is the function of Tasks which I want to accomplish,

// isPresent checks the existence of webyclip-widget-3 element.                                                                                                                                                    
func isPresent(url string, res *[]byte) cdp.Tasks {                                                                                                                                                                

        return cdp.Tasks{                                                                                                                                                                                          
                cdp.Navigate(url),                                                                                                                                                                                 
                cdp.Sleep(15 * time.Second),
                cdp.EvaluateAsDevTools("if (document.getElementById('webyclip-thumbnails')) {document.getElementById('webyclip-thumbnails').childElementCount;} else {console.log('0')}", res),                    
        }                                                                                                                                                                                                          

}                         

[tarikguney]

@yogesh-desai Thanks for sharing your solution and opinions. I have been thinking to solve this problem and send a pull request. Hopefully, soon I will come up with a pull request. At first, I and a friend of mine had been thinking to develop our own headless chrome library written in Go based on puppeteer and simple-headless-chrome developed in NodeJS, but later I decided to work on this project instead. I think I would be more effective by working on an existing headless chrome library and improving it instead. To my luck, JetBrains (known with their famous IntelliJ and Resharper tools) has been developing a new IDE (temporarily named Gogland), and it will make things a little easier for me going forward while contributing to this library.

[yogesh-desai]

@atarikguney @kenshaw Update: My problem is solved now. Thank you for the help and your time.