Chromedp: How to set http proxy user and password in chromedp?

[shadow1163]

Hi, I know the chromedp could set http proxy on option "runner.Proxy(http://localhost:8000/)", but how to set the proxy's user and password?
Thanks

[kenshaw]

That's simply passed to Chrome. You would set it in that URL: runner.Proxy("http://<user>:<pass>@<host>:<port>/").

[judgedim]

@shadow1163 @kenshaw Is it really work for you? Because I got error

[judgedim]

Looks like I found problem https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212

@kenshaw, Is cromedp support similar thing as page.authenticate() as Puppeteer has?

[clanstyles]

Bump since this was never resolved?

[yavuzyurtbegendi]

@kenshaw I am having the same issue runner.ProxyServer("proxy with auth") does not work

[clanstyles]

Hey,

Auth doesnt work. It was a pain to try and support. Instead I booted a
local proxy server that handled the auth and accepted no password. That
worked fine.

On Wed, Dec 26, 2018, 1:15 AM Yavuz Yurtbeğendi <[email protected]
wrote:

@kenshaw https://github.com/kenshaw I am having the same issue runner.ProxyServer("proxy
with auth") does not work

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/chromedp/chromedp/issues/190#issuecomment-449926808,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnQxW-FrlXm3jh8bj0zdm6gv1C4vDTBks5u8zAcgaJpZM4UERAI
.

[yavuzyurtbegendi]

hi, thanks

Hey, Auth doesnt work. It was a pain to try and support. Instead I booted a local proxy server that handled the auth and accepted no password. That worked fine.
…
On Wed, Dec 26, 2018, 1:15 AM Yavuz Yurtbeğendi @.* wrote: @kenshaw https://github.com/kenshaw I am having the same issue runner.ProxyServer("proxy with auth") does not work — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#190 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnQxW-FrlXm3jh8bj0zdm6gv1C4vDTBks5u8zAcgaJpZM4UERAI .

[yavuzyurtbegendi]

@clanstyles, I am using this and tried so many other options, I cannot find other options, can you help me on this issue

http_access allow all
http_port 3128

coredump_dir /var/spool/squid3

cache_peer MY_EXTERNAL_PROXY_IP parent MY_EXTERNAL_PROXY_PORT 0 no-query default login=my_login_user:my_login_pass
never_direct allow all

[clanstyles]

@yavuzyurtbegendi I used https://github.com/elazarl/goproxy to setup a local proxy and just forward it. This proxy handles the auth.

ChromeDP ends up listening on localhost:8888 (proxy server) and the goproxy ends up forwarding the request and satisfying the auth requirements. I'm also collecting meta data about the proxy and sending it to prometheus. That was just a benefit.

[yavuzyurtbegendi]

@clanstyles I have made it work yesterday with squid but forgot to write it here, I will also try goproxy, the benefit looks good, thanks

[mvdan]

Thanks all for the information. There's something we might be able to do here for v2, perhaps something similar to what Puppeteer does.

[tmm1]

The simplest way to pass proxy info to chrome (including auth) seems to be via standard environment variables. Is there a simple way to set environment variables for chromedp.ExecAllocator?

[kenshaw]

@tmm1 the way that the ExecAllocator works (it's just a wrapper around Go's standard os/exec package, and is a mostly self-contained version of the previous, separated chromedp/runner package [mentioned earlier]) is that child processes inherit the parent's environment variables.

If Chrome can pick up environment variables, then whatever built application you're using chromedp in, should just pass on the environment variables to any launched process.

For what it's worth, someone submitted an example using the ProxyServer flag. I have not had time to verify that, as I am currently refactoring the way the ExecAllocator and other internals work, to make it easier to, for example, set the proxy server flag. In the interim, try the example and let me know if that works for your use case.

[tmm1]

For what it's worth, someone submitted an example using the ProxyServer flag. I have not had time to verify that, as I am currently refactoring the way the ExecAllocator and other internals work, to make it easier to, for example, set the proxy server flag. In the interim, try the example and let me know if that works for your use case.

The problem is the --proxy-server cli flag cannot accept authentication credentials. According to the chrome team, this is a feature: https://bugs.chromium.org/p/chromium/issues/detail?id=615947

If Chrome can pick up environment variables, then whatever built application you're using chromedp in, should just pass on the environment variables to any launched process.

I don't want to pollute the environment of my application and accidentally pass proxy variables to other sub-processes I may be invoking. So ideally I could tell chromedp to pass specific environment variables via cmd.Env before it calls cmd.Start() here: https://github.com/chromedp/chromedp/blob/8798fb24e2e7e057cab32d992b77fbe03727e9a0/allocate.go#L175

[kenshaw]

@tmm1 it wouldn't make any sense to, eg, enable this for chromedp. If your concern is that environment variables can be read via, eg, /proc, this is also true for a launched child process.

Also, note you can manually set environment variables using os.Setenv with Go.

[tmm1]

I don't care about exposing the variables. My issue is that I _only_ want chrome to use the proxy. If I use os.Setenv, that's going to affect my application as well and I don't want net/http to start using the proxy or for other programs I might be executing to use the proxy. I only want chrome to use it.

I think what may make sense is a new chromedp.Env(key, value string) that's similar to chromedp.Flag(), but adds to chrome's environment instead of cli flags.

[tmm1]

Stated another way, os.Setenv alters global state and is inherently racy. So if I have two goroutines launching two instances of chrome simultaneously, I cannot ensure that one uses a specific proxy and the other does not.

[kenshaw]

@tmm1 clearly you're trying to use multiple proxies, such as multiple TOR socks proxies simultaneously. I can appreciate what you're doing, but it is not likely that regular users of chromedp would ever need to have multiple instances of chromedp running in the same app build using two different proxies at the same time.

I'm willing to review a PR that enables this, but this is the absolutely the least important thing on the chromedp backlog.

[tmm1]

Totally understood. It's quite an obscure use-case and not a priority for most users.

I will send a PR- appreciate your time in discussing tradeoffs and helping review!

FWIW I think the chrome team's argument here is pretty weak- it's just as easy to cat /proc/pid/cmdline as it is to cat /proc/pid/environ and so allowing auth variables via one but not the other doesn't make a lot of sense.

[DeadNumbers]

@tmm1

The simplest way to pass proxy info to chrome (including auth) seems to be via standard environment variables. Is there a simple way to set environment variables for chromedp.ExecAllocator?

There is a reason you cannot build a proxy string from ENV?

login  := os.Getenv("PROXY_LOGIN")
password := os.Getenv("PROXY_PASSWORD")
proxy_addr := os.Getenv("PROXY_ADDR") //127.0.0.1:1080
schema := os.Getenv("PROXY_TYPE") //socks5:// or http://
chromedp.ProxyServer(fmt.Sprintf("%s%s:%s@%s", schema,login,password,proxy_addr))

[tmm1]

There is a reason you cannot build a proxy string from ENV?

This does not work. See my earlier comment:

The problem is the --proxy-server cli flag cannot accept authentication credentials. According to the chrome team, this is a feature: https://bugs.chromium.org/p/chromium/issues/detail?id=615947

[mvdan]

With @tmm1's PR being merged, you can now set environment variables on an ExecAllocator. Is that enough to close this issue?

[nylin93]

This does not seem to work for me. Maybe I'm missing something:

    opts := append(
        chromedp.DefaultExecAllocatorOptions[:], 
        chromedp.Flag("headless", false),
        chromedp.Env("http_proxy='<user>:<password>@<host>:<port>'"),
        chromedp.Env("HTTP_PROXY='<user>:<password>@<host>:<port>'"),
        chromedp.Env("https_proxy='<user>:<password>@<host>:<port>'"),
        chromedp.Env("HTTPS_PROXY='<user>:<password>@<host>:<port>'"),
    )

    ctx, _ := chromedp.NewExecAllocator(
        context.Background(),
        opts...
    )

    ctx, _ = chromedp.NewContext(ctx)

    err := chromedp.Run(ctx,
        chromedp.Navigate("http://ipinfo.io"),
    )

    if err != nil {
        log.Fatal(err)
    }

When I look at what ipinfo shows, it is the same IP address from my home wifi. I would expect to see the ip address of the proxy instead.

[gadelkareem]

I managed to forward the http request to the proxy using the username and pass but I am not sure if https is possible. Any ideas?

import (
    "context"
    "github.com/chromedp/chromedp"
    "github.com/cssivision/reverseproxy"
    "github.com/gadelkareem/go-helpers"
    "net/http"
)

func main(){
proxyurl := h.ParseUrl("http://user:[email protected]")
pxy := &reverseproxy.ReverseProxy{
        Transport: &http.Transport{Proxy: http.ProxyURL(proxyurl),},
        Director: func(req *http.Request) {
            req.URL = proxyurl
        },
    }
    go func() {
        err := http.ListenAndServe("localhost:9000", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if r.Method == "CONNECT" {
                pxy.ProxyHTTPS(w, r)
            } else {
                pxy.ProxyHTTP(w, r)
            }
        }))
        h.PanicOnError(err)
    }()

err := chromedp.Run(ctx,
              chromedp.ProxyServer("http://localhost:9000"),
        chromedp.Navigate("http://ipinfo.io"),
    )
}

@clanstyles any ideas?

[gadelkareem]

Same problem with github.com/elazarl/goproxy. The proxy is used only with http.

    pxy := goproxy.NewProxyHttpServer()
    pxy.Tr.Proxy = http.ProxyURL(proxyurl)

    go func() {
        err := http.ListenAndServe("localhost:9000", pxy)
        h.LogOnError(err)
    }()

[bostin]

This answer is working for me.
https://github.com/chromedp/chromedp/issues/645#issuecomment-704764992

[rcmonitor]

I managed to forward

err := chromedp.Run(ctx,
              chromedp.ProxyServer("http://localhost:9000"),
      chromedp.Navigate("http://ipinfo.io"),
  )
}

With this usage attempt you should be facing an error:
Cannot use 'chromedp.ProxyServer("http://localhost:9000")' (type ExecAllocatorOption) as type Action Type does not implement 'Action' as some methods are missing: Do(context.Context) error