Php-webdriver: Proxy auth fails from example page

[fissben]

Bug description

Proxy with auth not working. Used this page as example https://github.com/php-webdriver/php-webdriver/wiki/HowTo-Work-with-proxy

Selenium constantly logging error like this:
Response code 400. Message: httpProxy is not of the form host[:port]: cleared_user:cleared_password@cleared_ip:cleared_port

Which is fair enough according to java doc https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/Proxy.html#setHttpProxy(java.lang.String)

How could the issue be reproduced

Steps to reproduce the behavior:

1. add WebDriverCapabilityType::PROXY caps according to example page

namespace Facebook\WebDriver;

use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\WebDriverCapabilityType;

require_once('vendor/autoload.php');

$host = 'http://localhost:4444/wd/hub';
$proxyUrl = 'user:pass@proxy_host:proxy_port';

$caps = DesiredCapabilities::firefox();
$caps->setCapability(WebDriverCapabilityType::PROXY, [
    'proxyType' => 'manual',
    'httpProxy' => $proxyUrl,
    'sslProxy' => $proxyUrl,
]);

$driver = RemoteWebDriver::create($host, $caps);
$driver->get("https://2ip.ru");
sleep(2);
$driver->takeScreenshot("2ip.jpeg");


$driver->close();
$driver->quit();

<!--no html-->

Expected behavior

WebDriver should connect to site 2ip.ru through proxy

Details

• Php-webdriver version: 1.10
• PHP version: 7.3.25
• How do you start the browser: docker-hub+firefox node
• Selenium server version: 4.0.0-beta-3
• Operating system: MacOS with docker
• Browser used + version: Firefox
• Browser driver (firefox) version: n/a

Additional context

docker-compose.yml

version: "3"
services:
  firefox:
    image: selenium/node-firefox:4.0.0-beta-3-prerelease-20210402
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
  selenium-hub:
    image: selenium/hub:4.0.0-beta-3-prerelease-20210402
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

[OndraM]

Hi, thanks for the report.

I can confirm this behavior in Firefox (geckodriver). Can you try using Chrome?

[fissben]

In chrome it pass thru, but chrome not using it at all(request performing from my external ip)
Here what I can see in the logs:

 Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.114, chrome: {chromedriverVersion: 89.0.4389.23 (61b08ee2c5002..., userDataDir: /tmp/.com.google.Chrome.4C21sg}, goog:chromeOptions: {debuggerAddress: localhost:37791}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: {httpProxy: user:pass@hostname.., proxyType: MANUAL, sslProxy:  user:pass@hostname..}, 

[OndraM]

So it does work with chrome, right?

[fissben]

Depends on what do mean by "works". Selenium chrome-node getting it from php-webdriver, but not using this proxy config at all - all request performing directly from my external ip without proxy. I'm not really shure about where the problem is - misconfiguration of WebDriverCapabilityType or proxy not working with chrome at all.

[OndraM]

I reported the Firefox issue here: https://github.com/mozilla/geckodriver/issues/1872 - it seems it simply does not accept the credentials in URL.

About chrome, I will have a look later, can you provide the configuration you use for Chrome?

[fissben]

I reported the Firefox issue here: mozilla/geckodriver#1872 - it seems it simply does not accept the credentials in URL.

About chrome, I will have a look later, can you provide the configuration you use for Chrome?

Thnx!

Sure, here chrome config :

namespace Facebook\WebDriver;

use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\WebDriverCapabilityType;

require_once('vendor/autoload.php');

$host = 'http://localhost:4444/wd/hub';
$proxyUrl = "cleared:cleared@host:port";

$caps = DesiredCapabilities::chrome();
$caps->setCapability(WebDriverCapabilityType::PROXY,
    [
        'proxyType' => 'manual',
        'httpProxy' => $proxyUrl,
        'sslProxy' => $proxyUrl,

    ]);

$driver = RemoteWebDriver::create($host, $caps);
$driver->get("http://2ip.ru");
sleep(2);
$driver->takeScreenshot("2ip.jpeg");


$driver->close();
$driver->quit();

[OndraM]

From my research, it seems this is not implemented in Chrome either :thinking: https://bugs.chromium.org/p/chromium/issues/detail?id=615947

However this is not an issue of php-webdriver but limitation of the browsers, so I'm closing this issue - there is nothing we can do on our side.

For a workaround, you can for example run some local unauthenticated proxy, which will pass the requests to the remote proxy requiring authentication. Some examples for further research:

• https://docs.mitmproxy.org/stable/concepts-modes/#upstream-proxy
• https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212
• https://www.npmjs.com/package/proxy-chain

[OndraM]

For the reference, related Firefox issue for https://github.com/mozilla/geckodriver/issues/1872 is here: https://bugzilla.mozilla.org/show_bug.cgi?id=1395886

[OndraM]

I updated the wiki page.

Example with mitmproxy how to start local proxy (without authentication) which forwards request to remote proxy with credentials:

mitmproxy --mode upstream:https://hostname:port --upstream-auth username:password

and use localhost:8080 as proxy in capabilities.