Mitmproxy: Change upstream HTTPs proxy: HTTP2 layer should already have a connection

Created on 10 Sep 2017  路  3Comments  路  Source: mitmproxy/mitmproxy
Steps to reproduce the problem:

With 2 HTTPs proxies setup with https://tinyproxy.github.io/

  1. Use the code from here: https://github.com/mitmproxy/mitmproxy/blob/master/examples/complex/change_upstream_proxy.py
  2. Use mitmdump -Uhttp://default-upstream-proxy.local:8080/ -s change_upstream_proxy.py`

It works fine on https://whatismyipaddress.com/ but it errors out on https://whoer.net/ and https://www.bing.com/

System information

Mitmproxy version: 2.0.2 (release version)
Python version: 3.5.3
Platform: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-17.04-zesty
SSL version: OpenSSL 1.0.2g 1 Mar 2016
Linux distro: Ubuntu 17.04 zesty

stdout:

127.0.0.1:57574: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57574: clientdisconnect
127.0.0.1:57548: HTTP/2 connection terminated by server: error code: 0, last stream id: 1, additional data: None
127.0.0.1:57548: clientdisconnect
127.0.0.1:57486: clientdisconnect
127.0.0.1:57562: clientdisconnect
127.0.0.1:57558: clientdisconnect
127.0.0.1:57552: clientdisconnect
127.0.0.1:57556: clientdisconnect
127.0.0.1:57526: HTTP/2 connection terminated by client: error code: 0, last stream id: 0, additional data: None
127.0.0.1:57560: clientdisconnect
127.0.0.1:57526: clientdisconnect
127.0.0.1:57554: clientdisconnect
127.0.0.1:57532: HTTP/2 connection terminated by client: error code: 0, last stream id: 0, additional data: None
127.0.0.1:57532: clientdisconnect
127.0.0.1:57534: HTTP/2 connection terminated by client: error code: 0, last stream id: 0, additional data: None
127.0.0.1:57534: clientdisconnect
127.0.0.1:57488: HTTP/2 connection terminated by client: error code: 0, last stream id: 0, additional data: None
127.0.0.1:57488: clientdisconnect
picture zero-master

Most helpful comment

If this is happening on domains you don't need to go trough mitmproxy, you can add this in your ~/.mitmproxy/config.yaml:

http2: false

It will disable HTTP2 and thus get rid of all the random domains failing.

Hoping #1775 can fix this at some point.

picture localhostdotdev on 12 Mar 2019
❤2 🎉1

All 3 comments

This might as well help:

Summary

I was able open https://www.google.de but when i clicked the second search page page, it again threw same error.

Surprisingly, I am able to use github through this setup and navigating between its page is not resulting in this error.

Command: mitmdump -U http://proxy-2:9999/ -s upstream.py

Loading script: upstream.py
Proxy server listening at http://0.0.0.0:8080
127.0.0.1:57698: clientconnect
Using proxy-2:9999
127.0.0.1:57698: GET https://www.bing.com/search?q=wow&go=Submit+Query&qs=bs&form=QBLH HTTP/2.0
              << 200  40.21k
Using proxy-1:9999
127.0.0.1:57698: GET https://www.bing.com/sa/simg/SharedSpriteDesktopTwoToneLogoTealSpy_0817.png HTTP/2.0
 << HTTP2 layer should already have a connection.
127.0.0.1:57698: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57698: clientdisconnect
127.0.0.1:57700: clientconnect
Using proxy-2:9999
Using proxy-1:9999
127.0.0.1:57700: GET https://www.bing.com/search?q=wow&go=Submit+Query&qs=bs&first=9&FORM=PERE HTTP/2.0
 << HTTP2 layer should already have a connection.
127.0.0.1:57700: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57700: clientdisconnect
127.0.0.1:57702: clientconnect
127.0.0.1:57704: clientconnect
127.0.0.1:57702: clientdisconnect
127.0.0.1:57706: clientconnect
127.0.0.1:57704: Client Handshake failed. The client may not trust the proxy's certificate for www.google.com.
127.0.0.1:57704: clientdisconnect
Using proxy-1:9999
127.0.0.1:57706: AttributeError("'NoneType' object has no attribute 'write'",)
127.0.0.1:57706: clientdisconnect
127.0.0.1:57708: clientconnect
127.0.0.1:57710: clientconnect
127.0.0.1:57712: clientconnect
127.0.0.1:57714: clientconnect
127.0.0.1:57708: clientdisconnect
127.0.0.1:57710: clientdisconnect
127.0.0.1:57712: clientdisconnect
127.0.0.1:57714: clientdisconnect
127.0.0.1:57716: clientconnect
Using proxy-1:9999
127.0.0.1:57716: clientdisconnect
127.0.0.1:57718: clientconnect
127.0.0.1:57720: clientconnect
127.0.0.1:57722: clientconnect
127.0.0.1:57718: clientdisconnect
127.0.0.1:57720: clientdisconnect
127.0.0.1:57724: clientconnect
127.0.0.1:57722: clientdisconnect
Using proxy-2:9999
127.0.0.1:57724: GET https://whatismyipaddress.com/
              << 200 OK 46.28k
127.0.0.1:57726: clientconnect
127.0.0.1:57728: clientconnect
127.0.0.1:57730: clientconnect
Using proxy-2:9999
Using proxy-1:9999
127.0.0.1:57728: clientdisconnect
Using proxy-1:9999
127.0.0.1:57730: GET https://api.optnmstr.com/v1/optins/site/29382/58bf18cdc87b0 HTTP/2.0
 << HTTP2 layer should already have a connection.
127.0.0.1:57730: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57730: clientdisconnect
127.0.0.1:57726: GET https://apis.google.com/js/platform.js HTTP/2.0
              << 200  16.87k
127.0.0.1:57732: clientconnect
Using proxy-2:9999
127.0.0.1:57732: GET https://cdn.whatismyipaddress.info/images/marker_sq.png HTTP/2.0
              << 200  272b
127.0.0.1:57736: clientconnect
127.0.0.1:57738: clientconnect
127.0.0.1:57740: clientconnect
127.0.0.1:57742: clientconnect
127.0.0.1:57736: clientdisconnect
127.0.0.1:57738: clientdisconnect
127.0.0.1:57740: clientdisconnect
127.0.0.1:57744: clientconnect
127.0.0.1:57742: Client Handshake failed. The client may not trust the proxy's certificate for www.google.com.
127.0.0.1:57742: clientdisconnect
Using proxy-1:9999
127.0.0.1:57744: GET https://whoer.net/ HTTP/2.0
 << HTTP2 layer should already have a connection.
127.0.0.1:57744: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57744: clientdisconnect
127.0.0.1:57746: clientconnect
127.0.0.1:57748: clientconnect
127.0.0.1:57750: clientconnect
127.0.0.1:57752: clientconnect
127.0.0.1:57754: clientconnect
127.0.0.1:57746: clientdisconnect
127.0.0.1:57748: clientdisconnect
127.0.0.1:57750: clientdisconnect
127.0.0.1:57752: clientdisconnect
127.0.0.1:57754: clientdisconnect
127.0.0.1:57756: clientconnect
Using proxy-1:9999
127.0.0.1:57756: GET https://github.com/
              << 200 OK 15.3k
127.0.0.1:57758: clientconnect
127.0.0.1:57760: clientconnect
127.0.0.1:57762: clientconnect
127.0.0.1:57764: clientconnect
127.0.0.1:57766: clientconnect
127.0.0.1:57768: clientconnect
Using proxy-1:9999
Using proxy-1:9999
Using proxy-2:9999
Using proxy-1:9999
Using proxy-1:9999
Using proxy-2:9999
127.0.0.1:57762: GET https://assets-cdn.github.com/assets/site-877643c520258c4fa15ac8d1664d84efd0e3db56f5e544ccac58da0e50489904.css
              << 200 OK 8.17k
127.0.0.1:57768: GET https://assets-cdn.github.com/favicon.ico
              << 200 OK 6.37k
127.0.0.1:57758: GET https://assets-cdn.github.com/assets/frameworks-bedfc518345498ab3204d330c1727cde7e733526a09cd7df6867f6a231565091.css
              << 200 OK 23.18k
127.0.0.1:57760: GET https://assets-cdn.github.com/assets/github-980cd404854e87ee88b12c7281a0875365a8f966c834dc30f3f656011f7f4df6.css
              << 200 OK 93.3k
Using proxy-1:9999
127.0.0.1:57770: clientconnect
127.0.0.1:57772: clientconnect
127.0.0.1:57774: clientconnect
127.0.0.1:57776: clientconnect
127.0.0.1:57778: clientconnect
127.0.0.1:57764: GET https://assets-cdn.github.com/assets/frameworks-d3f8e3cb05c4495afc78c37af32e877f99d47cb06a8158efafa2aebf5567c830.js
              << 200 OK 89.62k
127.0.0.1:57766: GET https://assets-cdn.github.com/assets/github-8a662586ed50ce5d122edec0eca4c2db052b63588ab3fc192b418f261d2d0b53.js
              << 200 OK 183.77k
Using proxy-1:9999
Using proxy-2:9999
Using proxy-2:9999
Using proxy-1:9999
Using proxy-1:9999
127.0.0.1:57768: GET https://assets-cdn.github.com/images/modules/site/integrators/zenhubio.png
              << 200 OK 891b
Using proxy-1:9999
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/integrators/atom.png
              << 200 OK 8.76k
Using proxy-2:9999
127.0.0.1:57768: GET https://assets-cdn.github.com/images/modules/site/stories/developers/ariya.png
              << 200 OK 112.05k
Using proxy-2:9999
127.0.0.1:57776: GET https://assets-cdn.github.com/images/modules/site/integrators/codeship.png
              << 200 OK 9.19k
Using proxy-2:9999
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/integrators/travis-ci.png
              << 200 OK 24.39k
Using proxy-2:9999
127.0.0.1:57778: GET https://assets-cdn.github.com/images/modules/site/integrators/codeclimate.png
              << 200 OK 142.54k
Using proxy-1:9999
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/stories/developers/freakboy3742.png
              << 200 OK 121.05k
127.0.0.1:57774: GET https://assets-cdn.github.com/images/modules/site/integrators/circleci.png
              << 200 OK 97.42k
Using proxy-1:9999
Using proxy-1:9999
127.0.0.1:57774: GET https://assets-cdn.github.com/images/modules/site/logos/airbnb-logo.png
              << 200 OK 20.43k
Using proxy-2:9999
127.0.0.1:57776: GET https://assets-cdn.github.com/images/modules/site/stories/developers/kris-nova.png
              << 200 OK 28.19k
Using proxy-1:9999
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/stories/developers/jessfraz.png
              << 200 OK 13.61k
Using proxy-1:9999
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/stories/developers/yyx990803.png
              << 200 OK 33.46k
Using proxy-1:9999
127.0.0.1:57774: GET https://assets-cdn.github.com/images/modules/site/logos/sap-logo.png
              << 200 OK 12.71k
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/heroes/simple-codelines.svg
              << 200 OK 4.46k
Using proxy-1:9999
Using proxy-2:9999
127.0.0.1:57778: GET https://assets-cdn.github.com/images/modules/site/stories/customers/mapbox.png
              << 200 OK 323.99k
127.0.0.1:57776: GET https://assets-cdn.github.com/images/modules/site/logos/ibm-logo.png
              << 200 OK 8.27k
Using proxy-2:9999
Using proxy-2:9999
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/logos/google-logo.png
              << 200 OK 7.21k
Using proxy-1:9999
127.0.0.1:57768: GET https://assets-cdn.github.com/images/modules/site/stories/customers/mailchimp.png
              << 200 OK 369.79k
Using proxy-1:9999
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/logos/bloomberg-logo.png
              << 200 OK 18.88k
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/home-illo-conversation.svg
              << 200 OK 2.37k
Using proxy-2:9999
Using proxy-2:9999
127.0.0.1:57774: GET https://assets-cdn.github.com/images/modules/site/logos/paypal-logo.png
              << 200 OK 24.16k
Using proxy-2:9999
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/home-illo-business.svg
              << 200 OK 2.25k
Using proxy-1:9999
127.0.0.1:57776: GET https://assets-cdn.github.com/images/modules/site/universe-wordmark.png
              << 200 OK 8.93k
Using proxy-1:9999
127.0.0.1:57778: GET https://assets-cdn.github.com/images/modules/site/universe-octoshop.png
              << 200 OK 25.53k
Using proxy-1:9999
127.0.0.1:57768: GET https://assets-cdn.github.com/images/modules/site/home-illo-chaos.svg
              << 200 OK 2.21k
Using proxy-2:9999
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/logos/spotify-logo.png
              << 200 OK 19k
Using proxy-2:9999
127.0.0.1:57774: GET https://assets-cdn.github.com/images/modules/site/logos/swift-logo.png
              << 200 OK 20.2k
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/logos/facebook-logo.png
              << 200 OK 1.44k
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/integrators/slackhq.png
              << 200 OK 56.96k
127.0.0.1:57778: GET https://assets-cdn.github.com/images/modules/site/logos/nasa-logo.png
              << 200 OK 16.79k
127.0.0.1:57776: GET https://assets-cdn.github.com/images/modules/site/logos/node-logo.png
              << 200 OK 20.74k
127.0.0.1:57768: GET https://assets-cdn.github.com/images/modules/site/logos/walmart-logo.png
              << 200 OK 18.9k
Using proxy-1:9999
127.0.0.1:57756: GET https://github.com/business
              << 200 OK 13.77k
Using proxy-1:9999
Using proxy-1:9999
Using proxy-2:9999
Using proxy-1:9999
Using proxy-1:9999
Using proxy-1:9999
Using proxy-1:9999
127.0.0.1:57778: GET https://assets-cdn.github.com/images/modules/site/avatars/hubspot-quote.jpg
              << 200 OK 100.73k
127.0.0.1:57778: Error in HTTP connection: TcpDisconnect("(32, 'EPIPE')",)
127.0.0.1:57778: clientdisconnect
127.0.0.1:57776: GET https://assets-cdn.github.com/images/modules/site/product-illo/phone-auth.png
              << 200 OK 193.93k
127.0.0.1:57776: Error in HTTP connection: TcpDisconnect("(32, 'EPIPE')",)
127.0.0.1:57776: clientdisconnect
127.0.0.1:57756: GET https://github.com/explore
              << 200 OK 22.74k
127.0.0.1:57780: clientconnect
127.0.0.1:57782: clientconnect
127.0.0.1:57786: clientconnect
127.0.0.1:57784: clientconnect
127.0.0.1:57788: clientconnect
127.0.0.1:57790: clientconnect
127.0.0.1:57792: clientconnect
127.0.0.1:57770: GET https://assets-cdn.github.com/images/modules/site/product-illo/img-projects.png
              << 200 OK 45.02k
127.0.0.1:57770: Error in HTTP connection: TcpDisconnect("(32, 'EPIPE')",)
127.0.0.1:57770: clientdisconnect
127.0.0.1:57768: GET https://assets-cdn.github.com/images/modules/site/avatars/mapbox-quote.jpg
              << 200 OK 59.75k
127.0.0.1:57768: Error in HTTP connection: TcpDisconnect("(32, 'EPIPE')",)
127.0.0.1:57768: clientdisconnect
127.0.0.1:57772: GET https://assets-cdn.github.com/images/modules/site/product-illo/img-comment-context.png
              << 200 OK 58.98k
127.0.0.1:57772: Error in HTTP connection: TcpDisconnect("(32, 'EPIPE')",)
127.0.0.1:57772: clientdisconnect
127.0.0.1:57774: GET https://assets-cdn.github.com/images/modules/site/product-illo/img-search-business.svg
              << 200 OK 29.11k
127.0.0.1:57774: Error in HTTP connection: TcpDisconnect("(32, 'EPIPE')",)
127.0.0.1:57774: clientdisconnect
Using proxy-2:9999
Using proxy-1:9999
Using proxy-2:9999
Using proxy-1:9999
Using proxy-1:9999
Using proxy-2:9999
Using proxy-2:9999
127.0.0.1:57786: GET https://avatars0.githubusercontent.com/u/16903644?v=4&s=80
              << 200 OK 1.82k
127.0.0.1:57788: GET https://avatars3.githubusercontent.com/u/10217535?v=4&s=80
              << 200 OK 1.58k
127.0.0.1:57784: GET https://avatars3.githubusercontent.com/u/674766?v=4&s=80
              << 200 OK 2.72k
127.0.0.1:57792: GET https://assets-cdn.github.com/images/modules/explore/following_stars.v1.png
              << 200 OK 112.54k
127.0.0.1:57780: GET https://avatars3.githubusercontent.com/u/1062491?v=4&s=80
              << 200 OK 1.72k
127.0.0.1:57782: GET https://avatars2.githubusercontent.com/u/579?v=4&s=80
              << 200 OK 7.1k
127.0.0.1:57790: GET https://avatars1.githubusercontent.com/u/5747538?v=4&s=80
              << 200 OK 1.78k
127.0.0.1:57794: clientconnect
127.0.0.1:57796: clientconnect
127.0.0.1:57794: clientdisconnect
127.0.0.1:57796: clientdisconnect
127.0.0.1:57798: clientconnect
Using proxy-1:9999
127.0.0.1:57798: GET http://google.com/
              << 302 Found 268b
127.0.0.1:57798: clientdisconnect
127.0.0.1:57802: clientconnect
Using proxy-1:9999
127.0.0.1:57802: GET http://www.google.de/?gfe_rd=cr&dcr=0&ei=eUu1Wf_fKYyr8wffypCADw
              << 302 Found 284b
127.0.0.1:57802: clientdisconnect
127.0.0.1:57804: clientconnect
Using proxy-2:9999
127.0.0.1:57804: GET https://www.google.de/?gfe_rd=cr&dcr=0&ei=eUu1Wf_fKYyr8wffypCADw&gws_rd=ssl HTTP/2.0
              << 200  62.19k
Using proxy-1:9999
Using proxy-2:9999
127.0.0.1:57804: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57804: clientdisconnect
127.0.0.1:57806: clientconnect
127.0.0.1:57808: clientconnect
127.0.0.1:57810: clientconnect
Using proxy-2:9999
Using proxy-1:9999
127.0.0.1:57808: clientdisconnect
Using proxy-1:9999
127.0.0.1:57810: GET https://www.google.de/xjs/_/js/k=xjs.s.de.bjSZvqjVlAQ.O/m=sx,c,sb,cdos,cr,elog,hsm,jsa,r,qsm,d,csi/am=wCKMHC84APJ_QigMbiEskAVMHEA/rt=j/d=1/t=zcms/rs=ACT90oG-VbbImgwlgV8QJBJcWpV2HrOuGg HTTP/2.0
 << HTTP2 layer should already have a connection.
127.0.0.1:57810: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57810: clientdisconnect
127.0.0.1:57806: GET https://consent.google.com/status?continue=https://www.google.de&pc=s&timestamp=1505053564 HTTP/2.0
              << 204  0b
127.0.0.1:57812: clientconnect
Using proxy-1:9999
127.0.0.1:57812: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57812: clientdisconnect
127.0.0.1:57814: clientconnect
Using proxy-1:9999
127.0.0.1:57814: GET https://www.google.de/search?dcr=0&source=hp&q=wow&btnK=Google-Suche HTTP/2.0
 << HTTP2 layer should already have a connection.
127.0.0.1:57814: ValueError('file descriptor cannot be a negative integer (-1)',)
127.0.0.1:57814: clientdisconnect
zero-master picture zero-master on 10 Sep 2017

Thanks for the report. We currently don't support changing the host for HTTP2 connections, sorry. This may be fixed by #1775 ultimately. It's already listed there, so I'll close this issue if you don't mind.

mhils picture mhils on 11 Sep 2017

If this is happening on domains you don't need to go trough mitmproxy, you can add this in your ~/.mitmproxy/config.yaml:

http2: false

It will disable HTTP2 and thus get rid of all the random domains failing.

Hoping #1775 can fix this at some point.

localhostdotdev picture localhostdotdev on 12 Mar 2019
❤2 🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

prastut picture prastut  路  5Comments
Dauth picture Dauth  路  3Comments
mhils picture mhils  路  4Comments
nesttle picture nesttle  路  4Comments
mhils picture mhils  路  4Comments