Requests: Problem with missing cookies after redirect

[JohnCC330]

I sent this by e-mail - no response. I think this might be of interest to others:

I have a problem when connecting to a site. Here's the scenario:

1) I enter a login page, which has a form
2) I send (using Requests) a POST with the username, pw, etc.
(This POST includes the SESSIONID)
3) The webpage with a 302,
4) To which requests does automatically a GET to the new address
5) In Firefox, this works, In Requests, I get redirected to the
login - page (with another 302).

The only important difference I can detect is that in point 4),
Firefox repeats automatically the SESSION ID, which Requests does
not do. Can I enable this?

I solved the problem by disabling automatic redirects, and creating
a new request manually, with the sessionid cookie. Now the process
runs successfully.

This confirms the necessity of the repeating the cookie in the
request after the 302, but it defeat the 'neatness' of the auto
redirects.

Cheers,
John

[sigmavirus24]

@JohnCC330 how are you passing the SESSIONID cookie? Could you give us a snippet so we can attempt to reproduce it?

[JohnCC330]

On Fri, 01 Mar 2013 15:56:14 -0800
Ian Cordasco [email protected] wrote:

@JohnCC330 how are you passing the SESSIONID cookie? Could you give us a snippet so we can attempt to reproduce it?

Hope this helps... Tell me if you need anything else.

John

            print "\tLog in..."
            params = {"cmd": "login", \
                  "username": dl["user"], \
                  "pwd": dl["pass"]}

            try:
                r = session.post(dl["posturl1"], \
                    headers = {"User-Agent": dl["agent"], \
                               "Referrer": "http://web.xxx.yyy/md.xtp"}, \
                    data = params, \
                    allow_redirects = False) # Have to disable automatic redirects,
                     # because they don't repeat the cookie
            except requests.ConnectionError:
                print "* Can't log in"         
                exit(1);

            redir_url = r.headers["Location"]
                     # By doing a manual get, the cookie gets 
                     # included automatically
            if r.status_code == 302:
                r = session.get(redir_url, \
                            headers = {"User-Agent": dl["agent"], \
                                       "Referrer": "http://web.xxx.yyy/md.xtp"})

[sigmavirus24]

Wow I'm an idiot. You installed this from PyPI right? I think this has already been fixed in master. We really need to do a new release.

[JohnCC330]

On Fri, 01 Mar 2013 19:30:38 -0800
Ian Cordasco [email protected] wrote:

Wow I'm an idiot. You installed this from PyPI right? I think this has already been fixed in master. We really need to do a new release.

I believe I downloaded the source using this link:

https://github.com/kennethreitz/requests/tarball/master

So that would include the change?
John

[sigmavirus24]

Maybe. Especially if you did it within the last week or two.

In that case, I'm going to assume that the SESSION ID cookie is set when you originally do a GET on the login page (unless you're setting it manually which it doesn't seem as if you are). A POST to it might not set it properly at the start. In other words, you might need that cookie set in the header when making the post, otherwise it will prevent you from logging in. However, if the above code works (not letting us handle the redirects) that would seem to disprove my idea.

Maybe @Lukasa has some insight into this. I'm 99% sure the logic for this is correct in resolve_redirects. :/

[JohnCC330]

On Fri, 01 Mar 2013 20:15:04 -0800
Ian Cordasco [email protected] wrote:

In that case, I'm going to assume that the SESSION ID cookie is set when you originally do a GET on the login page (unless you're setting it manually which it doesn't seem as if you are). A POST to it might not set it properly at the start. In other words, you might need that cookie set in the header when making the post, otherwise it will prevent you from logging in. However, if the above code works (not letting us handle the redirects) that would seem to disprove my idea.

Sorry - the cookie _was_ sent correctly in the post - it was only missing from the request after
the redirect. I deleted the capture files from Wireshark,
and the previous versions of the program, so I can't send you the capture or recreate easily.
I'm swamped with work right now (it's 2 AM here...), next week I could probably reconstruct the
scenario and test it again.

I can also re-install the latest version and test it again.

John

[neurostar]

I have same issue with today's github source.
When request redirect base on header['Location'], it does not include cookie.
following code solve the issue temporarily tough.

f = self.reqSession.get(url, allow_redirects=False)
if f.headers['Location']:
f = self.reqSession.get(self.BASE_URL + f.headers['Location'], headers={'Referer' : referer}, cookies=f.cookies)

[miikka]

I tried to reproduce this, but couldn't. I tried both requests 1.1.0 and rev. 13de692 (latest master as of writing) on Python 2.7.2. Here's a Flask server that sets a cookie and returns a 302 redirect:

from flask import Flask, redirect, request, make_response
app = Flask(__name__)

@app.route("/step0")
def step0():
    resp = make_response("step0")
    resp.set_cookie("test_cookie", "step0")
    return resp

@app.route("/step1", methods=["GET", "POST"])
def step1():
    return redirect("/step2")

@app.route("/step2")
def step2():
    return "cookies: {0}".format(request.cookies)

if __name__ == "__main__":
    app.run(debug=True)

Here's what happens:

>>> import requests
>>> s = requests.Session()
>>> s.get("http://localhost:5000/step0")
>>> r = s.post("http://localhost:5000/step1")
>>> print r.text
cookies: {'test_cookie': u'step0'}

[neurostar]

As you see from following, on first request, requests receive cookie and send the cookie with second request correctly. When redirected by base on header['Location'] on third get, the requests send without the cookie.

Try followings:

import requests
reqSession = requests.Session()
response_1 =reqSession.get('http://www.vev.kr/board_YvuS41/33699')
response_2 =reqSession.get('http://www.vev.kr/?>module=file&act=procFileDownload&file_srl=33700&sid=9d1a1c7dd7a4f25556bb364b386216fd')

I got followings from wireshark:

GET /board_YvuS41/33699 HTTP/1.1
Host: www.vev.kr
Accept-Encoding: gzip, deflate, compress
Accept: _/_
User-Agent: python-requests/1.1.0 CPython/2.7.0 Windows/7

HTTP/1.1 200 OK
Date: Fri, 08 Mar 2013 19:03:19 GMT
Server: Apache/2.2.17 (Ubuntu)
Set-Cookie: PHPSESSID=2hqfaqamumnl5g7bb0pslld351; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: mobile=false; path=/
Set-Cookie: user-agent=cb644817ea39140848b0a70d0266418a; path=/
Content-Encoding: gzip
Last-Modified: Fri, 08 Mar 2013 19:03:19 GMT
Cache-Control: post-check=0, pre-check=0
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

273e

GET /?module=file&act=procFileDownload&file_srl=33700&sid=9d1a1c7dd7a4f25556bb364b386216fd HTTP/1.1
Host: www.vev.kr
Cookie: mobile=false; PHPSESSID=2hqfaqamumnl5g7bb0pslld351; user-agent=cb644817ea39140848b0a70d0266418a
Accept-Encoding: gzip, deflate, compress
Accept: _/_
User-Agent: python-requests/1.1.0 CPython/2.7.0 Windows/7

HTTP/1.1 302 Found
Date: Fri, 08 Mar 2013 19:03:20 GMT
Server: Apache/2.2.17 (Ubuntu)
X-Powered-By: PHP/5.3.5-1ubuntu7.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: http://www.vev.kr/index.php?act=procFileOutput&file_srl=33700&file_key=9f5f2b6cd31883919719567b30583537
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 20
Content-Type: text/html

....................GET /index.php?act=procFileOutput&file_srl=33700&file_key=9f5f2b6cd31883919719567b30583537 HTTP/1.1
Host: www.vev.kr
Accept-Encoding: gzip, deflate, compress
Accept: _/_
User-Agent: python-requests/1.1.0 CPython/2.7.0 Windows/7

HTTP/1.1 200 OK
Date: Fri, 08 Mar 2013 19:03:20 GMT
Server: Apache/2.2.17 (Ubuntu)
X-Powered-By: PHP/5.3.5-1ubuntu7.11
Set-Cookie: PHPSESSID=f224o1ju76d45frnv6fadgtes3; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: mobile=false; path=/
Set-Cookie: user-agent=cb644817ea39140848b0a70d0266418a; path=/
Content-Encoding: gzip
Last-Modified: Fri, 08 Mar 2013 19:03:20 GMT
Cache-Control: post-check=0, pre-check=0
Vary: Accept-Encoding
Content-Length: 3391
Content-Type: text/html; charset=UTF-8


[JohnCC330]

On Fri, 08 Mar 2013 10:48:47 -0800
Miikka Koskinen [email protected] wrote:

import requests
s = requests.Session()
s.get("http://localhost:5000/step0")
r = s.post("http://localhost:5000/step1")
print r.text
cookies: {'test_cookie': u'step0'}

Yes, r does receive the cookie. But, as neurostar shows, it does not include
it in the second reply (the one after the 302), if the redirection is automatic.
So the _server_ does not get the cookie (there may be some difference in
servers, but in my case the server requires the cookie in the redirect).

So, do only

import requests
s = requests.Session()
r = s.post("http://localhost:5000/step1")

And follow the code with Wireshark. You'll see that s.post sends the post
_with_ cookie, then gets the 302, then sends a get without the cookie.

John

[miikka]

@neurostar, @JohnCC330: Right, sorry for my confusion. I botched my versions yesterday and only tested using the released version, which works. Git version indeed fails.

[paparomeo]

Any idea when mikka's fix will be merged? At the moment this is the only issue preventing me from migrating to requests 1.x.

[tumb1er]

It seems that cookies are not preserved after redirect in v2.0.0

[Lukasa]

@tumb1er That doesn't seem to be true for me:

>>> import requests
>>> requests.__version__
'2.0.0'
>>> r = requests.get('http://httpbin.org/cookies/set', params={'hi': 'there'})
>>> r.history
(<Response [302]>,)
>>> r.request.url
u'http://httpbin.org/cookies'
>>> r.history[0].url
u'http://httpbin.org/cookies/set?hi=there'
>>> r.history[0].headers['Set-Cookie']
'hi=there; Path=/'
>>> print r.content
{
  "cookies": {
    "hi": "there"
  }
}
>>> print r.request.headers
CaseInsensitiveDict({'Cookie': 'hi=there', 'Accept-Encoding': 'gzip, deflate, co
mpress', 'Accept': '*/*', 'User-Agent': 'python-requests/2.0.0 CPython/2.7.4 Win
dows/7'})

Looks to me like they were set on the 302 and sent back on the request after the redirect.

[tumb1er]

Sorry, similar, but not same bug.

>>> import requests
>>> requests.__version__
'2.0.0'
>>> r = requests.get("http://10.1.23.92/file/036ae155-45a5-443c-967c-ee7f30563c5b", cookies={'fhauth':'cacb87b4f894bc0525ccd3004d3a6dab580acd07'})
>>> r.history
(<Response [301]>,)
>>> r.request.headers
CaseInsensitiveDict({'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/2.0.0 CPython/2.7.5 FreeBSD/9.1-RELEASE-p5'})
# Added / to prevent 301 redirect
>>> r = requests.get("http://10.1.23.92/file/036ae155-45a5-443c-967c-ee7f30563c5b/", cookies={'fhauth':'cacb87b4f894bc0525ccd3004d3a6dab580acd07'})
>>> r.history
[]
>>> r.request.headers
CaseInsensitiveDict({'Cookie': 'fhauth=cacb87b4f894bc0525ccd3004d3a6dab580acd07', 'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/2.0.0 CPython/2.7.5 FreeBSD/9.1-RELEASE-p5'})

[Lukasa]

Ah, good spot. I see the problem in the code. I'll open a new issue for it.

[Lukasa]

New issue: #1630.