Requests: Cookie is not set in requests.Session when "Set-Cookie" header is received

[dmfigol]

requests.Session does not honor Set-Cookie header

Expected Result

The cookie should be set for the session.

Actual Result

Cookie is not set.

Reproduction Steps

session = requests.Session()
r = session.get(AUTH_URL, auth=(USERNAME,PASSWORD), verify=False)
print(r.status_code)
print(r.headers)
print(r.content)
print(session.cookies, r.cookies)
r = session.get(GROUP_URL, verify=False)
print(r.content)
print(r.status_code)

########## OUTPUT #########
200
{'Date': 'Mon, 04 Dec 2017 00:30:40 GMT', 'Content-Type': 'application/json', 'Content-Length': '7', 'Connection': 'keep-alive', 'Set-Cookie': 'X-JWT-ACCESS-TOKEN=<long-token>;Version=-1;Comment=;Domain=;Path=/;Secure', 'X-Kong-Upstream-Latency': '200', 'X-Kong-Proxy-Latency': '31', 'Via': 'kong/0.9.7'}
b'success'
<RequestsCookieJar[]> <RequestsCookieJar[]> ##### << here, cookies are NOT set
b'{"message":"Unauthorized"}\n'
401

Additional Information

I see a bug in API: 'Content-Type' for the response is 'application/json', but the response itself is not a valid JSON (b'success'). I am going to contact API developers to fix this. I am not 100% sure it is related to my problem, but I thought you should know.

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.3"
  },
  "platform": {
    "release": "7",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

[sigmavirus24]

This is what you've provided as the value of Set-Cookie: X-JWT-ACCESS-TOKEN=<long-token>;Version=-1;Comment=;Domain=;Path=/;Secure. Judging by that value, it appears that there are a few things wrong with this string (presuming <long-token> is the only editing you did).

The header in this case is defined as 1 or more av-pairs where cookiename=cookievalue is the only required one. In this case, I presume the server intends for X-JWT-ACCESS-TOKEN to be the cookie name. With that in mind, everything afterward is optional. Further, an av-pair is defined as: attr ["=" value] where attr is the only required portion of this. That means if no value is provided they probably shouldn't send = but that probably parses fine... except that they send an empty Domain. If they don't specify their own domain attribute, I believe that we force it to be the host of the URL that was requested when we made the request. However, they specify it as empty. I think that's what is tripping up the standard library here.

[dmfigol]

@sigmavirus24 thank you for your comments. Yes, I only replaced the token with <long-token>. I agree with you, that this API returns some strange values and I am going to contact API developers to get this fixed. Is there any way to check if the issue is indeed caused by an empty Domain av-pair and fix if needed? Browsers and Postman are able to set this cookie successfully.

[kennethreitz42]

Closing due to inactivity — thank you very much for your patience ✨🍰✨

[snowe2010]

I'm seeing the exact same thing, except I have provided the domain. Here are the headers returned with the jwt token shortened.

{'X-Application-Context': 'application:8080', 'correlation-id': '66279bda-8d55-4d80-80e9-7dd105c8daaf', 'Set-Cookie': 'X-AUTH-TOKEN=eyJ0eX.eyJzdWIiOiJ7XCJ0ZW.g-ZVOQ; Max-Age=43200; Expires=Sun, 06-May-2018 10:53:21 GMT; Domain=localhost; Path=/; HttpOnly, X-REFRESH-TOKEN=eyJiJ9.eyJzdWIiOiIxMTExMTExQ.4wg; Max-Age=43200; Expires=Sun, 06-May-2018 10:53:21 GMT; Domain=localhost; Path=/; HttpOnly', 'Date': 'Sat, 05 May 2018 22:53:21 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked'}

and the cookies are coming back. <RequestsCookieJar[]>

[remingtonc]

I too am having this issue. Latest requests. Python 3.6.

# Session cookies
INFO:root:<RequestsCookieJar[]>
# Response cookies
INFO:root:<RequestsCookieJar[]>
# Response headers
INFO:root:{'Content-Type': 'application/json', 'Content-Length': '7', 'Connection': 'keep-alive', 'Date': 'Fri, 01 Jun 2018 21:57:17 GMT', 'Set-Cookie': 'X-JWT-ACCESS-TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YTU4Y2QzN2UwNWJiYTAwOGVmNjJiOTIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbGVzIjpbIjVhMzE1MTYwOTA5MGZiYTY5OGIyZjViNyJdLCJ0ZW5hbnRJZCI6IjVhMzE1MTlkZTA1YmJhMDA4ZWY2MWYwYSIsImV4cCI6MTUyNzg5MzgzNywidXNlcm5hbWUiOiJkZXZuZXR1c2VyIn0.VeP442Qt7iyAnQTNISQ8tuXzf1fWqAdDMdKrBJQsims;Version=1;Comment=;Domain=;Path=/;Max-Age=3600;Secure;HttpOnly', 'X-Kong-Upstream-Latency': '292', 'X-Kong-Proxy-Latency': '4', 'Via': 'kong/0.11.0', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'}

It does work via Postman.

Debug logs:

DEBUG:http.cookiejar:add_cookie_header
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): X
/root/.local/share/virtualenvs/data-I7nS9QO2/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
DEBUG:urllib3.connectionpool:X "GET X HTTP/1.1" 200 7
DEBUG:http.cookiejar:extract_cookies: Content-Type: application/json
Content-Length: 7
Connection: keep-alive
Date: Fri, 01 Jun 2018 22:48:34 GMT
Set-Cookie: X-JWT-ACCESS-TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YTU4Y2QzN2UwNWJiYTAwOGVmNjJiOTIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbGVzIjpbIjVhMzE1MTYwOTA5MGZiYTY5OGIyZjViNyJdLCJ0ZW5hbnRJZCI6IjVhMzE1MTlkZTA1YmJhMDA4ZWY2MWYwYSIsImV4cCI6MTUyNzg5NjkxNCwidXNlcm5hbWUiOiJkZXZuZXR1c2VyIn0.J5vECtvkjYc9Zwu85R51Lu_3Z4jv5buf8JuKKwXS2pw;Version=1;Comment=;Domain=;Path=/;Max-Age=3600;Secure;HttpOnly
X-Kong-Upstream-Latency: 131
X-Kong-Proxy-Latency: 4
Via: kong/0.11.0
Strict-Transport-Security: max-age=31536000; includeSubDomains


DEBUG:http.cookiejar: - checking cookie X-JWT-ACCESS-TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YTU4Y2QzN2UwNWJiYTAwOGVmNjJiOTIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbGVzIjpbIjVhMzE1MTYwOTA5MGZiYTY5OGIyZjViNyJdLCJ0ZW5hbnRJZCI6IjVhMzE1MTlkZTA1YmJhMDA4ZWY2MWYwYSIsImV4cCI6MTUyNzg5NjkxNCwidXNlcm5hbWUiOiJkZXZuZXR1c2VyIn0.J5vECtvkjYc9Zwu85R51Lu_3Z4jv5buf8JuKKwXS2pw
DEBUG:http.cookiejar:   non-local domain . contains no embedded dot
DEBUG:http.cookiejar:extract_cookies: Content-Type: application/json
Content-Length: 7
Connection: keep-alive
Date: Fri, 01 Jun 2018 22:48:34 GMT
Set-Cookie: X-JWT-ACCESS-TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YTU4Y2QzN2UwNWJiYTAwOGVmNjJiOTIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbGVzIjpbIjVhMzE1MTYwOTA5MGZiYTY5OGIyZjViNyJdLCJ0ZW5hbnRJZCI6IjVhMzE1MTlkZTA1YmJhMDA4ZWY2MWYwYSIsImV4cCI6MTUyNzg5NjkxNCwidXNlcm5hbWUiOiJkZXZuZXR1c2VyIn0.J5vECtvkjYc9Zwu85R51Lu_3Z4jv5buf8JuKKwXS2pw;Version=1;Comment=;Domain=;Path=/;Max-Age=3600;Secure;HttpOnly
X-Kong-Upstream-Latency: 131
X-Kong-Proxy-Latency: 4
Via: kong/0.11.0
Strict-Transport-Security: max-age=31536000; includeSubDomains


DEBUG:http.cookiejar: - checking cookie X-JWT-ACCESS-TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YTU4Y2QzN2UwNWJiYTAwOGVmNjJiOTIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbGVzIjpbIjVhMzE1MTYwOTA5MGZiYTY5OGIyZjViNyJdLCJ0ZW5hbnRJZCI6IjVhMzE1MTlkZTA1YmJhMDA4ZWY2MWYwYSIsImV4cCI6MTUyNzg5NjkxNCwidXNlcm5hbWUiOiJkZXZuZXR1c2VyIn0.J5vECtvkjYc9Zwu85R51Lu_3Z4jv5buf8JuKKwXS2pw
DEBUG:http.cookiejar:   non-local domain . contains no embedded dot
INFO:root:<RequestsCookieJar[]>
INFO:root:<RequestsCookieJar[]>
INFO:root:{'Content-Type': 'application/json', 'Content-Length': '7', 'Connection': 'keep-alive', 'Date': 'Fri, 01 Jun 2018 22:48:34 GMT', 'Set-Cookie': 'X-JWT-ACCESS-TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YTU4Y2QzN2UwNWJiYTAwOGVmNjJiOTIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbGVzIjpbIjVhMzE1MTYwOTA5MGZiYTY5OGIyZjViNyJdLCJ0ZW5hbnRJZCI6IjVhMzE1MTlkZTA1YmJhMDA4ZWY2MWYwYSIsImV4cCI6MTUyNzg5NjkxNCwidXNlcm5hbWUiOiJkZXZuZXR1c2VyIn0.J5vECtvkjYc9Zwu85R51Lu_3Z4jv5buf8JuKKwXS2pw;Version=1;Comment=;Domain=;Path=/;Max-Age=3600;Secure;HttpOnly', 'X-Kong-Upstream-Latency': '131', 'X-Kong-Proxy-Latency': '4', 'Via': 'kong/0.11.0', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'}
INFO:root:success
DEBUG:http.cookiejar:add_cookie_header
/root/.local/share/virtualenvs/data-I7nS9QO2/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
DEBUG:urllib3.connectionpool:X "GET X HTTP/1.1" 401 None
DEBUG:http.cookiejar:extract_cookies: Date: Fri, 01 Jun 2018 22:48:34 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: kong/0.11.0
Strict-Transport-Security: max-age=31536000; includeSubDomains


DEBUG:http.cookiejar:extract_cookies: Date: Fri, 01 Jun 2018 22:48:34 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: kong/0.11.0
Strict-Transport-Security: max-age=31536000; includeSubDomains

The Domain appears to be failing this condition.
https://github.com/python/cpython/blob/3.6/Lib/http/cookiejar.py#L1033

It's detected as not Absent despite a falsey value..what should the behavior be?
https://github.com/python/cpython/blob/3.6/Lib/http/cookiejar.py#L1504

@sigmavirus24 Given you seem to have some understanding here - is an empty Domain value considered invalid?

EDIT: Looking at the RFC, an empty Domain value looks to be not completely defined. Is not providing a value the same as omitting the attribute, in which case it is still valid and should only be useful for the origin server? Or is it invalid and should be dropped?

EDIT EDIT: According to the RFC syntax specification, a domain-value being empty is invalid. This is a bug from the API.

[Audace]

Experiencing the same issue. The Set-Cookie header successfully leads to the cookie's presence in response.cookies but does not lead to a cookie in the session's cookie jar.

Header is the following:

{
  "Set-Cookie": "foo=bar; "
}

Response.cookies is the following:

<RequestsCookieJar[<Cookie foo=bar for />]>

Session.cookies is the following:

<RequestsCookieJar[]>

Version: 2.19.1

[nateprewitt]

Hi @Audace, it doesn't look like this is the same issue. The above is discussing cookies with empty domain values.

For your issue, we'll need more info and it should probably be opened as it's own ticket. A quick POC shows this should be working as expected though, so I'd suggest double checking your code first. Thanks!

import requests

s = Requests.Session()
resp = s.get('https://httpbin.org/cookies/set?foo=bar')
print(s.cookies)
>>>
<RequestsCookieJar[Cookie(version=0, name='foo', value='bar', port=None, port_specified=False, domain='httpbin.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False)]>

[mathewcohle]

Hi @nateprewitt,

I'm facing the same issue when trying to mock the response. Following example reproduces it:

import requests
import responses

session = requests.Session()
headers = {'Set-Cookie': 'foo=bar'}

with responses.RequestsMock() as rsps:
    rsps.add(
        responses.GET, 'https://google.com', headers=headers)
    r = session.get('https://google.com')


print(r.cookies)
>>> <RequestsCookieJar[<Cookie foo=bar for />]>

print(session.cookies)
>>> <RequestsCookieJar[]>

$ python --version
Python 3.7.0

$ pip freeze 
responses==0.9.0
requests==2.19.1

[nateprewitt]

Hi @mathewcohle, as was said above, this isn't related to this issue. The issue you're describing appears to be related to responses and how it's mocking functionality works. I would suggest following up over there.