Got: POST request does not follow 301/ 302 redirects

[gajus]

It is common for a HTTP POST/ PUT request to get a 3xx response. In this case, the expected behaviour is that the HTTP method is changed to GET and client is redirected to the target destination.

I was looking for a reference to back up this claim in the official specs, but the closest I found is Moz suggesting that in case of permanent redirects (301) "[requests] may or may not be changed to GET."

Currently the following test case is failing:

test('3xx redirect changes the request method to GET if the original request method is not safe to repeat (e.g. POST)', async (t) => {
  nock('http://applaudience.com')
    .post('/')
    .reply(301, 'foo', {
      location: 'http://applaudience.com/foo'
    });

  nock('http://applaudience.com')
    .get('/foo')
    .reply(200, 'bar');

  const response = await got('http://applaudience.com/', {
    method: 'post'
  });

  t.true(response.body === 'bar');
});

I would advocate for following the 3xx redirects using GET at all time, as this the behaviour that all major browsers implement. However, since this is not defined anywhere in the spec (as far as I can tell), the decision could be delegated to a followRedirects option as a boolean function (followRedirects: (request, response) => { ... }).

[gajus]

It appears that this behaviour is implemented for 303 response type:

Note that if a 303 is sent by the server in response to any request type (POST, DELETE, etc.), Got will automatically request the resource pointed to in the location header via GET. This is in accordance with the spec.

Problem is that not all services use the appropriate HTTP status code to signal redirect, esp. when it comes to 301/ 302/ 303.

[brandon93s]

According to RFC 2616, for 301:

If the 301 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.

Note: When automatically redirecting a POST request after
receiving a 301 status code, some existing HTTP/1.0 user agents
will erroneously change it into a GET request.

It goes on the read the following for 302:

Note: RFC 1945 and RFC 2068 specify that the client is not allowed
to change the method on the redirected request.

And finally for 303:

The response to the request can be found under a different URI and
SHOULD be retrieved using a GET method on that resource.

---

301 and 302 are explicit in maintaining the request type. Further, they require user confirmation before attempting the redirect given the implication of retrying non-idempotent requests.

303 is explicit in always performing a GET request.

I believe the current implementation to be RFC compliant and should remain as implemented.

[gajus]

I believe the current implementation to be RFC compliant and should remain as implemented.

Problem is that in practise there are a lot of websites that are not compliant with this behaviour.

Either way, I have solved this by using a wrapper around Got.

I have raised this as a problem to bring it to the attention of the library maintainers.

[krishna1st]

@gajus can you share how have you done it with wrapper

[sa-spag]

Hey there, RFC 2616 is obsoleted by RFC 7231 which states:

Over 10 years later, most user agents still do method rewriting for 301 and 302; therefore, this specification makes that behavior conformant when the original request is POST.

Should we reopen this issue?

[Hoishin]

@brandon93s Would you consider re-open and potentially fix this problem?

[szmarczak]

@sindresorhus I think @sa-spag is right.

[gomes7296]

Quick fix (temporary)

async postUrl(url, form, externalOptions = {}, saveNavigation = true, ) {
    const options = {
      // agent: this.tunnelingAgent,
      headers,
      decompress: true,
      cookieJar: this.cookieJar,
      method: 'POST',
      body: form,
      json: true,
      ...externalOptions,
    };

    let res;
    try {
      res = await got(url, options);
    } catch (error) {
      if (error.statusCode.toString().substring(0, 1) === '3' && error.headers && error.headers.location) {
        res = await (this.getUrl(error.headers.location));
      } else {
        throw error
      }
    }

    if (saveNavigation) {
      this.html = res.body
      this.$ = cheerio.load(this.html)
      this.lastNavigation = url
    }

    return res
  }

Fix solution: (temporary)

    let res;
    try {
      res = await got(url, options);
    } catch (error) {
      if (error.statusCode.toString().substring(0, 1) === '3' && error.headers && error.headers.location) {
        res = await (this.getUrl(error.headers.location));
      } else {
        throw error
      }
    }

[ash0080]

@szmarczak
Not works well even on 10.6.0, can't parse setCookie on redirection even with cookieJar
compare with the needle example

const resp = await needle('post', loginEntryPoint, searchParams.toString(), {
      headers: {
        'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1'
      },
      follow_max: 5,
      parse_cookies: true,
      follow_set_cookies: true,
      follow_set_referer: true
    })
    console.log(resp.cookies) 
// here I get what I want

    const cookieJar = new CookieJar()
    const resp      = await got.post(loginEntryPoint, {
      headers: {
        'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1'
      },
      form: {
        j_username: account.username,
        j_password: account.passwd
      },
      cookieJar,
      followRedirect: true,
      timeout: 10000
    })

//  403 Forbidden