Crystal: HTTP::Server long url 400 Bad Request

[ghost]

Summary

I found that HTTP::Server will appear 400 Bad Request for very long URLs.
Some requests require the use of "extra long characters".
Maybe it's because the request URL is longer than 4096 bytes (I guess).
src/http/request.cr#L100

Example

• curl

curl -v "http://127.0.0.1:8080/list_detail_rate.htm/_____tmd_____/verify/?nc_token=33bffaa55be619b46359e6353d1bd6d9&nc_session_id=01VsUAH0NTPMVtfd7lyQMahoibWdyK1P8oUMd5M98WlcQ84bFi0PnmHETc3rGPtk3VN8HPWMx6QlMWV-Nw9LKhFln6-0AXmw2z9nEi93BnEET98VsjMF0mhWdmdvkUgS9eiw8IygcyecEQzuZucDqgaEY3jNRF8BNoe2JrjpADDmLoEzvs7GjkrYL_EbxDjGJoHvJG0DpdSHZOdyTDPcKc6A&nc_sig=059HQ0XsykTiYwN9Dn38G9XAcFHfXDvRtfFNlgtdIua129sbzmSlbWHFUzKuj3U0IGA3bzvprIaQ2IoO5j4l7TyEi806jDSf_kjePvkLjnFM0XA-ChnD7rcqrnGe1b_Mp-1YxplkweRUguUHMefTFifKoi1GowPa_H7SLr8CzbOCPNs6gWLiqxnwjLVLEgRlHUhzIiGJzKxLpS1LnjuS6F8jnEPs_3D0-lJiwaplLuVDbhA7fg4QFn39t4VWlR50gyAaalQwqubUnJlldueM0Dq6XPfM_6Ndy6_nQAW-3Asvt74RX7LO3w2WYnYgtfI6m2UL8jXC9Yiq4UBV1U7xZ-DVecZNUTkrHO1432KkP1TBgo2o3GOxbowBqfBsLfhdzrze9Jac4FCUrw_wieJX98p1sAxQHojeOF39MSL4bxBD7NQBcMbHWkrAmSui0NXBs_ie1EDml3wzWml9OAPM062sXW_TewgrvFacZBClGCSEA&x5secdata=5e0c8e1365474455070961b803bd560607b52cabf5960afff39b64ce58073f7844a8dd94c89edef01763e7bda2153e72fcab45d62d62d1d0fb9af411e98157f55151e037d30b894fa1b9f0617640df860a901a36bd6d324642091753f1253ec47fad7705c7bf26a7ef1aa109e380a620464f3a20661c6f7fc8f51770469f5a56dfb299a2f7461e469e75ebb7a26a42c122e1e84e4d46a143d8a56b702df58595e0496fd369fe90d5a5cb0968745f36293173959473a52df991db9a057aa860e6ec7493fb62f9278e24f1101ec2a88496be64fe7c2b8f69454c3030f98912b20e57c906a34cd3389933259d1cdc60837267dab14593dfc4341433f7b16aa8ec901a66fc14f736ed45bc53517cde99db61ef652477ccb4bf7caafe10e57362d226cd6a27906d66221653de01d0c8621b7f55c9edda4450e5b8708806aa671a4bb3bcfb0c3aefac537538fb3c4518b2f3cc9bebe3999de89b715520fa7bf25efa24321bd982834025777f5d720b283432993f456a22bf2ab1f7cadbabe78d5bcac146261ddfd78f42f556a153d04afeba97bfabe7e1b1bab5dc205ea1f1772a9e65ebfada88c577dfe8684438af0009bcfe91f53e8c9985011a1825bee7f3ef3adc7f6d15cb7c55489af26f039b67e10ebc3a2a53505010ede26797ee181e46d402671754d17dd5564f0e6611fe3655c144b68760cd919376ac3255c9fdbe366c7f7491c9324eb505b51100f3fd67251bb6e2a290d7c56b4a58098f06437f2e333ec3f6cf76e3ee75950ff412008f8c60927d405808a1ec5d84986c8b4bfe6c1e8a02d3f35015456c911ce421c3db22e7dd7e8d2edbf804b54b3801df2b2e32f800688d952d5e81b28316b4a8faf6e0540b8f4f9af6b1939a7d4c1814504c58f7372dfab298da2b638e7353318951b2757802182c5a4d949f7b7cfb0c1514ffde316a0778c2614fbf02ef5a29f033ba085d9500a8d98a373378e3ee3378f7ded7a3d74c77b43f11261b7b5f9e067a45acd7049b011eb1eb6e406071ebfa38086534305cddfd9db238b6528227ddc64a8214a4fa79e1b099381f6184d8b9c22d44361260404ad583b0233d028008edadc8580c9131e1410ec11e8297afd457bfd2817f80767c4d0c0a99304b99dfaf59d60322c79b3c37149d26b625db6229276943592642f84cd7a095cb5ad2399a28978f59d5e9d86b51cc6aeff759b1dae767c0c84d5a6408e232f59efbf383b923ad6738552238aae00e4ef5273f3f5b494ef2dd23f6c603fbb0c26db20e320529bc443d9b58aef1bc5836971b4f2acdf250b8eaf5df0c4800d019ccf2d4ded894b11504dc360b969530e713e02e5dbb6c5fb99b7729e541499ce3b3fbadf0bb88f623a6eee018a68a18b6f6f57292612caaebcfc792fa88dbf234633645d5865da5fe3bcf39c2600b79f3139dec529c808a88f9b065b25cf09c510dc5ddf56de47dd462e881c5061646c4b408768ad67cd2d8169abaf9c277889456442b3945a56eb91dd6836a1974366dd76c88c3bc1f35791e3cb48ef26868ec56dfa812950018110c97f8a6c1c39298c1f4ffcca95d95763f08473cd9a7dd538b15c3b99f3b8355dead71c32615ed937157fbf55d5b324abaee42404d6aff7d2caa016798b16d603567006c733c01a8f1728ec15bccc9d999a0c5d5271e68f3b53046e1b78c87ff1e0e343d02c851be03a191caddb580b1ea01eadb271272c50e4903455e4f80978e507b4d09db62dd59fd0134bb84f4cf72a713a47b125a9b090df20c3deed1c4679609bb800706191dec560a966dfd1bb36c01ab8433e9cedd47ca33aaca9305a8f190f356f78f9b9db2eefc41e0a0b402d48bb4e7125f25c0c632c5f2f2cccd5f5a4ffa813fe009cdc216dc960b657a675d487b0c0c172e97e5dc19a64974c66abd2203b4080fd9399cf1643d1c951dbc90e2eb1f6656a20d2c9eec9ae5af685031c4de9c02987273551a7383c54d7e16d6281bb4d02c1f6cff44d4c5f30d925a321a444351c29418abe135cf4f7954f786e32852fe5258f493bc9dcbfcdff230c31f036b565b1cdef31ecb62172ceaba403026211aec825c4bd8a8b1c71fd35be13938c426cb2fdd3a19f5f44c77d8675d6f5b846473a0e4247623faad34e271cced3d4ecb531650820913eafb414d95b956ff1a5e2d48105115312ee8ba9cd0045d9f1efb69a18b35502ac6bf035c52f679ead961a4c9b50c2a0bc7055232b5d3fe3bb5246562980d3e13c4f821158f623135bc32c61b49e558e226a11aef5d9a678964b11679bf2531504047e854&x5step=100&nc_app_key=X82Y__6eaa13982eaa135add1e2d6e36ba98f9"

• Server

require "http/server"

server = HTTP::Server.new do |context|
  context.response.content_type = "text/plain"
  context.response.print "Hello world!"
end

address = server.bind_tcp 8080
puts "Listening on http://#{address}"
server.listen

Verbose

$ curl -v "http://127.0.0.1:8080/list_detail_rate.htm/_____tmd_____/verify/?nc_token=33bffaa55be619b46359e6353d1bd6d9&nc_session_id=01VsUAH0NTPMVtfd7lyQMahoibWdyK1P8oUMd5M98WlcQ84bFi0PnmHETc3rGPtk3VN8HPWMx6QlMWV-Nw9LKhFln6-0AXmw2z9nEi93BnEET98VsjMF0mhWdmdvkUgS9eiw8IygcyecEQzuZucDqgaEY3jNRF8BNoe2JrjpADDmLoEzvs7GjkrYL_EbxDjGJoHvJG0DpdSHZOdyTDPcKc6A&nc_sig=059HQ0XsykTiYwN9Dn38G9XAcFHfXDvRtfFNlgtdIua129sbzmSlbWHFUzKuj3U0IGA3bzvprIaQ2IoO5j4l7TyEi806jDSf_kjePvkLjnFM0XA-ChnD7rcqrnGe1b_Mp-1YxplkweRUguUHMefTFifKoi1GowPa_H7SLr8CzbOCPNs6gWLiqxnwjLVLEgRlHUhzIiGJzKxLpS1LnjuS6F8jnEPs_3D0-lJiwaplLuVDbhA7fg4QFn39t4VWlR50gyAaalQwqubUnJlldueM0Dq6XPfM_6Ndy6_nQAW-3Asvt74RX7LO3w2WYnYgtfI6m2UL8jXC9Yiq4UBV1U7xZ-DVecZNUTkrHO1432KkP1TBgo2o3GOxbowBqfBsLfhdzrze9Jac4FCUrw_wieJX98p1sAxQHojeOF39MSL4bxBD7NQBcMbHWkrAmSui0NXBs_ie1EDml3wzWml9OAPM062sXW_TewgrvFacZBClGCSEA&x5secdata=5e0c8e1365474455070961b803bd560607b52cabf5960afff39b64ce58073f7844a8dd94c89edef01763e7bda2153e72fcab45d62d62d1d0fb9af411e98157f55151e037d30b894fa1b9f0617640df860a901a36bd6d324642091753f1253ec47fad7705c7bf26a7ef1aa109e380a620464f3a20661c6f7fc8f51770469f5a56dfb299a2f7461e469e75ebb7a26a42c122e1e84e4d46a143d8a56b702df58595e0496fd369fe90d5a5cb0968745f36293173959473a52df991db9a057aa860e6ec7493fb62f9278e24f1101ec2a88496be64fe7c2b8f69454c3030f98912b20e57c906a34cd3389933259d1cdc60837267dab14593dfc4341433f7b16aa8ec901a66fc14f736ed45bc53517cde99db61ef652477ccb4bf7caafe10e57362d226cd6a27906d66221653de01d0c8621b7f55c9edda4450e5b8708806aa671a4bb3bcfb0c3aefac537538fb3c4518b2f3cc9bebe3999de89b715520fa7bf25efa24321bd982834025777f5d720b283432993f456a22bf2ab1f7cadbabe78d5bcac146261ddfd78f42f556a153d04afeba97bfabe7e1b1bab5dc205ea1f1772a9e65ebfada88c577dfe8684438af0009bcfe91f53e8c9985011a1825bee7f3ef3adc7f6d15cb7c55489af26f039b67e10ebc3a2a53505010ede26797ee181e46d402671754d17dd5564f0e6611fe3655c144b68760cd919376ac3255c9fdbe366c7f7491c9324eb505b51100f3fd67251bb6e2a290d7c56b4a58098f06437f2e333ec3f6cf76e3ee75950ff412008f8c60927d405808a1ec5d84986c8b4bfe6c1e8a02d3f35015456c911ce421c3db22e7dd7e8d2edbf804b54b3801df2b2e32f800688d952d5e81b28316b4a8faf6e0540b8f4f9af6b1939a7d4c1814504c58f7372dfab298da2b638e7353318951b2757802182c5a4d949f7b7cfb0c1514ffde316a0778c2614fbf02ef5a29f033ba085d9500a8d98a373378e3ee3378f7ded7a3d74c77b43f11261b7b5f9e067a45acd7049b011eb1eb6e406071ebfa38086534305cddfd9db238b6528227ddc64a8214a4fa79e1b099381f6184d8b9c22d44361260404ad583b0233d028008edadc8580c9131e1410ec11e8297afd457bfd2817f80767c4d0c0a99304b99dfaf59d60322c79b3c37149d26b625db6229276943592642f84cd7a095cb5ad2399a28978f59d5e9d86b51cc6aeff759b1dae767c0c84d5a6408e232f59efbf383b923ad6738552238aae00e4ef5273f3f5b494ef2dd23f6c603fbb0c26db20e320529bc443d9b58aef1bc5836971b4f2acdf250b8eaf5df0c4800d019ccf2d4ded894b11504dc360b969530e713e02e5dbb6c5fb99b7729e541499ce3b3fbadf0bb88f623a6eee018a68a18b6f6f57292612caaebcfc792fa88dbf234633645d5865da5fe3bcf39c2600b79f3139dec529c808a88f9b065b25cf09c510dc5ddf56de47dd462e881c5061646c4b408768ad67cd2d8169abaf9c277889456442b3945a56eb91dd6836a1974366dd76c88c3bc1f35791e3cb48ef26868ec56dfa812950018110c97f8a6c1c39298c1f4ffcca95d95763f08473cd9a7dd538b15c3b99f3b8355dead71c32615ed937157fbf55d5b324abaee42404d6aff7d2caa016798b16d603567006c733c01a8f1728ec15bccc9d999a0c5d5271e68f3b53046e1b78c87ff1e0e343d02c851be03a191caddb580b1ea01eadb271272c50e4903455e4f80978e507b4d09db62dd59fd0134bb84f4cf72a713a47b125a9b090df20c3deed1c4679609bb800706191dec560a966dfd1bb36c01ab8433e9cedd47ca33aaca9305a8f190f356f78f9b9db2eefc41e0a0b402d48bb4e7125f25c0c632c5f2f2cccd5f5a4ffa813fe009cdc216dc960b657a675d487b0c0c172e97e5dc19a64974c66abd2203b4080fd9399cf1643d1c951dbc90e2eb1f6656a20d2c9eec9ae5af685031c4de9c02987273551a7383c54d7e16d6281bb4d02c1f6cff44d4c5f30d925a321a444351c29418abe135cf4f7954f786e32852fe5258f493bc9dcbfcdff230c31f036b565b1cdef31ecb62172ceaba403026211aec825c4bd8a8b1c71fd35be13938c426cb2fdd3a19f5f44c77d8675d6f5b846473a0e4247623faad34e271cced3d4ecb531650820913eafb414d95b956ff1a5e2d48105115312ee8ba9cd0045d9f1efb69a18b35502ac6bf035c52f679ead961a4c9b50c2a0bc7055232b5d3fe3bb5246562980d3e13c4f821158f623135bc32c61b49e558e226a11aef5d9a678964b11679bf2531504047e854&x5step=100&nc_app_key=X82Y__6eaa13982eaa135add1e2d6e36ba98f9"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 1087 (#0)
> GET http://127.0.0.1:8080/list_detail_rate.htm/_____tmd_____/verify/?nc_token=33bffaa55be619b46359e6353d1bd6d9&nc_session_id=01VsUAH0NTPMVtfd7lyQMahoibWdyK1P8oUMd5M98WlcQ84bFi0PnmHETc3rGPtk3VN8HPWMx6QlMWV-Nw9LKhFln6-0AXmw2z9nEi93BnEET98VsjMF0mhWdmdvkUgS9eiw8IygcyecEQzuZucDqgaEY3jNRF8BNoe2JrjpADDmLoEzvs7GjkrYL_EbxDjGJoHvJG0DpdSHZOdyTDPcKc6A&nc_sig=059HQ0XsykTiYwN9Dn38G9XAcFHfXDvRtfFNlgtdIua129sbzmSlbWHFUzKuj3U0IGA3bzvprIaQ2IoO5j4l7TyEi806jDSf_kjePvkLjnFM0XA-ChnD7rcqrnGe1b_Mp-1YxplkweRUguUHMefTFifKoi1GowPa_H7SLr8CzbOCPNs6gWLiqxnwjLVLEgRlHUhzIiGJzKxLpS1LnjuS6F8jnEPs_3D0-lJiwaplLuVDbhA7fg4QFn39t4VWlR50gyAaalQwqubUnJlldueM0Dq6XPfM_6Ndy6_nQAW-3Asvt74RX7LO3w2WYnYgtfI6m2UL8jXC9Yiq4UBV1U7xZ-DVecZNUTkrHO1432KkP1TBgo2o3GOxbowBqfBsLfhdzrze9Jac4FCUrw_wieJX98p1sAxQHojeOF39MSL4bxBD7NQBcMbHWkrAmSui0NXBs_ie1EDml3wzWml9OAPM062sXW_TewgrvFacZBClGCSEA&x5secdata=5e0c8e1365474455070961b803bd560607b52cabf5960afff39b64ce58073f7844a8dd94c89edef01763e7bda2153e72fcab45d62d62d1d0fb9af411e98157f55151e037d30b894fa1b9f0617640df860a901a36bd6d324642091753f1253ec47fad7705c7bf26a7ef1aa109e380a620464f3a20661c6f7fc8f51770469f5a56dfb299a2f7461e469e75ebb7a26a42c122e1e84e4d46a143d8a56b702df58595e0496fd369fe90d5a5cb0968745f36293173959473a52df991db9a057aa860e6ec7493fb62f9278e24f1101ec2a88496be64fe7c2b8f69454c3030f98912b20e57c906a34cd3389933259d1cdc60837267dab14593dfc4341433f7b16aa8ec901a66fc14f736ed45bc53517cde99db61ef652477ccb4bf7caafe10e57362d226cd6a27906d66221653de01d0c8621b7f55c9edda4450e5b8708806aa671a4bb3bcfb0c3aefac537538fb3c4518b2f3cc9bebe3999de89b715520fa7bf25efa24321bd982834025777f5d720b283432993f456a22bf2ab1f7cadbabe78d5bcac146261ddfd78f42f556a153d04afeba97bfabe7e1b1bab5dc205ea1f1772a9e65ebfada88c577dfe8684438af0009bcfe91f53e8c9985011a1825bee7f3ef3adc7f6d15cb7c55489af26f039b67e10ebc3a2a53505010ede26797ee181e46d402671754d17dd5564f0e6611fe3655c144b68760cd919376ac3255c9fdbe366c7f7491c9324eb505b51100f3fd67251bb6e2a290d7c56b4a58098f06437f2e333ec3f6cf76e3ee75950ff412008f8c60927d405808a1ec5d84986c8b4bfe6c1e8a02d3f35015456c911ce421c3db22e7dd7e8d2edbf804b54b3801df2b2e32f800688d952d5e81b28316b4a8faf6e0540b8f4f9af6b1939a7d4c1814504c58f7372dfab298da2b638e7353318951b2757802182c5a4d949f7b7cfb0c1514ffde316a0778c2614fbf02ef5a29f033ba085d9500a8d98a373378e3ee3378f7ded7a3d74c77b43f11261b7b5f9e067a45acd7049b011eb1eb6e406071ebfa38086534305cddfd9db238b6528227ddc64a8214a4fa79e1b099381f6184d8b9c22d44361260404ad583b0233d028008edadc8580c9131e1410ec11e8297afd457bfd2817f80767c4d0c0a99304b99dfaf59d60322c79b3c37149d26b625db6229276943592642f84cd7a095cb5ad2399a28978f59d5e9d86b51cc6aeff759b1dae767c0c84d5a6408e232f59efbf383b923ad6738552238aae00e4ef5273f3f5b494ef2dd23f6c603fbb0c26db20e320529bc443d9b58aef1bc5836971b4f2acdf250b8eaf5df0c4800d019ccf2d4ded894b11504dc360b969530e713e02e5dbb6c5fb99b7729e541499ce3b3fbadf0bb88f623a6eee018a68a18b6f6f57292612caaebcfc792fa88dbf234633645d5865da5fe3bcf39c2600b79f3139dec529c808a88f9b065b25cf09c510dc5ddf56de47dd462e881c5061646c4b408768ad67cd2d8169abaf9c277889456442b3945a56eb91dd6836a1974366dd76c88c3bc1f35791e3cb48ef26868ec56dfa812950018110c97f8a6c1c39298c1f4ffcca95d95763f08473cd9a7dd538b15c3b99f3b8355dead71c32615ed937157fbf55d5b324abaee42404d6aff7d2caa016798b16d603567006c733c01a8f1728ec15bccc9d999a0c5d5271e68f3b53046e1b78c87ff1e0e343d02c851be03a191caddb580b1ea01eadb271272c50e4903455e4f80978e507b4d09db62dd59fd0134bb84f4cf72a713a47b125a9b090df20c3deed1c4679609bb800706191dec560a966dfd1bb36c01ab8433e9cedd47ca33aaca9305a8f190f356f78f9b9db2eefc41e0a0b402d48bb4e7125f25c0c632c5f2f2cccd5f5a4ffa813fe009cdc216dc960b657a675d487b0c0c172e97e5dc19a64974c66abd2203b4080fd9399cf1643d1c951dbc90e2eb1f6656a20d2c9eec9ae5af685031c4de9c02987273551a7383c54d7e16d6281bb4d02c1f6cff44d4c5f30d925a321a444351c29418abe135cf4f7954f786e32852fe5258f493bc9dcbfcdff230c31f036b565b1cdef31ecb62172ceaba403026211aec825c4bd8a8b1c71fd35be13938c426cb2fdd3a19f5f44c77d8675d6f5b846473a0e4247623faad34e271cced3d4ecb531650820913eafb414d95b956ff1a5e2d48105115312ee8ba9cd0045d9f1efb69a18b35502ac6bf035c52f679ead961a4c9b50c2a0bc7055232b5d3fe3bb5246562980d3e13c4f821158f623135bc32c61b49e558e226a11aef5d9a678964b11679bf2531504047e854&x5step=100&nc_app_key=X82Y__6eaa13982eaa135add1e2d6e36ba98f9 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Proxy-Connection: keep-alive
< 
400 Bad Request
* Connection #0 to host 127.0.0.1 left intact

[asterite]

Yes, that's correct, and I think that's expected. I don't see this as a bug.

[ghost]

@asterite But sometimes you need to use a long URL, is there any solution? I am making a traffic relay(forwarder) (similar to MITM Proxy / BurpSuite)

[asterite]

Yes, I think there's something to be fixed here. Maybe the limit should be (much) longer. It works fine with a Go server.

[ghost]

@asterite Okay, Thanks. I often encounter long url situations (the above URL comes from Taobao captcha security verification) 😂.

[asterite]

It seems Go uses 1MB limit by default. It depends on each framework/library/server, but maybe 1MB is good. Then we should also make this configurable.

[ghost]

@asterite Okay.

[kazzkiq]

Just to elaborate a little more.

In the past, most browsers accepted fairily low URL lengths (IE had a max limit of 2083 chars).

But those limits were never part of any HTTP specs, just something browsers came with at some point.

To sum it up, this is basically what specs says about URL length: Your server should handle URLs as big as possible, but if you really needs to setup a limit, then send a 414 Request-URI Too Long error to client.

So I think we should do two things here:

1. Raise the URL length as far as possible;
2. Change the error to: 414 Request-URI Too Long in case of limit crossed.

[straight-shoota]

I have a patch for this. It's already based on #8002 but I'll wait for the followup,