Arduino-esp32: How to change maximum header field length in processing GET requests (http server)

[SchemingWeasels]

What I'm trying to do:

Access ESP32 Camera Http Server Example (included in examples) over the internet, on an Android smartphone running chrome

What is failing: (what I'm failing at)

• Header field in GET request exceeds ESPs limit
• No idea how to change the limit

Hardware:

Board: ESP32-Cam module
Link to Board : https://www.aliexpress.com/item/32992663411.html?spm=a2g0s.9042311.0.0.15764c4dzmz0WC
Core Installation/update date: v1.0.2 in Boards Manager
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: Yes
Upload Speed: 921600
Computer OS: Windows 10
Phone: Oneplus 6T running Andorid 9

Description:

• Based on the Camera Example found in the ESP32 category
• I changed the port of the index page and the MJPEG stream service (config.server_port and ctrl_port in startCameraServer())
• I set the port forwarding rules for my router to point those 2 ports to my ESP32-Cam
• I get the error "Header fields are too long for the server to interpret" when I attempt to load the webpage from my Android Smartphone running Chrome.
• Accessing the ESP via my external IP in INCOGNITO mode works fine
• Accessing the ESP via my external IP without incognito HARDLY WORKS, but randomly works now and then
• I don't know how to see the headers my phone is sending

What I've tried

• I have tried changing the definition of the limits in my sketch and in app_httpd.cpp

#include "sdkconfig.h"
#include "esp_http_server.h"
#undef CONFIG_HTTPD_MAX_REQ_HDR_LEN
#define CONFIG_HTTPD_MAX_REQ_HDR_LEN 1024
#undef HTTPD_MAX_REQ_HDR_LEN
#define HTTPD_MAX_REQ_HDR_LEN 1024

• Changing the max_resp_headers
  config.max_resp_headers = 50;

Found this other person asking for help on the espressive forums, https://esp32.com/viewtopic.php?t=7685 but I have no idea what to do to change the limit.
How do I use the Kconfig file thing? It's right there but I have no idea how do I make the compiler use my values instead.
Read through this as well but I think it's meant for building the sdkconfig.h file https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/kconfig.html

Debug Messages:

Nothing of note. Posted either way

[D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled

[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 0 - WIFI_READY
[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 2 - STA_START
.[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 4 - STA_CONNECTED
[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 7 - STA_GOT_IP
[D][WiFiGeneric.cpp:379] _eventCallback(): STA IP: 192.xxx.xxx.xxx, MASK: 255.255.255.0, GW: 192.xxx.xxx.xxx
.
WiFi connected
Starting web server on port: 'xxxx'
Starting stream server on port: 'xxxx'
Camera Ready! Use 'http://192.xxx.xxx.xxx' to connect

IP redacted

Apologies in Advance

I've got no formal education in coding, I've only picked up what I can from youtube tutorials and Arduino projects. I'm also not a native to the English language. Bare with me while I attempt to learn something in the process!

[SchemingWeasels]

@me-no-dev Sorry for tagging you into this, but I could use some help! Thanks

[SchemingWeasels]

I presume I can edit sdkconfig.h, but right at the very top of the file, there's a comment there that says :

/*
 *
 * Automatically generated file; DO NOT EDIT.
 * Espressif IoT Development Framework Configuration
 *
 */

Soooo what's the proper way to edit the settings/configs?

[jvpernis]

I don't really have a solution for you, but at least I can answer some of your questions:

• I get the error "Header fields are too long for the server to interpret" when I attempt to load the webpage from my Android Smartphone running Chrome.
• Accessing the ESP via my external IP in INCOGNITO mode works fine
• Accessing the ESP via my external IP without incognito HARDLY WORKS, but randomly works now and then

There is a handy little tool called Postman with which you could try access the ESP32 server. This tool gives you fine control over what information gets sent in the HTTP headers. With this you can try to figure out what works and what doesn't.

• I don't know how to see the headers my phone is sending

If you just want to see the headers you're sending to the ESP32, you could also use the Developer Tools in a browser on your PC whenever you navigate to the page (see the instructions for Firefox or Chrome).

How do I use the Kconfig file thing? It's right there but I have no idea how do I make the compiler use my values instead.

Soooo what's the proper way to edit the settings/configs?

Well, usualy, with the ESP-IDF framework, you can change configuration items using a config menu. This menu is invoked by executing make menuconfig. Every component or module can provide a Kconfig file, which is like a recipe for the configuration menu to describe what can be configured. The configuration menu then generates the sdkconfig.h file which is included in many ESP components.

Now, this Arduino core for the ESP32 is built around the ESP-IDF. Unfortunately, when you're using the Arduino IDE, there's currently no way of invoking the configuration menu that I know of.

Someone please correct me if I'm wrong!

I presume I can edit sdkconfig.h

Because of the lack of invoking the configuration menu, I actually believe editing the sdkconfig.h file directly won't cause any harm, despite of the warnings. Just remember to edit this file again whenever you update the board definitions.

Hope this helps!

[jiapei100]

@SchemingWeasels
Same issue here. Do you have a solution NOW?
It seems I'm OK with a single camera most of the time. However

• If there are MULTIPLE esp32-cams
• Even if ONLY a SINGLE esp32-cam, if we go WIFI
  I still have this issue: Header fields are too long for server to interpret

Any suggestions?

[pasquy85]

Confirm. I'm developing an android app with MIT App Inventor and I received the error "Header fields are too long for server to interpret" when trying to connect to the camera streaming page. Only solution was to compile the camera example directly with ESP-IDF and chaning via make menuconfig the value HTTPD_MAX_REQ_HDR_LEN from 512 to 1024. It could be great if I could use Arduino IDE instead of ESP-IDF, especially for me that I'm not a professional.

[stale[bot]]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[easytarget]

Rather than let this go stale, I think it really needs to be addressed somehow, 'Internet of things' devices that cannot be accessed via proxies or from remote networks are probably not what we want.

Following the above and setting CONFIG_HTTPD_MAX_REQ_HDR_LEN to 1024 allowed me to access the cameras over a simple SSH tunnel, but did not allow apache proxying to work.

I'm going to make a second test later today to see what happens with CONFIG_HTTPD_MAX_REQ_HDR_LEN set to 2048 and 'max_resp_headers' set to 16 from the default of 8

[stale[bot]]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[easytarget]

Ok, I was able to do a couple of tests, connecting via VPN/SSH tunnel works for me so I have my solution. But I was not able to get an apache proxy working; even when changing CONFIG_HTTPD_MAX_REQ_HDR_LEN to 2048 and CONFIG_HTTPD_MAX_URI_LEN to 1024.
I'm guessing that Apache adds a heavy additional header load to the proxy (Looking at the apache 2.4 docs I could not see an easy solution to this).
Apache proxied (secure) access is a 'nice to have' for me, but not really that important. I suspect the issue could be solved with a smarter Apache config, or some other proxy system.

It would still be nice to have these options exposed, somehow, in the Arduino IDE. Though I realise that is much easier to say than to actually do.

[stale[bot]]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.

[bengelhaupt]

@easytarget Yes, it seems like the proxy is adding some header load to the original request. I was able to solve it in nginx by adding the proxy_pass_request_headers off directive. Maybe similar behavior can be achieved via ProxyAddHeaders Off in Apache.