Arduino-esp32: WIFI, BLE, HTTP, SD_MMS error download file. Small free heap?

[ArTyGill]

Hardware:

Board: ESP32 Dev Module
Chip: ESP32D0WDQ6 (revision 0)
Core Installation/update date: 11/jul/2017
IDE name: VisualStudio (Arduino IDE)
Flash Frequency: 80Mhz
Upload Speed: 921600
esptool.py : v2.1

Description:

I use wifi and https client to download the file to the SD card, everything works fine, but when I initialize the bluetooth le when accessing the server using the get query, it returns an error code (-1). Although the esp itself is connected to the wifi and pings in my local network with bluetooth without problems.
I think this is a shortage of a free heap memory, but I'm not sure about it.

Sketch:

/******************** Includes **********************/
#include <Arduino.h>

#include <WiFi.h>
#include <WiFiMulti.h>
#include <HTTPClient.h>

#include <FS.h>
#include "SD_MMC.h"

#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>
#include <BLE2902.h>

/********************** Definitions ***************************/

#define UUID_SERVISE         (BLEUUID("00000000-0000-0000-0000-111111111111"))
#define UUID_CHARACH         (BLEUUID("00000000-0000-0000-0000-222222222222"))


/********************** Variables *****************************/

WiFiMulti Multiwifi;
const char* ssid = "VB";
const char* password = "C039D2E9";

HTTPClient http;

BLEAdvertising *pAdvertising;

uint8_t cardMounted;


/****************** Function prototypes ********************/

void sd_init(void);
void wifi_init(void);
void BLE_init(void);
void ble_adv_start(void);
void writeFile(fs::FS &fs, const char * path, const char * message);

/************************ Functions *************************/
void sd_init(void) {

    gpio_set_pull_mode(GPIO_NUM_2, GPIO_PULLUP_ONLY); //set Pullup for SD_MMC slot
    gpio_set_pull_mode(GPIO_NUM_4, GPIO_PULLUP_ONLY);
    gpio_set_pull_mode(GPIO_NUM_12, GPIO_PULLUP_ONLY);
    gpio_set_pull_mode(GPIO_NUM_13, GPIO_PULLUP_ONLY);
    gpio_set_pull_mode(GPIO_NUM_15, GPIO_PULLUP_ONLY);
    gpio_set_pull_mode(GPIO_NUM_14, GPIO_PULLUP_ONLY);
    if (!SD_MMC.begin()) {
    cardMounted = 0;
    Serial.println("Card Mount Failed");
    return;
    }
    uint8_t cardType = SD_MMC.cardType();

    if (cardType == CARD_NONE) {
    cardMounted = 0;
    Serial.println("No SD_MMC card attached");
    return;
    }

    Serial.print("\r\nSD_MMC Card Type: ");
    if (cardType == CARD_MMC) {
    Serial.println("MMC");
    }
    else if (cardType == CARD_SD) {
    Serial.println("SDSC");
    }
    else if (cardType == CARD_SDHC) {
    Serial.println("SDHC");
    }
    else {
    Serial.println("UNKNOWN");
    }

    uint64_t cardSize = SD_MMC.cardSize() / (1024 * 1024);
    Serial.printf("\r\nSD_MMC Card Size: %lluMB\n", cardSize);
    cardMounted = 1;
}

void wifi_init(void)
 {
    Multiwifi.addAP(ssid, password);
    Serial.println("Connecting Wifi...");
    if (Multiwifi.run() == WL_CONNECTED) {
        Serial.println("");
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
    }
}

void BLE_init(void)
{
    BLEDevice::init("ESP32");   

    BLEServer *pServer = BLEDevice::createServer();                     
    BLEService *pService = pServer->createService
                       (
                       UUID_SERVISE
                       );       
    BLECharacteristic *pCharacteristicBut = pService->createCharacteristic 
                      (  UUID_CHARACH,
                 BLECharacteristic::PROPERTY_READ |
                 BLECharacteristic::PROPERTY_NOTIFY
                  );
    pService->start();
    pAdvertising = pServer->getAdvertising();
}

void ble_adv_start(void){
   pAdvertising->start();
}

void writeFile(fs::FS &fs, const char * path, const char * message){
    Serial.printf("Writing file: %s\n", path);

    File file = fs.open(path, FILE_WRITE);
    if(!file){
        Serial.println("Failed to open file for writing");
        return;
    }

    if(file.print((char*)message)){
        Serial.println("File written");
    } else {
        Serial.println("Write failed");
    }
    file.close();
}

/*************** Main functions *******************/

void setup() {
    sd_init();
    wifi_init();
    BLE_init(); // if not init ble all worked!

    http.begin("https://localhost:8080/test/new.txt"); 
    int request = http.GET();
    Serial.printf("\r\n Get request %d", request);
    if (request)
    { 
         writeFile(SD_MMC, "/new.txt", http.getString().c_str());
    }  
    ble_adv_start();
}

void loop() {

}

Debug Messages:

/******************************************************************************************/
//----------------------------------With BLE_init();-------------------------------------------
/******************************************************************************************/

Enable Core debug level: Debug on tools menu of Arduino IDE, then put the serial output here 
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:956
load:0x40078000,len:0
load:0x40078000,len:13076
entry 0x40078a58
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 2 - STA_START
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 4 - STA_CONNECTED
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 7 - STA_GOT_IP
[D][BLEDevice.cpp:69] createServer(): >> createServer
[D][BLEServer.cpp:304] registerApp(): >> registerApp - 0
[D][BLEDevice.cpp:95] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 3] ... ESP_GATTS_REG_EVT
[D][BLEUtils.cpp:1647] dumpGattServerEvent(): GATT ServerEvent: ESP_GATTS_REG_EVT
[D][BLEUtils.cpp:1791] dumpGattServerEvent(): [status: ESP_GATT_OK, app_id: 0]
[D][BLEServer.cpp:176] handleGATTServerEvent(): >> handleGATTServerEvent: ESP_GATTS_REG_EVT
[[][BBLEerrve..ppp3084] egisteGApp(erverEvegi():rApp
haD][eGADeSiceerEp:76]
createServer(): << createServer
[D][BLEServer.cpp:76] createService(): >> createService - 00000000-0000-0000-0000-111111111111
[D][BLEDevice.cpp:95] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 3] ... ESP_GATTS_CREATE_EVT
[D][BLEUtils.cpp:1647] dumpGattServerEvent(): GATT ServerEvent: ESP_GATTS_CREATE_EVT
[D][BLEUtils.cpp:1716] dumpGattServerEvent(): [status: ESP_GATT_OK, service_handle: 40 0x28, service_id: [uuid: 00000000-0000-0000-0000-111111111111, inst_id: 0]]
[D][BLEServer.cpp:176] handleGATTServerEvent(): >> handleGATTServerEvent: ESP_GATTS_CREATE_EVT
[D][BLEService.cpp:160] setHandle(): >> setHandle - Handle=0x28, service UUID=00000000-0000-0000-0000-111111111111)
[D][BLEService.cpp:166] setHandle(): << setHandle
[D[DBLELESvicer.cp:829 e ecudlereTte(rv <Evextc)teC< hted
eDATTLErveverent
93] createService(): << createService
[D][BLEService.cpp:188] addCharacteristic(): >> addCharacteristic()
[D][BLEService.cpp:191] addCharacteristic(): Adding characteristic: uuid=00000000-0000-0000-0000-222222222222 to service: UUID: 00000000-0000-0000-0000-111111111111, handle: 0x28
[D][BLEService.cpp:203] addCharacteristic(): << addCharacteristic()
[D][BLEService.cpp:125] start(): >> start(): Starting service (esp_ble_gatts_start_service): UUID: 00000000-0000-0000-0000-111111111111, handle: 0x28
[D][BLECharacteristic.cpp:83] executeCreate(): >> executeCreate()
[D][BLECharacteristic.cpp:94] executeCreate(): Registering characteristic (esp_ble_gatts_add_char): uuid: 00000000-0000-0000-0000-222222222222, service: UUID: 00000000-0000-0000-0000-111111111111, handle: 0x28
[D][BLEDevice.cpp:95] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 3] ... ESP_GATTS_ADD_CHAR_EVT
[D][BLEUtils.cpp:1647] dumpGattServerEvent(): GATT ServerEvent: ESP_GATTS_ADD_CHAR_EVT
[D][BLEUtils.cpp:1669] dumpGattServerEvent(): [status: ESP_GATT_OK, attr_handle: 42 0x2a, service_handle: 40 0x28, char_uuid: 00000000-0000-0000-0000-222222222222]
[D][BLEServer.cpp:176] handleGATTServerEvent(): >> handleGATTServerEvent: ESP_GATTS_ADD_CHAR_EVT
[D][BLECharacteristic.cpp:609] setHandle(): >> setHandle: handle=0x2a, characteristic uuid=00000000-0000-0000-0000-222222222222
[D][BLECharacteristic.cpp:611] setHandle(): << setHandle
[D][BLECharacteristic.cpp:209] handleGATTServerEvent(): >> handleGATTServerEvent: ESP_GATTS_ADD_CHAR_EVT
[DD[BLLECaraattriistc.ccpp:131] exedueGATeate(Ser: rEvexe(): << ate
eGATTServerEvent
[D][BLEServer.cpp:294] handleGATTServerEvent(): << handleGATTServerEvent
[D][BLEDevice.cpp:95] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 3] ... ESP_GATTS_START_EVT
[D][BLEUtils.cpp:1647] dumpGattServerEvent(): GATT ServerEvent: ESP_GATTS_START_EVT
[D][BLEUtils.cpp:1805] dumpGattServerEvent(): [status: ESP_GATT_OK, service_handle: 0x28]
[D][BLEServer.cpp:176] handleGATTServerEvent(): >> handleGATTServerEvent: ESP_GATTS_START_EVT
[V][HTTPClient.cpp:140] beginInternal(): url: https://localhost:8080/test/new.txt
[W][HTTPClient.cpp:152] beginInternal(): unexpected protocol: https, expected http
[V][HTTPClient.cpp:140] beginInternal(): url: https://localhost:8080/test/new.txt
[D][HTTPClient.cpp:181] beginInternal(): host: localhost port: 8080 url: /test/new.txt
[V][ssl_client.cpp:48] start_ssl_client(): Free heap before TLS 50524
[V][ssl_client.cpp:50] start_ssl_client(): Starting socket
[V][ssl_client.cpp:86] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:95] start_ssl_client(): Setting up the SSL/TLS structure...
[I][ssl_client.cpp:119] start_ssl_client(): WARNING: Use certificates for a more secure communication!
[V][ssl_client.cpp:143] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:158] start_ssl_client(): Performing the SSL/TLS handshake...
[E][ssl_client.cpp:28] handle_error(): RSA - The public key operation failed : BIGNUM - Memory allocation failed
[E][ssl_client.cpp:30] handle_error(): MbedTLS message code: -17040
[E][WiFiClientSecure.cpp:108] connect(): lwip_connect_r: 12
[V][ssl_client.cpp:209] stop_ssl_socket(): Cleaning SSL connection.
[D][HTTPClient.cpp:848] connect(): failed connect to localhost:8080
[W][HTTPClient.cpp:1126] returnError(): error(-1): connection refused
[W][HTTPClient.cpp:1126] returnError(): error(-4): not connected


/******************************************************************************************/
//----------------------------------Without BLE_init();--------------------------------------//
/******************************************************************************************/

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:956
load:0x40078000,len:0
load:0x40078000,len:13076
entry 0x40078a58
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 2 - STA_START
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 4 - STA_CONNECTED
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 7 - STA_GOT_IP
[V][HTTPClient.cpp:140] beginInternal(): url: https://localhost:8080/test/new.txt
[W][HTTPClient.cpp:152] beginInternal(): unexpected protocol: https, expected http
[V][HTTPClient.cpp:140] beginInternal(): url: https://localhost:8080/test/new.txt
[D][HTTPClient.cpp:181] beginInternal(): host: localhost port: 8080 url: /test/new.txt
[V][ssl_client.cpp:48] start_ssl_client(): Free heap before TLS 123148
[V][ssl_client.cpp:50] start_ssl_client(): Starting socket
[V][ssl_client.cpp:86] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:95] start_ssl_client(): Setting up the SSL/TLS structure...
[I][ssl_client.cpp:119] start_ssl_client(): WARNING: Use certificates for a more secure communication!
[V][ssl_client.cpp:143] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:158] start_ssl_client(): Performing the SSL/TLS handshake...
[V][ssl_client.cpp:177] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:186] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:201] start_ssl_client(): Free heap after TLS 76112
[D][HTTPClient.cpp:852] connect():  connected to localhost:8080
[V][ssl_client.cpp:240] send_ssl_data(): Writing HTTP request...
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'X-Powered-By: Express'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Accept-Ranges: bytes'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Cache-Control: public, max-age=0'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Last-Modified: Sat, 19 May 2018 08:19:26 GMT'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'ETag: W/"1-163777b66b0"'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Content-Type: text/plain; charset=UTF-8'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Content-Length: 1'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Date: Sat, 19 May 2018 10:37:58 GMT'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: 'Connection: close'
[V][HTTPClient.cpp:940] handleHeaderResponse(): RX: ''
[D][HTTPClient.cpp:970] handleHeaderResponse(): code: 200
[D][HTTPClient.cpp:973] handleHeaderResponse(): size: 1
[V][ssl_client.cpp:209] stop_ssl_socket(): Cleaning SSL connection.
[V][ssl_client.cpp:209] stop_ssl_socket(): Cleaning SSL connection.
[D][HTTPClient.cpp:1103] writeToStreamDataBlock(): connection closed or file end (written: 1).
[V][HTTPClient.cpp:246] end(): tcp is closed

[lbernstone]

You should try to use streams to download large amounts of data (large is not all that large in esp32). https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/examples/StreamHttpClient/StreamHttpClient.ino

[ArTyGill]

Yes, that's right, I know this, you need to use stream to upload large files, BUT this is a text file in which only one character is written, and the error also occurs as with normal print and when recording with stream.

[ArTyGill]

Is there any reason to use WiFiClientSecure instead of HTTPClient, to save RAM?

[lbernstone]

I'm sorry. I didn't really read your post properly the first time through. Heap errors generally throw kernel exceptions. This is a connection error, as it says. I don't see where you are setting up the socket on localhost, but if it is handled in loop() this could be an indication that the cpu is busy and doesn't ack before timing out.

[ArTyGill]

I have a free 50KB of RAM, ssl opens a socket, but still if you look at the logs, you will see there malloc error.
[E][ssl_client.cpp:28] handle_error(): RSA - The public key operation failed : BIGNUM - Memory allocation failed
[E][ssl_client.cpp:30] handle_error(): MbedTLS message code: -17040

Why does Bluetooth violate http, maybe someone knows why this is happening?

[negativekelvin]

Free heap before TLS 123148 - Free heap after TLS 76112 = 47036 which is pretty close to 50524 especially when you consider heap fragmentation

[ArTyGill]

Well, then this is exactly the lack of RAM memory.
Then what are the options for solving this problem?

[negativekelvin]

1. esp_bt_controller_mem_release(ESP_BT_MODE_CLASSIC_BT)
2. Change TLS content length but you have to recompile from idf

[ArTyGill]

1. The function esp_bt_controller_mem_release(ESP_BT_MODE_CLASSIC_BT) does not help and only clears several kilobytes of memory.
2. I'll try to check. I will change the TLS package.

[ArTyGill]

I changed the sdkconfig.h file, and got 64KB of free heap, this allowed me to use the HTTP client in this case, but increasing the project still leads to a decrease in the free heap, which in the future will also represent a problem.
It is worthwhile to see more options for cleansing the heap.

[ArTyGill]

Is there any reason to use pure "C", and eclipse ide with isp-idf?

[twinotter]

ArTyGill, can you tell me the steps you used to increase the heap space?

Anyone aware of a way to get the TLS system to use the 32bit word addressable memory? It would be great if we could move it into a less frequently used area..

[ArTyGill]

In sdkconfig you need to disable all unnecessary and unused parameters, and it is possible to try to reduce CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN.
Files:
c:\Users\"YouUserNAME"\Documents\Arduino\hardware\espressif\esp32\toolssdk\include\configsdkconfig.h
C:\Users\"YouUserNAME"\Documents\Arduino\hardware\espressif\esp32\toolssdksdkconfig

[ArTyGill]

And you can use ESP32 with external RAM.
But I hope the SDK will optimize soon.

[ArTyGill]

And do not forget to do Backup.

[wegunterjr]

but making these changes suggested (in the sdk config and the CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN) doesn't work in the arduino IDE...right?

[ArTyGill]

These are used by the Arduino environment, and therefore work, but do not expect a large increase in the free heap.

[kapollo]

I'm struggling with the same problem, the bt uses whole lot of heap
Could we recompile the sdk to remove classic bt, there is an option in the sdkconfig for that, how then use that with arduino ide or platform.io?

[ArTyGill]

I'm not sure that we can do this, but I think the developers are esp-arduino, and espressif can.
And try not to use a free real-time operating system, for every task you statically allocate a free heap that is not released.

[ArTyGill]

After I started using Esp32-Wrover with external RAM, everything works fine.