Mbed-os: DNS error (-3009) when using ESP8266 with NUCLEO-091RC on some Wi-Fi router

Created on 28 Nov 2019 · 20Comments · Source: ARMmbed/mbed-os

Description of defect

The problem seems about compatibility of mbed+ESP8266 with some WiFi router setup. When tested on one particular WiFi router, mbed+ESP8266 could connect to wifi, get IP address, network mask, gateway etc. But then stuck when trying connect to any http URLs. The error code is: -3009.
However, this doesn't happen with other Wi-Fi routers. The only thing particular about the problematic wifi router is, it has a large black list of URL such as youtbe/facebook etc. But test URLs in mbed-http-example are not in that blacklist, and can be accessed from a computer on the same network.

Target(s) affected by this defect ?

NUCLEO-091RC

Toolchain(s) (name and version) displaying this defect ?

mbed-cli(1.8.3)

What version of Mbed-os are you using (tag or sha) ?

5.13.4

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli, 1.8.3 from windows installer 0.4.10

How is this defect reproduced ?

This happens only with one particular WiFi router so far, cannot be repeated on other routers.

CLOSED mirrored bug

Source

zhiyong80

All 20 comments

Internal Jira reference: https://jira.arm.com/browse/MBOTRIAGE-2419

ciarmcom on 28 Nov 2019

Some other information that might be relevant: 1) Hardware flow control was not enabled. 2) When esp8266.debug was set to true and stdio.baud-rate was set to 9600 instead of 115200, I got a different error 3012 with a previously working wifi router when debug was not turned on.
I suspect this has something to do with buffer overrun. Unfortunately so far I couldn't get hardware flow control to work.

zhiyong80 on 29 Nov 2019

Visited the Wifi router again with hardware flow controll and debug more turned on, still got the same error. This seems to have something to do with google's 8.8.8.8 DNS server:

Connecting to WiFi...
AT> AT
AT? OK
%n
AT< slAT< “obŒl`r’ŸîAT< ll`Œâs“lŒlŒìl`Œãr“lìlŒl`Œãr’pƒŸAT! ready
AT? OK
%n
AT< 
AT(Timeout)
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> AT+RST
AT? OK
%n
AT< AT+RST
AT= OK

AT? %n
AT? %n
AT<  ets Jan  8 2013,rst cause:1, boot mode:(3,7)
AT< load 0x40100000, len 2592, room 16 
AT< tail 0
AT< chksum 0xf3
AT< load 0x3ffe8000, len 764, room 8 
AT< tail 4
AT< chksum 0x92
AT< load 0x3ffe82fc, len 676, room 4 
AT< tail 0
AT< chksum 0x22
AT< csum 0x22
AT< 2nd boot version : 1.7(üŒÅ2±0x1fc000Œâìƒoì’sƒòn|ìlŽìŽlc“žŒ
AT! ready
AT? %n
AT< 
AT? %n
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> ATE0
AT? OK
%n
AT< ATE0
AT= OK

AT> AT+UART_CUR=115200,8,1,0,3
AT? OK
%n
AT= OK

AT> AT+GMR
AT? AT version:%*d.%*d.%*d.%*d%n
AT= AT version:1.7.0.0
AT? OK
%n
AT< (Aug 16 2018 00:57:04)
AT< SDK version:3.0.0(d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+GMR
AT? SDK version:%*d.%*d.%*d%n
AT< AT version:1.7.0.0(Aug 16 2018 00:57:04)
AT= SDK version:3.0.0
AT? OK
%n
AT< (d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+CWMODE_DEF=1
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_DEF=0,"CN",1,13
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_CUR=0,"CN",1,13
AT? OK
%n
AT= OK

AT> AT+CIPRECVMODE=1
AT? OK
%n
AT= OK

AT> AT+CWMODE_CUR=1
AT? OK
%n
AT= OK

AT> AT+CIPMUX=1
AT? OK
%n
AT= OK

AT> AT+CWDHCP_CUR=1,1
AT? OK
%n
AT= OK

AT> AT+CWJAP_CUR="XXXXXXXXXXXXXXXXXX","YYYYYYYYYYYYYYY"
AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= CONNECTED

AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= GOT IP

AT? OK
%n
AT= OK

SuccAT? %ess
n

AT> AT+CIFSR
AT? +CIFSR:STAMAC,"%*17[^"]"%n
AT< +CIFSR:STAIP,"192.168.1.34"
AT= +CIFSR:STAMAC,"b4:e6:2d:6b:0e:65"
AT? OK
%n
AT< 
AT= OK

AT? %MAC: n
b4:e6:2d:6b:0e:65
AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.34"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:0e:65"
AT= OK

AT? IP: 1%n
92.168.1.34
AT> AT+CIPSTA_CUR?
AT? +CIPSTA_CUR:netmask:"%*15[^"]"%n
AT< +CIPSTA_CUR:ip:"192.168.1.34"
AT< +CIPSTA_CUR:gateway:"192.168.1.1"
AT= +CIPSTA_CUR:netmask:"255.255.255.0"
AT? OK
%n
AT< 
AT= OK

AT? %Netman
sk: 255.255.255.0
AT> AT+CIPSTA_CUR?
AT? +CIPSTA_CUR:gateway:"%*15[^"]"%n
AT< +CIPSTA_CUR:ip:"192.168.1.34"
AT= +CIPSTA_CUR:gateway:"192.168.1.1"
AT? OK
%n
AT< 
AT< +CIPSTA_CUR:netmask:"255.255.255.0"
AT= OK

AT? Gatew%n
ay: 192.168.1.1
AT> AT+CWJAP_CUR?
AT? +CWJAP_CUR:"%*[^"]","%*17[^"]"%n
AT= +CWJAP_CUR:"XXXXXXXXXXXXXXXXX","b0:39:56:ec:07:df"
AT? OK
%n
AT< ,3,-63,0
AT= OK

AT? %n
AT> AT+CWLAP="","b0:39:56:ec:07:df",
AT? OK
%n
AT! +CWLAP:
AT? OK
%n
AT< 
AT= OK

RSAT? %SI: -n
55

AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.34"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:0e:65"
AT= OK

AT? %n
AT? %n
AT> AT+CIPSTART=1,"UDP","8.8.8.8",53
AT? OK
%n
AT< 1,CONNECT
AT= OK

AT? %n
AT> AT+CIPSEND=1,34
AT? >%n
AT< OK
AT= >
AT? SEND OK%n
AT<  
AT< Recv 34 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPSEND=1,34
AT? >%n
AT< OK
AT= >
AT? SEND OK%n
AT<  
AT< Recv 34 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPSEND=1,34
AT? >%n
AT< OK
AT= >
AT? SEND OK%n
AT<  
AT< Recv 34 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPCLOSE=1
AT? OK
%n
AT! 1,CLOSED
AT? OK
%n
AT< 
AT= OK

AT? %
HttpRn
equest failed (error code -3009)

----- HTTP GET response -----


++ MbedOS Error Info ++
Error Status: 0x8001011F Code: 287 Module: 1
Error Message: Operator new out of memory

Location: 0x8010713
Error Value: 0x80003F0
Current Thread: main  Id: 0x20002E3C Entry: 0x8015705 StackSize: 0x1000 StackMem: 0x200017B0 SP: 0x200026A4 
For more info, visit: https://mbed.com/s/error?error=0x8001011F&tgt=NUCLEO_F091RC
-- MbedOS Error Info --

zhiyong80 on 29 Nov 2019

The problem seems to be ESP8266 not receiving anything from the DNS server. When URLs are replaced to IP address in HTTP request, the http example code runs just fine.
When a direct request to the same DNS server on the same network and PC, response via UDP, correct response is received.

zhiyong80 on 30 Nov 2019

cc @ARMmbed/mbed-os-wan

0xc0170 on 3 Dec 2019

Maybe HTTPExample does not need so much memory?
And when debug is enabled it affects significantly to memory usage
++ MbedOS Error Info ++
Error Status: 0x8001011F Code: 287 Module: 1
Error Message: Operator new out of memory

AnttiKauppila on 4 Dec 2019

@AnttiKauppila this part actually puzzles me as well. In the example, there are two HTTP requests, 1st for GET, then a POST. If the 1st request fails somehow, http_req would be deleted, then a new request will be created, and mbed shouldn't crash. But oddly, as in log, once that 3009 error shows up, mbed os just crashes.

zhiyong80 on 5 Dec 2019

Hi @zhiyong80 , I wonder - when you disable the blacklisting in your router, are you still able to reproduce the issue?
Can you also try the same router with a device that has more memory, like K64F+ESP8266? 32kB of RAM that NUCLEO_F091RC has is not a lot...

EDIT: I came up with more questions...
Would you also proivde us with the application code that you are using and your mbed_app.json? Does the program fail immediatelly or does it only fail after a while? (Perhaps the heap has been indeed depleted)

Antti suggested that you used http example . If this is the case, then please note, that this project explicitly states:

Note: HTTPS requests do not work on targets with less than 128K of RAM due to the size of the TLS handshake. For more background see mbed-http.

The mbed-http may work on smaller targets, but to mitigate memory issued it has some suggestions on the project's page. Have you tried that out?

Furthermore - enabling debug will surely impact the memory usage, especially on a target with 32 kB of RAM. Are you able to attach a debugger instead of using debug prints?

michalpasztamobica on 31 Dec 2019

@michalpasztamobica I tried STM32F401RE which has 64KB ram, and got the same results, also with mbed-os-5.14.2 and 5.15. The only notable difference with the newer version of mbed-os(5.15?) is, the app will try 3 different DNS servers before giving up. So I don't think this is a ram related problem.

More likely, this might be something related to ESP8266 itself. I dug up some discussion about DNS over UDP failures from years ago. This may still exist on some routers.

I will test the router later today with SparkFun ESP8266 Thing which has nothing to do with Mbed and see what I get.

About app crashing, it is unrelated. I accidentally added an extra line.

        HttpRequest* post_req = new HttpRequest(network, HTTP_POST, "SOME URL");
        post_req->set_header("Content-Type", "application/x-www-form-urlencoded");

        const char body[] = "DUMMY POST FORM DATA";

        HttpResponse* post_res = post_req->send(body, strlen(body));
        if (!post_res) {
            printf("HttpRequest failed (error code %d)\n", post_req->get_error());
            //return 1;
        } else {
           printf("\n----- HTTP POST response -----\n");
           dump_response(post_res);
        }
        delete post_res;   // This line crashes app when (!post_res) or causes memory leak when the code runs in a loop.
        delete post_req;

zhiyong80 on 31 Dec 2019

Just tested the same wifi hotspot with Sparkfun ESP8266 Thing Dev, HTTP GET to http://httpbin.org/status/418 worked just fine. So the problem is either in the wifi module (esp-wroom-02 + 1.70 at firmware) or ESP8266 driver on mbed side.

The problem now narrows down to, using UDP socket instead of builtin command (AT+CIPDOMAIN) to resolve domain to IP might be problematic on certain wifi hotspot setups.

zhiyong80 on 1 Jan 2020

👍1

@AnttiKauppila , @VeijoPesonen , @SeppoTakalo , have you had a chance to try ESP8266's built-in DNS mechanism (AT+CIPDOMAIN)? Would you recommend that we try to use it instead of mbed's nsapi_dns? It could be made optional, dsiabled by default and only enable in mbed_app.json.
I don't see any better way to address this issue.

michalpasztamobica on 2 Jan 2020

I don't think we have tried that.
But the linked forum post is about DNS issues when using ESP's own DNS resolver, so it makes me wonder whether it helps at all in this case.

If the device does not receive anything from the given DNS server, then I don't know how switching a resolver would make this issue disappear. Maybe the WiFi router, in this case, does not route the DNS queries back to ESP.

This probably have to be verified with Wireshark, but to me, it sounds like network issue.
However, the mentioned memory leak or crash have to be studied.

SeppoTakalo on 2 Jan 2020

@SeppoTakalo , further down in the forum thread the ESP team claims to have fixed the issue in their AT command set ("long time ago" they said in March 2018).
Also - according to @zhiyong80 , the SparkFun device is using the built-in DNS resolver, so I think switching to it could fix the issue.

At the end of his previous post @zhiyong80 explained that he accidentally introduced the crash/memory leak, so no work needed for that.

@zhiyong80 , can you run wireshark, as @SeppoTakalo suggested and see if the packet towards ESP is actually visible?

michalpasztamobica on 2 Jan 2020

@michalpasztamobica any instruction on how to tap communication between esp8266 and mbed? My plan is solder the RX line of USB-UART chip to TX line of ESP8266 and see what is actually received or not by mbed after sending AT+CIPDOMAIN.

zhiyong80 on 2 Jan 2020

Your suggestion is one way to go, but I suggest setting up Wireshark and connecting to the same network that ESP8266 is using. Here you can read about the basics. I think it would be good to sniff UDP on port 53 (typically used for DNS communication). If only ESP, the router and your host machine are on the network I don't expect much difficulty figuring out ingoing and outgoing packets. Not sure how well Wireshark is able to decode DNS packets structure, but I'd assume it is quite a usual use case. I didn't read the details, but perhaps this helps?
If you see the DNS packets going from the router to the ESP and still see that mbed doesn't get it - then it's time to either use soldering iron or try the built-in DNS resolution.

michalpasztamobica on 2 Jan 2020

@michalpasztamobica Unfortunately I don't have much control over the wifi hotspot, I have password and can access it, but not much beyond. Spent a good part of today trying to figure out how to capture packets on windows with no luck so far. None of the laptops and USB wifi adapters I have supports monitor mode.

On a side note, I was able to confirm that AT+CIPDOMAIN does work on this router. I connecte a esp-wroom-02 evaluation board that has 1.70 firmware and the command works just fine.

AT+CWMODE=1
//0D0A
OK
AT+CWJAP="SSID","PASSWORD"
WIFI CONNECTED
WIFI GOT IP
//0D0A
OK
AT+CIPDOMAIN="iot.espressif.cn"
+CIPDOMAIN:115.29.202.58
//0D0A
OK

But the accessing DNS server directly via UDP scokets didn't work. Sending the following commands:

AT+CIPSTART=1,"UDP","8.8.8.8",53
AT+CIPSEND=29
AA AA 01 00 00 01 00 00 00 00 00 00 07 65 78 61 6d 70 6c 65 03 63 6f 6d 00 00 01 00 01(hex string to resolve example.com)

should receive something like the following:

+IPD,45 ......

But on this particular wifi hotspot, nothing comes out of ESP8266.
I will conclude this is a compatibility issue of ESP8266 firmware 1.70, based on the following summary:
1) AT+CIPDOMAIN works.
2) Sparkfun ESP8266 thing that based on ESP8266 IC but has a different firmware works
3) Accessing DNS server via UDP socket didn't work, either when connected to mbed or operated via serial tool.

@michalpasztamobica Do you or your colleague have a direct line to manufacturer of ESP8266? Some sort of firmware update will be needed.

zhiyong80 on 3 Jan 2020

@zhiyong80 , I doubt that we have a direct line to Espressif, but I'll forward the question to @AnttiKauppila , perhaps he knows more...

It seems like switching to built-in DNS handling improved your situation and is a valid option to resolve this task? I'll try to add it to mbed's driver and come back to you with a PR to try out.

michalpasztamobica on 3 Jan 2020

@michalpasztamobica Yes switching to built-in DNS command can improve my situation with that particular wifi router. I tried to implement gethostbyname() in ESP8266Interface but that task seems beyond me due to layers of abstraction.

My teammate came up with a bandage by tweaking dns_lookup() in ESP8266 driver:

bool done = _parser.send("AT+CIPDOMAIN=\"%s\"", name) && _parser.recv("+CIPDOMAIN:%15[^\n]\n", ip) && _parser.recv("OK\n"); // This line as of 5.15.1 seems not working because ESP8266 sends out extra 0x0D0x0AOK.

The following is code that works on Sparkfun ESP8266 Thing, included in case it will help. I assumed ESP8266 addon for Arduino uses builtin AT command to resolve DNS, but I am not sure.

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>
#include <WiFiClient.h>
ESP8266WiFiMulti WiFiMulti;

void setup() {
Serial.begin(115200);
Serial.setDebugOutput(true);

Serial.println();
Serial.println();
Serial.println();

for (uint8_t t = 4; t > 0; t--) {
Serial.printf("[SETUP] WAIT %d...\n", t);
Serial.flush();
delay(1000);
}

WiFi.mode(WIFI_STA);
WiFiMulti.addAP("SSID", "PASSWORD");
}

void loop() {
// wait for WiFi connection
if ((WiFiMulti.run() == WL_CONNECTED)) {

WiFiClient client;
HTTPClient http;

Serial.printf("\n----- HTTP GET response -----\n");
if (http.begin(client, "http://httpbin.org/status/418")) { // HTTP

Serial.printf("[HTTP] GET...\n");
// start connection and send HTTP header
int httpCode = http.GET();

// httpCode will be negative on error
if (httpCode > 0) {
// HTTP header has been send and Server response header has been handled
Serial.printf("[HTTP] GET... code: %d\n", httpCode);

// file found at server
if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY || httpCode == 418 /* I'm a teapot */) {
String payload = http.getString();
Serial.println(payload);
}
} else {
Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
}

http.end();
} else {
Serial.printf("[HTTP] Unable to connect\n");
}
}
else
{
Serial.print(".");
delay(500);
}

delay(1000);
}

I also tried to NTP on esp-wroom-02_EVM, the following commands worked as expected while UDP socket for DNS didn't work.

AT+CIPSTART="UDP","us.pool.ntp.org",123
AT+CIPSEND=48
1B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00(hex string)

zhiyong80 on 3 Jan 2020

EDIT: sorry, I mixed up ESP tasks :)
@zhiyong80, here's how I addressed the issue: https://github.com/michalpasztamobica/mbed-os/tree/esp8266_builtin_dns . With this code I could resolve DNS using NetworkInterface::gethostbyname(). Most tests are failing on ESP now, but I will try to figure out the way to use the nsapi_dns for asnychronous resolution.
EDIT2: You need to add "esp8266.built-in-dns": true to your mbed_app.json to enable the feature.

michalpasztamobica on 7 Jan 2020

My previous comment was based on the repo I downloaded 3 days ago, so it was not relevant therefore was deleted.
The latest commit (a33d3972b219104424450a11a26f0f681f391c63) seems to work fine so far. I will test it against the problematic wifi router later when the business opens.

zhiyong80 on 10 Jan 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings