Esp32-snippets: Arduino: BLE Client Notification Error

[WALLTECH]

I put my uuids into the client example and set it up to connect to my peripheral by mac address rather than service as the ones I want aren't advertised, and I get this out with no further data when I expect notifications :

Created client
E (49918) BT: l2cble_start_conn_update, the last connection update command still pending.
E (49918) BT: l2cble_start_conn_update, the last connection update command still pending.
Connected to server
E (50267) BT: l2cble_start_conn_update, staus = 1, line = 509
E (50755) BT: set_read_value unformat.len = 18
The characteristic value was:
E (51039) BT: bta_gattc_cmpl_cback unknown conn_id = 3, ignore data

[nkolban]

In the current BLE Client sample ... the client (ESP32) listens for advertised BLE servers (peripherals) which are advertising a specific service UUID. I am hearing that you have changed the code to look for a specific bluetooth device address (bdAddress) instead of the service. Ok .. no problem there.

The client (sample as-is) then connects to the found BLE server and reads the current value of a characteristic and displays it to the console. If there is no value, nothing is logged. Next the sample loops forever writing to the characteristic value one a second a text string that is how long the ESP32 has been up and running. If we were to examine the characteristic value in nRF Connect we would see the value change. Finally, if the BLE Server issues a notification, the sample app detects this and logs to the console the fact that a notification was asynchronously received.

In your issue report, you say "... I get this out with no further data when I expect notifications :".

The message shows that we connected to the server (good). We also see a message that says we read the characteristic and its value was "" (not unexpected). And then we see no further messages.

How confident are you that your BLE server is "actually" sending notifications which should then show up as messages in the log? Can you pastebin the actual code that you are using to test against? What happens if you test with nRF Connect? (I understand you are using a real-world BLE device).

[WALLTECH]

1. I'm quite confident my peripheral should be sending notifications, as when one connects to it with nRF and enables notifications on the same characteristic, they stream in.
2. https://pastebin.com/pJbRPqtL
3. When I test with nRF Connect replacing the peripheral and duplicating its service, that line doesn't appear and notifications go through to the ESP32 no problem.

[nkolban]

We can switch on ESP32 BLE C++ class diagnostics in the Arduino environment. Find and edit the file called:

<ArduinoIDE>/hardware/espressif/esp32/tools/sdk/include/config/sdkconfig.h

Find the line which reads:

#define CONFIG_LOG_DEFAULT_LEVEL 1

and change that to:

#define CONFIG_LOG_DEFAULT_LEVEL 5

Rebuild/flash your Arduino application. We should now find a lot of good diagnostics written to the console. Run your app again and capture/pastebin the results. That will then show us in context what is happening inside the ESP32 BLE support.

[WALLTECH]

https://pastebin.com/aiCKcqKm

{Kolban: This is a BLE C++ trace}

[nkolban]

The trace looks clean ... no obvious errors in it. The next thing I can think of is running the test again using nRF Connect to connect to your BLE peripheral device (no ESP32 involved). Using nRF Connect we can send requests to the peripheral and see the responses coming back from it. This may require that you install the tool called "nRF Logger" (get it from where you got nRF Connect). This will capture the trace log sent from and received by nRF Connect and we can examine what a "working" environment looks like and compare that against the messages sent and received by the ESP32.

[WALLTECH]

https://pastebin.com/VvQVcvC7 - ESP32 monitor output using nRF as the peripheral
https://pastebin.com/4WLHz2aX - nRF log connecting to the peripheral and getting notifications once the fff1 characteristic of service fff0 is enabled for notifications.

[WALLTECH]

Also worth noting, once subscribed all attempts to read the characteristic that should have values updating with the notifications yet it's returning blank values when itshouldn't. Something is quite wrong here.

[nkolban]

This is obviously a tricky one that is going to be a royal pain to diagnose. What troubles me most is that I am under the impression that when you test your Arduino BLE app with nRF Connect it works but when you test with your actual BLE peripheral device it doesn't. Naturally that limits my ability to test/recreate here as I don't have your target device. I am also cognizant of the assertion that when you connect to your target BLE devices using nRF Connect as the BLE client then your device responds as it should. Looking at the two traces, I am not yet seeing a distinction between what nRF Connect is doing vs what Arduino BLE Client is doing. There are no obvious return code errors from the BLE APIs.

The Arduino BLE library is written in C++ and uses the exposed ESP-IDF BLE APIs. When you make Arduino BLE calls in your app, those are received by the Arduino BLE Library which then makes the appropriate ESP-IDF BLE calls. The ESP-IDF BLE API I also understand to be a wrapper on 3rd party BLE implementation (Bluedroid). As such, we have a lot of opportunities for error:

1. How you are calling Arduino BLE APIs
2. How the Arduino BLE APIs are calling ESP-IDF APIs
3. How the ESP-IDF APIs are calling Bluedroid
4. How the BLE peripheral is supposed to respond

The next trick I can think off is one I have personally no experience in .. and that will be to see if Wireshark can "sniff" the network protocols going on between the ESP32 and your peripheral. If we could somehow get Wireshark to do that, then we could run it twice. Once for communicating with your Peripheral and once for communicating with your nRF Connect.

Another thought would be to get a second ESP32 and write an app against it so that the second ESP32 simulates your peripheral. Again I am going back to the notion that when you use your ESP32 running your Arduino BLE Client app against nRF Connect acting as a BLE Server, I believe it all works. If we wrote an Arduino BLE server app that simulated your actual BLE Peripheral and it still worked then we could potentially point to the BLE Peripheral as "not playing correctly". The Arduino BLE library is as much a pass through to the underlying BLE APIs provided by ESP-IDF. Another thought would be to try and write a BLE Client app using a different API ... for example BLE API on Android or BLE API on Linux and see if your new client can connect to your BLE Peripheral. If it could or can't that would be another clue. All we can see right now is "The BLE Peripheral is not causing the Arduino BLE notification callback to be invoked". We have no other obvious symptoms such as error messages (that mean anything to me).

[WALLTECH]

Yes, it's quite strange. The callback isn't even required for my use, but I can't even read the value when I know it's there also. Can we enable notifications by setting the characteristic descriptor 2902 or whatever it is to 01-00 without using callbacks? Notifications have to be enabled to read the characteristic value in my case. The example code works perfectly in curieBLE.

[nkolban]

The BLE characteristic descriptor with UUID of 0x2902 is part of the BLE specification. Its philosophy works as follows:

A BLE Server is responsible for determining WHEN to issue a notification. Presumably this happens when a device detects a change it wants others to know about. For example, the BLE Server detects movement or the temperature changes. However, if the BLE Server publishes a notification and the client doesn't actually care about receiving it, the client will just ignore the event. While this sounds harmless, the idea is that we want to waste as little energy as possible. To solve this puzzle, IF a BLE Server exposes a descriptor of type 0x2902 then the client is supposed to toggle a bit in that descriptor value which says "I AM / AM NOT interested in receiving notifications". When the BLE server senses a data change that it would like to publish (notify), it checks the corresponding flag in descriptor 0x2902 and based on what is set there, determines whether to publish or ignore. And that's it. That's the end of the story. In theory, your client could toggle on the bit in 0x2902 that says I want to receive notifications but actually do nothing with the notification when it arrives ... but we are entering "odd" territory here.

[WALLTECH]

Ok, I' testing with a different peripheral now and notifications still don't go to the callback. Subscribing works and I can read the characteristics (which shows that subscribe worked) but updates don't go to the callbacks and I really need them to so I don't have to poll the characteristics constantly. Would you please mind checking out how you pass notifications to the callback?

[nkolban]

Howdy @WALLTECH
My thinking works as follows ...

Your app is a BLE Client ... which means that it connects to a BLE Server. It is the BLE Server that "owns" the characteristics. It owns their current values and are maintained by the BLE server. A client can connect to the server and perform the following operations:

1. Read the current value of a characteristic
2. Write a new value to the characteristic

In addition, a client can "request" notifications.

What this means is that the BLE Server can "recognize" that the client is interested in receiving asynchronous updates when the BLE Server chooses to push them ... associated with which ever characteristics the client made known that it wishes to receive those notifications.

The BLE Server is responsible for sending notifications when it sees fit / chooses. Typically, this is as a result of the code running on the BLE Server detecting a new sensor event (temperature has changed, heart beat has risen etc etc). The BLE Server can then choose to:

1. Set the value of the characteristic to the newly sensed value
2. Transmit a notification to the client assuming that the client has made it known that it wishes to receive such notifications.

It is the logic implemented in the BLE server that chooses to do 1, 2, 1 and 2 or neither 1 or 2. It isn't part of the BLE spec that says that something should happen ... only that it can.

If a client has made known that it wishes to receive notifications, then if the BLE server DOES send a notification, then that will arrive as a new incoming message to the BLE client. In the ESP32 world, this will manifest itself as a new asynchronously delivered event passed to the event handler that has been registered by the C++ BLE library. On receipt of such an event, the BLE C++ library will examine the characteristic object that it is maintaining to see if there are any application callbacks registered and, if there are, invoke them.

We see the registration of the callback in BLERemoteCharacteristic::registerForNotify() and the corresponding event handler at BLEClient::gattClientEventHandler(). Within that later function, we see the logic for processing incoming notifications through the event received of type ESP_GATTC_NOTIFY_EVT.

Again ... this is my "model" of what is happening ... so please, if you see me wrong somewhere, call it out.

[WALLTECH]

Ok, I understand. I'm going off of what "should" happen, and what happens when one enables notifications in nRF and they come in from the peripheral. Notifications haven't worked with two peripherals that work in nRF and should with this, so I'm not sure what's going wrong. In CurieBLE I can invoke their valueUpdated() boolean method just fine as notifications are sent to it. In the case of situation 1, is there a method in your library that does this, like valueUpdated?

[nkolban]

Another thought strikes me ... in the Linux world there are a variety of BLE tools including "bluetoothctl". If you don't have Linux, you can run it under Virtual Box. An interesting test might be to get a Bluetooth dongle (or laptop with bluetooth) and see what "bluetoothctl" reports. It appears to be able to act as a client and register for notifications. With that in place, you can have your BLE server transmit notifications and see how/if they are received by the PC. If that works (or doesn't) we can then switch on Wireshark which can also sniff blue tooth traffic if it passes through the PC. From there, we can examine the protocol packets and see what's being sent/received.

As for CurieBLE ... I haven't studied that so don't yet know the semantics of "valueUpdated". I'll take a look and see if I can understand it.

[WALLTECH]

Ok, is nRFs log not enough?

[nkolban]

It should have been ... but let's compare the trace ... first the ESP32 trace:

D (17086) BLERemoteCharacteristic: >> registerForNotify()
V (17091) FreeRTOS: Semaphore taking: name: RegForNotifyEvt (0x3ffd6b90), owner: <N/A> for registerForNotify
V (17101) FreeRTOS: Semaphore taken:  name: RegForNotifyEvt (0x3ffd6b90), owner: registerForNotify
V (17109) FreeRTOS: Semaphore waiting: name: RegForNotifyEvt (0x3ffd6b90), owner: registerForNotify for registerForNotify
D (17109) BLEDevice: gattClientEventHandler [esp_gatt_if: 4] ... ESP_GATTC_REG_FOR_NOTIFY_EVT
D (17128) BLEUtils: GATT Event: ESP_GATTC_REG_FOR_NOTIFY_EVT
D (17133) BLEUtils: [status: ESP_GATT_OK, srvc_id: <uuid: 0000fff0-0000-1000-8000-00805f9b34fb, inst_id: 0>, char_id: <uuid: 0000fff1-0000-1000-8000-00805f9b34fb, inst_id: 0>]
V (17149) FreeRTOS: Semaphore released: name: RegForNotifyEvt (0x3ffd6b90), owner: registerForNotify
V (17149) FreeRTOS: Semaphore giving: name: RegForNotifyEvt (0x3ffd6b90), owner: registerForNotify
D (17158) BLERemoteCharacteristic: << registerForNotify()

and now the nRF Connect application:

V   22:58:18.518    Enabling notifications for 0000fff1-0000-1000-8000-00805f9b34fb
D   22:58:18.519    gatt.setCharacteristicNotification(0000fff1-0000-1000-8000-00805f9b34fb, true)
D   22:58:18.524    gatt.writeDescriptor(00002902-0000-1000-8000-00805f9b34fb, value=0x0100)
I   22:58:18.580    Data written to descr. 00002902-0000-1000-8000-00805f9b34fb, value: (0x) 01-00
A   22:58:18.580    "Notifications enabled" sent
V   22:58:18.604    Notifications enabled for 0000fff1-0000-1000-8000-00805f9b34fb
I   22:58:18.607    Notification received from 0000fff1-0000-1000-8000-00805f9b34fb, value: 0 bytes
I   22:58:18.609    Notification received from 0000fff1-0000-1000-8000-00805f9b34fb, value: (0x) 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-4C-16
A   22:58:18.609    "(0x) 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-4C-16" received

In this story, both the nRF Connect and the ESP32 should be doing the same things ... and they apparently (at least as I can see) are ...

Specifically, both are informing the BLE server that they are interested in receiving notifications. Both receive clean/normal response codes from the server. The nRF connect seems to show notifications being received while the ESP32 doesn't. It isn't even the case that the ESP32 is receiving events and not invoking your handler as ALL incoming events are logged to trace ... irrespective of whether or not they are delivered to your app.

I am also of the belief that if we use the nRF Connect app as a BLE Server and connect our ESP32 to it and ask the nRF Connect to send a notification, I think we are saying it does indeed show a notification arriving at the ESP32.

Obviously, none of this immediately helps us ... we can't yet point to something and say "that doesn't work"!!! It could be the ESP-IDF BLE APIs that the BLE C++ API uses, it could be the BLE C++ API contains a bug, it could be Bluedroid which is the implementation layer underneath the ESP-IDF API that is giving us problems ... or it could be something about the BLE Server device you are using (a wristband?) that is simply not transmitting to the ESP32.

Now I do have a box of wrist bands and various other sensors. It might be I have an instance of what you are physically using. I think I also hear that we don't have a "data sheet" (official/public) that can be shared for examination.

The thinking of trying bluetoothctl on Linux is to give us another data point. For example, if we used bluetoothctl on Linux and did NOT receive notification, that would be one thing ... if bluetoothctl on Linux did receive notification, that would be a different thing.

I'm also extremely open to suggestions for how we can go about making progress? I am more than willing to spend as much time as needed in the source ... and we have pretty good internal tracing inside the BLE C++ libraries ... however the absence of crash and the absence of incoming events doesn't give me much to work upon. My primary test environment for BLE is nRF Connect on Android and Bluetoothctl on Linux. For the physical peripherals I actually have (which is "some" but not many) I don't believe I have any that expose notifications (or if they do, lacking a protocol description, I haven't been able to use them).

[WALLTECH]

Ok, since it isn't working for two separate totally different peripherals, try using some from your box of peripherals and see if notifications get through to the serial monitor.

[nkolban]

All the peripherals I have appear to have no documentation for their private service UUIDs and I don't believe I have any peripherals (which are documented) which give notifications.

[WALLTECH]

Hmm, they don't even show notify characteristics through nRF? Once connected to a phone you can see all their services and characteristics and if they send notifications.

[MaksymilianWojcik]

Hi,
Could you tell me what does the line:
E (51039) BT: bta_gattc_cmpl_cback unknown conn_id = 3, ignore data
mean? Why is conn_id = 3, and why does it ignore data?
I have similiar problem in my app using BLE and can't find the anwser anywhere...

[nkolban]

Howdy @MaksymilianWojcik

Unfortunately that message is coming from deep within the ESP32 internals implementation of Bluetooth. The BLE C++ classes (this project) sits on top of the ESP-IDF BLE APIs which sits on top of a distribution of a BLE implementation called "Bluedroid". The message we are seeing come from deep within the stacks much lower than the API exposed to ESP-IDF. We don't appear to be getting any error conditions bubbled up to the ESP-IDF surface. I'm not even sure that we are able to see any adverse effects resulting from the message.

What we need to do is post to the ESP-IDF Github issues and ask Espressif "What does this mean" and attempt to seek an explanation from the Espressif folks.

[nkolban]

Closing for just now.

[inphotalk]

I was facing similar problem (ESP32 in Arduino IDE). After calling registerForNotify notifications didn't come.
After writing 0x0100 to the 0x2902 descriptor of heart rate notify characteristic everything started to work (notifications come from the Heart Rate from MIband 1s).

So I'm rather sure that the registerForNotify just sets the address of callback function without signalling this to remote descriptor.

To read the descriptor I had to modify the lib (the BLERemoteDescriptor.cpp was lacking the gattClientEventHandler).
Writing the descriptor returns error from GATT but it works.

BT: ATT - Ignore wrong response. Receives (13) Request(40) Ignored

I've found the reason of the error - in the library function for writing the descriptor had fixed ESP_GATT_WRITE_TYPE_NO_RSP - Mi band requires write with response: ESP_GATT_WRITE_TYPE_RSP

[programmer131]

@inphotalk can you please share updated BLERemoteDescriptor.cpp? I'm also facing similar issue, unable to get notifications from my device.

[chegewara]

This is my solution. Only problem is i cant confirm it will work normal ble device, because for now i could test it only with two esp32 devices (1 client and 1 server).

        BLERemoteDescriptor *desc = pRemoteCharacteristic->getDescriptor(BLEUUID((uint16_t)0x2902));
        uint8_t val[] = {0x01,0x00};
        desc->writeValue(val, 2);

And of course you need this code to turn on norifications registration in bluedroid:
pRemoteCharacteristic->registerForNotify(notifyCallback);

[programmer131]

@chegewara this is for server? right? i don't have to write anything, just have to get notifications from another device that is not based on esp32, and I'm running ble client example for esp32, have updated service uuid and characteristic uuid and esp is able to find service and characteristic, but unable to get notifications.

[chegewara]

This is for client. Server can send notifications and client can receive notifications. When you look at BLE_client example you have this function:

bool connectToServer(BLEAddress pAddress) {
....

pRemoteCharacteristic->registerForNotify(notifyCallback);

// here add my snippet code
}

[chegewara]

Probably i should have to add this to ble library in BLERemoteCharacteristic, but this will break existing applications.

[programmer131]

@chegewara people are so smart
so we need to enable notifications from Client Characteristic Configuration, now I can get and see notification data, it works like a charm,
Thanks a million.

[techno94]

@programmer131 hello
is it possible to have a sample of you're code,
i try to make an esp32 to notify aan esp32 server but it don't work

here is the issue i ask for :
https://github.com/nkolban/esp32-snippets/issues/557

thanks a lot