Pubsubclient: KeepaliveTimeout error with different brokers

[hutje]

When I connect to the MQTT server it works, but after 10 - 30 seconds the broker gives a keepalive timeout and reconnects rightaway. The loop is continuously executed.
I tried different brokers but still the same problem. Mosquitto and Aedes (Node-Red) both are set to 60 seconds by default.
So the broker expects every 60 seconds a PINGREQ.
The esp8266 are default set to 15 seconds in the PubSubClient Library.
So every 15 seconds it sends a PINGREQ. The code is without any delay so client.loop() is constantly being updated.
Why do I still get all these disconnections?

void loop(){
 reconnectWifi();
 client.loop();
}

void reconnectWifi() { //(re)connect to WiFi and MQTT Server.
  if (WiFi.status() != WL_CONNECTED && currentMillis - wifiLostTimer >= 500UL ) {
    wifiLostTimer = currentMillis;
    delay(0); //to keep wifi connection alive same als yield()
    Serial.println("Wifi connecting");
    wifiLost = true;
    mqttLost = true;
    reconnectTried++;
    if (reconnectTried == 60 && digitalRead(relaySwitch) == relayOffOutput) {
      ESP.restart();
    }
  }

  else if (WiFi.status() == WL_CONNECTED && wifiLost == true) {
    reconnectTried = 0;
    wifiLost = false;
    Serial.println("Wifi Connected!!!");
  }
  // Loop until we're reconnected
  else if (client.connected() == false && currentMillis - timeOut >= 10000UL && WiFi.status() == WL_CONNECTED) {
    mqttLost = true;
    delay(0); //to keep wifi-connection alive
    timeOut = currentMillis;
    Serial.println("Attempting MQTT connection...");
    // Attempt to connect
    Serial.println(deviceID.c_str());
    client.connect(deviceID.c_str());
  }
  else if (client.connected() == true && wifiLost == false && mqttLost == true) {
    mqttLost = false;
    Serial.println("MQTT Connected!!!");
    String rxTopic = rxTopicFirst + deviceID;
    client.subscribe(rxTopic.c_str());
    Serial.println("Subscribed to: " + rxTopic);
  }
}

[hutje]

Found that the problem is caused by bad wifi-connection. This way the package takes too long to send.
I checked the RSSI and the problems occur from -77 and higher.

[hutje]

Apparently it is caused by not able to use QoS 1 for publishing. Hope this will be available soon as there are branches that have been able to make it work.

[knolleary]

This client library is unlike to support QoS 1 publishing any time soon for all the reasons I've given every time someone asks about it.

The keepalive handling in the client should keep the connection going regardless of the qos level being published at.

We have seen some issues with client.loop being called too frequently. Putting it in a tight loop like you have means the hardware spends all of its time checking the connection which gets in the way of proper handling of the data.

Have you tried adding a small delay in your loop function - delay(100)?

[hutje]

It works good in normal circumstances but when the signal strength is -80dBm or worse, some of the packages do not arrive at the broker. That is why I was asking for QoS 1 for publishing, because then the receiver does give feedback when it arrives.

[roberthend]

I have got this keepalive problem too, it seems that the client gets stucks waiting for a pringresponse, it doesnt get one and the server terminates connection. I changed PubSubClient.cpp like this:

Line 255:
lastInActivity = lastOutActivity = millis();

And added the following line right after it:
pingOutstanding = false;

This solved the issues for me.

[thomasjungblut]

@roberthend I think I just hit the same issue. Would you mind creating a pull request maybe?

[roberthend]

@thomasjungblut yes, no problem :) https://github.com/knolleary/pubsubclient/pull/802

Happy holidays!

[thomasjungblut]

@roberthend thanks a lot :) Happy holidays to you too!

[hutje]

Why does this fix the problem? Is it a quick-fix or is it a legitimate fix that does not have any consequences?

[hutje]

Why does this fix the problem? Is it a quick-fix or is it a legitimate fix that does not have any consequences?

@roberthend Is there an explanation for this fix?
Haven't tried your fix yet.
I keep getting a timeout. My timeout is set to 25 seconds inside the code. So 25*1.5 =37.5sec. But I send my own message every 20 seconds and this message is being received everytime so there should never a keep alive message being send because of it... But there is still a random timeout every 30 minutes or so.
I get the MQTT_CONNECTION_TIMEOUT and MQTT_CONNECTION_LOST as return error from the state() as -4 and -3

[RobertH1993]

I explained it already, see https://github.com/knolleary/pubsubclient/pull/802#issuecomment-750878124

[hutje]

@roberthend I have tried your fix, but compared to the original, I get more keepalive timeouts with your fix.
I have set up a database to monitor the reconnections and the reasons by using the PubSubClient state function.
In the picture you can see the different reasons and time interval of these reconnections. They are completely random.

[Edit:] without fix of @roberthend: I have found out that with 160MHz cpu speed the package handling is better, but not perfect.
Version 1.0.3 is 160MHz with the fix.
Version 1.0.2 is 160MHz without the fix.
Reason -1 was because I did a reset of the esp8266.

It seems with the publish function it has a delay.
Also during time-sensitive package sending it has a random delay of few seconds. Sometimes it is without any delay. Would the publish begin/publish end work better with this?