Espeasy: Strange behavior with ExtraTaskSettings.TaskDeviceName

Created on 14 Jul 2018  ·  73Comments  ·  Source: letscontrolit/ESPEasy





Summarize of the problem/feature request


I am self-compiling using PLUGIN_BUILD_TESTING. While adding new features to the Nextion Plugin I noticed an unusual behavior of the global ExtraTaskSettings.TaskDeviceName var. Sometimes the name is wrong during PLUGIN_WRITE.

Expected behavior


ExtraTaskSettings.TaskDeviceName should be the name of the Plugin that is in scope.

Actual behavior


When a PLUGIN_WRITE action is being process the ExtraTaskSettings.TaskDeviceName is unpredictable. It is always the name of a defined device, but not always the correct one. What I mean is that when a PLUGIN_WRITE event is being processed by Device A, the ExtraTaskSettings.TaskDeviceName within the WRITE case state is sometimes the name of Device B or Device C, and so on.

Steps to reproduce

  1. Define several devices that have Timer and Send-to-Controller actions. For example, switches and system RSSI.
  2. Edit a device (I use Nextion) and add a log event to the PLUGIN_WRITE that outputs the plugin's actual name and the name from ExtraTaskSettings.TaskDeviceName.
  3. Perform many actions that will trigger the edited device's PLUGIN_WRITE.
  4. Watch the log as timer and rule events are handled by the other devices.
  5. The log file will sometimes show the TaskDeviceName contains the wrong name.


System configuration

At power up the ExtraTaskSettings.TaskDeviceName is always correct.
Hardware: Lolin NodeMCU V3


ESP Easy version: ESP_Easy_mega-20180613_test_ESP8266_4096, Jun-12-2018
compiled with PLUGIN_BUILD_TESTING using Arduino 1.8.5
Compiler Settings: core_esp8266_esp8266_nodemcuv2_CpuFrequency_80,FlashSize_4M3M,LwIPVariant_v2mss536,Debug_Disabled,DebugLevel_None____,FlashErase_none



ESP Easy settings/screenshots:
esp1

Rules or log data


[esp_rules.txt](https://github.com/letscontrolit/ESPEasy/files/2195117/esp_rules.txt)
Settings Stabiliy Fixed Bug
Source
picture thomastech

All 73 comments

I forgot to mention that I also use http write to the plugin. Plus local triggers that write too.

thomastech picture thomastech on 14 Jul 2018

I flagged it "Needs info" for now, since I don't know yet what kind of bug it is.
But it sounds serious enough to take a look at it and it may lead to some really strange behavior.

TD-er picture TD-er on 14 Jul 2018

Thanks for looking into it. I hope it doesn't cause you too much pain to solve.

  • Thomas
thomastech picture thomastech on 15 Jul 2018

I've updated my branch from ESPEasy_mega-20180611 to ESPEasy_mega-20180714. The symptoms are the same. Here's more information:

My test code looks like this:

    case PLUGIN_WRITE: {
        String tmpString = string;
        int argIndex = tmpString.indexOf(',');
        if (argIndex) tmpString = tmpString.substring(0, argIndex);

        String log = F("Nextion TaskDeviceName : ");
        log += ExtraTaskSettings.TaskDeviceName;
        addLog(LOG_LEVEL_INFO, log);

        if (tmpString.equalsIgnoreCase(F("NEXTION"))) {     // Use hard coded name, patch for TaskDeviceName bug.
//      if (tmpString.equalsIgnoreCase(ExtraTaskSettings.TaskDeviceName)) { // Buggy! Plugin's TaskDeviceName not working correctly.
            argIndex = string.indexOf(',');
            tmpString = string.substring(argIndex + 1);
            sendCommand(tmpString.c_str(), HwSerial);

            String log = F("NEXTION075 : WRITE, ");
            log += F("Command is ");
            log += (tmpString.c_str());
            addLog(LOG_LEVEL_INFO, log);

            success = true;                             // Set true only if plugin found a command to execute.
        }
        break;
    }

At each write event the log file outputs the Nextion TaskDeviceName. The expected name is "NEXTION".

Unfortunately the active device inherits the name of a previous device's event. Here's a log file that shows "WASHER" (a switch device on 10sec interval) incorrectly appearing as the name after a WASHER event. See 141207 & 151204.

131320 : EVENT: NEXTION#idx=16.00
131359 : ACT  : Publish /ESPEZ_Laundry/NEXTION/idx,16
131363 : Nextion TaskDeviceName : NEXTION
131371 : Nextion TaskDeviceName : NEXTION
131400 : Command: publish
131473 : EVENT: NEXTION#value=0.00
140869 : NEXTION075 : code: |s,i26,sOff
140872 : EVENT: NEXTION#idx=26.00
140909 : ACT  : Publish /ESPEZ_Laundry/NEXTION/idx,26
140914 : Nextion TaskDeviceName : NEXTION
140925 : Nextion TaskDeviceName : NEXTION
140963 : Command: publish
141040 : EVENT: NEXTION#value=0.00
141119 : NEXTION075 : Pipe Command Sent: |s,i26,sOff26.00
141129 : SW   : State 0.00
141131 : EVENT: WASHER#ac=0.00
141166 : ACT  : NEXTION,page0.va_WasherAC.val=0
141207 : Nextion TaskDeviceName : WASHER
141273 : NEXTION075 : WRITE, Command is page0.va_WasherAC.val=0
141418 : EVENT: NEXTION#idx=26.00
141458 : ACT  : Publish /ESPEZ_Laundry/NEXTION/idx,26
141462 : Nextion TaskDeviceName : NEXTION
141470 : Nextion TaskDeviceName : NEXTION
141498 : Command: publish
141572 : EVENT: NEXTION#value=0.00
148977 : SYS  : -69.00
148981 : EVENT: RSSI#signal=-69.00
150866 : NEXTION075 : code: |s,i16,sOff
150871 : EVENT: NEXTION#idx=16.00
150908 : ACT  : Publish /ESPEZ_Laundry/NEXTION/idx,16
150911 : Nextion TaskDeviceName : NEXTION
150922 : Nextion TaskDeviceName : NEXTION
150960 : Command: publish
151036 : EVENT: NEXTION#value=0.00
151115 : NEXTION075 : Pipe Command Sent: |s,i16,sOff16.00
151125 : SW   : State 0.00
151127 : EVENT: WASHER#ac=0.00
151164 : ACT  : NEXTION,page0.va_WasherAC.val=0
151204 : Nextion TaskDeviceName : WASHER
151271 : NEXTION075 : WRITE, Command is page0.va_WasherAC.val=0
151418 : EVENT: NEXTION#idx=16.00
151457 : ACT  : Publish /ESPEZ_Laundry/NEXTION/idx,16
151461 : Nextion TaskDeviceName : NEXTION
151470 : Nextion TaskDeviceName : NEXTION
151496 : Command: publish
151569 : EVENT: NEXTION#value=0.00
152274 : LoopStats: shortestLoop: 49 longestLoop: 1627788 avgLoopDuration: 95.94 systemTimerDuration: 22.00 systemTimerCalls: 31 loopCounterMax: 612244 loopCounterLast: 308134 countFindPluginId: 0
152355 : WD   : Uptime 3 ConnectFailures 0 FreeMem 14896

I also discovered that this bug is involved in a crash/brick when I delete the plugin from ESPeasy. It's a convoluted issue involving the hard coded name used as a "fix" for the TaskDeviceName and the rules that reference the device name.

My best explanation for the crash/brick is that normally when a plugin is deleted the device structure elements become null. Calls from the rule file to the deleted device's PLUGIN_WRITE still occur, but would normally be ignored because ExtraTaskSettings.TaskDeviceName would be null and not match the device name used by the rule file. But in the workaround the rule actions will attempt to execute because their name will match the name of the deleted plugin's hardcoded name. This causes a error 28 crash.

If you need anything else please let me know.

  • Thomas
thomastech picture thomastech on 19 Jul 2018

Sorry for the log file mess. For some reason the CR/LF's were stripped out in the post.
EDIT: Never mind, edited it and now it looks fine.

  • Thomas
thomastech picture thomastech on 19 Jul 2018

I just edited your comment to make it easier to read ;)

And I am looking into scheduled actions and stuff.
I just made a number of issues to help me keep track of all those changes.
I will also look into the rules handling and maybe you should also look into null-pointer dereferences.
If you keep some pointers, make sure not to dereference them when it is null.
Also, when deleting the object, make sure to set pointers to NULL.

TD-er picture TD-er on 19 Jul 2018

It looks like new name is not set sometimes

czw., 19.07.2018, 21:51 użytkownik Thomas notifications@github.com
napisał:

Sorry for the log file mess. For some reason the CR/LF's were stripped out
in the post.

  • Thomas

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/letscontrolit/ESPEasy/issues/1574#issuecomment-406393923,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHOU-uqnv5dgdWHQu_Q-TGuWsCGVmvlks5uIOM9gaJpZM4VP7jW
.

uzi18 picture uzi18 on 19 Jul 2018

I also put it into the "timing" project, since I believe the right settings may not always be active in memory when being used.
Not entirely sure if "timing" is the best description, but you get my idea ;)

TD-er picture TD-er on 19 Jul 2018

Thanks for the comments/advice. If the plugin is introducing pointer issues after it is deleted then they are hiding from me. I think this will take someone with eagle eyes to find any evil lurking in it. Here's the plugin as of today:

//#######################################################################################################
//#######################################################################################################
//################################### Plugin 075: Nextion <[email protected]>  ###########################
//###################################   Created on the work of  majklovec     ###########################
//###################################    Revisions by BertB and ThomasB       ########################### 
//###################################    Last Revision: July-19-2018 (TB)     ###########################
//#######################################################################################################

#ifdef USES_P075

#include <ESPeasySoftwareSerial.h>

// *****************************************************************************************************
// Defines start here
// *****************************************************************************************************

// Plug-In defines
#define PLUGIN_075
#define PLUGIN_ID_075 75
#define PLUGIN_NAME_075 "Display - Nextion [TEST_V3.2.1]"
#define PLUGIN_VALUENAME1_075 "idx"
#define PLUGIN_VALUENAME2_075 "value"

// Configuration Settings. Custom Configuration Memory must be less than 512 Bytes!
//#define Nlines 12       // Qty of "optional" user entered Command-Text strings. 
//#define Lenlines 64     // Length of "optional" user entered Command-Text strings.
#define Nlines 8          // Qty of "optional" user entered Command-Text strings. 
#define Lenlines 48       // Length of "optional" user entered Command-Text strings.
char deviceTemplate[Nlines][Lenlines]; 

// Nextion defines
#define RXBUFFSZ  80     // Local Serial RxD buffer.  
#define TOUCH_BASE 500   // Base offset for 0X65 Touch Event Component ID.

// Serial defines
#define B9600    0  
#define B38400   1
#define B57600   2
#define B115200  3
#define DEFAULT_BAUD B9600

// Global vars
ESPeasySoftwareSerial *SoftSerial = NULL;
int rxPin = -1;
int txPin = -1;


// *****************************************************************************************************
// PlugIn starts here
// *****************************************************************************************************

boolean Plugin_075(byte function, struct EventStruct *event, String& string) 
{
  boolean success = false;
  static boolean HwSerial = false;
  static boolean AdvHwSerial = false;  
  uint32_t AdvHwBaud = 9600UL;

  switch (function) {

    case PLUGIN_DEVICE_ADD: {
      Device[++deviceCount].Number = PLUGIN_ID_075;
      Device[deviceCount].Type = DEVICE_TYPE_DUAL;
      Device[deviceCount].VType = SENSOR_TYPE_DUAL;
      Device[deviceCount].Ports = 0;
      Device[deviceCount].PullUpOption = true;
      Device[deviceCount].InverseLogicOption = false;
      Device[deviceCount].FormulaOption = false;
      Device[deviceCount].ValueCount = 2;
      Device[deviceCount].SendDataOption = true;
      Device[deviceCount].TimerOption = true;
      Device[deviceCount].TimerOptional = true;             // Allow user to disable interval function.
      Device[deviceCount].GlobalSyncOption = true;
      break;
    }


    case PLUGIN_GET_DEVICENAME: {
      string = F(PLUGIN_NAME_075);
      break;
    }


    case PLUGIN_GET_DEVICEVALUENAMES: {
      strcpy_P(ExtraTaskSettings.TaskDeviceValueNames[0],PSTR(PLUGIN_VALUENAME1_075));
      strcpy_P(ExtraTaskSettings.TaskDeviceValueNames[1],PSTR(PLUGIN_VALUENAME2_075));
      break;
    }


    case PLUGIN_GET_DEVICEGPIONAMES: {

      AdvHwSerial = Settings.TaskDevicePluginConfig[event->TaskIndex][0];
      rxPin = Settings.TaskDevicePin1[event->TaskIndex];
      txPin = Settings.TaskDevicePin2[event->TaskIndex];

      event->String1 = F("GPIO SS RX &larr; ");
      event->String2 = F("GPIO SS TX &rarr; ");

      if(AdvHwSerial == true) {
        if ((rxPin == 3 && txPin == 1) || (rxPin == 13 && txPin == 15)) {
            event->String1 = F("GPIO HW RX &larr; ");
            event->String2 = F("GPIO HW TX &rarr; ");
        }
      }
      break;
    }


    case PLUGIN_WEBFORM_LOAD: {
      rxPin = Settings.TaskDevicePin1[event->TaskIndex];
      txPin = Settings.TaskDevicePin2[event->TaskIndex];

      if (!((rxPin == 3 && txPin == 1) || (rxPin == 13 && txPin == 15))) { // Hardware Serial Compatible?
        Settings.TaskDevicePluginConfig[event->TaskIndex][0] = false;      // Not HW serial compatible, Reset Check Box.
      }

      if (rxPin == 3 && txPin == 1) {                                      // USB Port?
        if(Settings.TaskDevicePluginConfig[event->TaskIndex][0]==false &&  // Hardware serial disabled.
         Settings.TaskDeviceEnabled[event->TaskIndex] == true) {           // Plugin is enabled.
            Settings.TaskDevicePluginConfig[event->TaskIndex][0]=true;     // USB port access uses HW serial, Force set Check Box.
        }
      }

      if (Settings.TaskDevicePluginConfig[event->TaskIndex][0] == false) { // Softserial mode. 
        Settings.TaskDevicePluginConfig[event->TaskIndex][1] = B9600;      // Reset to 9600 baud.
      }

      addFormSeparator(2);
      addFormSubHeader(F("Enhanced Serial Communication"));
      addFormCheckBox(F("Use Hardware Serial"), F("AdvHwSerial"), Settings.TaskDevicePluginConfig[event->TaskIndex][0]);

      byte choice = Settings.TaskDevicePluginConfig[event->TaskIndex][1];
      String options[4];
      options[0] = F("9600");
      options[1] = F("38400");
      options[2] = F("57600");
      options[3] = F("115200");

      addFormSelector(F("Baud Rate"), F("plugin_075_baud"), 4, options, NULL, choice);      
      addFormNote(F("Un-check box for Soft Serial communication (low performance mode, 9600 Baud)."));
      addFormNote(F("Hardware Serial is available when the GPIO pins are RX=D7 and TX=D8."));
      addFormNote(F("D8 (GPIO-15) requires a Buffer Circuit (PNP transistor) or ESP boot may fail."));
      addFormNote(F("Do <b>NOT</b> enable the Serial Log file on Tools->Advanced->Serial Port."));

//    ** DEVELOPER DEBUG MESSAGE AREA **
//    addFormNote(ExtraTaskSettings.TaskDeviceName);    // Debug value.
//    int datax = (int)(Settings.TaskDeviceEnabled[event->TaskIndex]); // Debug value.
//    String Data = "Debug. Plugin Enable State: ";
//    Data += String(datax);
//    addFormNote(Data);


      addFormSubHeader(F("")); // Blank line, vertical space.
      addFormHeader(F("Nextion Command-Text Strings (Optional)"));

      char deviceTemplate[Nlines][Lenlines];
      LoadCustomTaskSettings(event->TaskIndex, (byte*)&deviceTemplate, sizeof(deviceTemplate));
      for (byte varNr = 0; varNr < Nlines; varNr++) {
        addFormTextBox(String(F("Line ")) + (varNr + 1), String(F("Plugin_075_template")) + (varNr + 1), deviceTemplate[varNr], Lenlines-1);

      }
      if( Settings.TaskDeviceTimer[event->TaskIndex]==0) {
        addFormNote(F("Interval Timer OFF: Nextion Lines <b>NOT</b> scheduled for updates."));
      }
      else {
        addFormNote(F("Interval Timer On: Nextion Lines (above) scheduled for updates."));
      }


      success = true;
      break;
    }


    case PLUGIN_WEBFORM_SAVE: {

        String argName;

        char deviceTemplate[Nlines][Lenlines];
        for (byte varNr = 0; varNr < Nlines; varNr++)
        {
          String arg = F("Plugin_075_template");
          arg += varNr + 1;
          String tmpString = WebServer.arg(arg);
          strncpy(deviceTemplate[varNr], tmpString.c_str(), sizeof(deviceTemplate[varNr])-1);
            deviceTemplate[varNr][Lenlines-1]=0;
        }
        if(ExtraTaskSettings.TaskDeviceName[0] == 0) {          // User forgot to enter device name!
            strcpy(ExtraTaskSettings.TaskDeviceName,"NEXTION"); // Give standard name.
        }
        Settings.TaskDevicePluginConfig[event->TaskIndex][0] = isFormItemChecked(F("AdvHwSerial"));
        Settings.TaskDevicePluginConfig[event->TaskIndex][1] = getFormItemInt(F("plugin_075_baud"));
        SaveCustomTaskSettings(event->TaskIndex, (byte*)&deviceTemplate, sizeof(deviceTemplate));

        success = true;
        break;
    }


    case PLUGIN_INIT: {

      AdvHwSerial = Settings.TaskDevicePluginConfig[event->TaskIndex][0];
      uint8_t BaudCode = Settings.TaskDevicePluginConfig[event->TaskIndex][1];
      Settings.TaskDevicePluginConfig[event->TaskIndex][2] = -999;      // Reset Last known idx.
      Settings.TaskDevicePluginConfig[event->TaskIndex][3] = -999;      // Reset Last known value.

      if(BaudCode > B115200) BaudCode = B9600;
      const uint32_t BaudArray[4] = {9600UL, 38400UL, 57600UL, 115200UL};
      AdvHwBaud = BaudArray[BaudCode];

      if (Settings.TaskDevicePin1[event->TaskIndex] != -1) {
        rxPin = Settings.TaskDevicePin1[event->TaskIndex];
      }
      if (Settings.TaskDevicePin2[event->TaskIndex] != -1) {
        txPin = Settings.TaskDevicePin2[event->TaskIndex];
      }

      if (SoftSerial != NULL) { 
        delete SoftSerial;
        SoftSerial = NULL;
      }

      String log = F("NEXTION075 : serial pin config RX:");
      log += rxPin;
      log += F(", TX:");
      log += txPin;
      if(Settings.TaskDeviceEnabled[event->TaskIndex]==true) 
       log += F(", Plugin Enabled");                    // Plugin is enabled.
      else log += F(", Plugin Disabled");
      addLog(LOG_LEVEL_INFO, log);

      if(Settings.TaskDeviceEnabled[event->TaskIndex] == true) { // Plugin is enabled.
      // Hardware serial is RX on 13 and TX on 15 (swapped hw serial)
        if (AdvHwSerial &&  rxPin == 13 && txPin == 15) {
            log = F("NEXTION075 : Using swap hardware serial");
            addLog(LOG_LEVEL_INFO, log);
            HwSerial = true;
            Settings.UseSerial = false;                 // Disable global Serial port.
            Settings.SerialLogLevel = 0;                // Disable logging on serial port.
            Settings.BaudRate = AdvHwBaud;              // Set BaudRate for Nextion.
            Serial.flush();
            Serial.begin(AdvHwBaud);
            Serial.swap();
        }
        // Hardware serial is RX on 3 and TX on 1. USB serial for Nextion IDE (User MCU Input function).
        else if(AdvHwSerial && rxPin == 3 && txPin == 1) {
            log = F("NEXTION075 : Using USB hardware serial");
            addLog(LOG_LEVEL_INFO, log);
            HwSerial = true;
            Settings.UseSerial = false;                 // Disable global Serial port.
            Settings.SerialLogLevel = 0;                // Disable logging on serial port.
            Settings.BaudRate = AdvHwBaud;              // Set BaudRate for Nextion.
            Serial.flush();
            Serial.begin(AdvHwBaud);
        }
        else {
            log = F("NEXTION075 : Using software serial");
            addLog(LOG_LEVEL_INFO, log);
            HwSerial = false;
            if (SoftSerial == NULL) {
                SoftSerial = new ESPeasySoftwareSerial(rxPin, txPin);
            } 
            SoftSerial->begin(9600);
            SoftSerial->flush();
        }
    }
    else {
    }
      success = true;
      break;
    }


    case PLUGIN_READ: {    // Get Plugin's optional command-text strings. Special RSSIBAR bargraph keyword is supported.
        char deviceTemplate[Nlines][Lenlines];
        int RssiIndex;
        String newString;
        String tmpString;
        String UcTmpString;

        LoadCustomTaskSettings(event->TaskIndex, (byte*)&deviceTemplate, sizeof(deviceTemplate));

        for (byte x = 0; x < Nlines; x++) {
          tmpString = deviceTemplate[x];
          if (tmpString.length()) {
            UcTmpString = deviceTemplate[x];
            UcTmpString.toUpperCase();
            RssiIndex = UcTmpString.indexOf(F("RSSIBAR"));  // RSSI bargraph Keyword found, wifi value in dBm.
            if(RssiIndex >= 0) {
              int barVal;
              newString = tmpString.substring(0, RssiIndex);
              int nbars = WiFi.RSSI();
              if (nbars < -100 || nbars >= 0)
                 barVal=0;
              else if (nbars >= -100 && nbars < -95)
                 barVal=5;
              else if (nbars >= -95 && nbars < -90)
                 barVal=10;
              else if (nbars >= -90 && nbars < -85)
                 barVal=20;
              else if (nbars >= -85 && nbars < -80)
                 barVal=30;
              else if (nbars >= -80 && nbars < -75)
                 barVal=45;
              else if (nbars >= -75 && nbars < -70)
                 barVal=60;
              else if (nbars >= -70 && nbars < -65)
                 barVal=70;
              else if (nbars >= -65 && nbars < -55)
                 barVal=80;
              else if (nbars >= -55 && nbars < -50)
                 barVal=90;
              else if (nbars >= -50)
                 barVal=100;

              newString += String(barVal,DEC);
            }
            else {
              newString = parseTemplate(tmpString, 0);
            }

            sendCommand(newString.c_str(), HwSerial);
            success = true;
          }
        }

        // At Interval timer, send idx & value data only if either changed. This eliminates repeating the data.
        if ((Settings.TaskDevicePluginConfig[event->TaskIndex][2] != UserVar[event->BaseVarIndex]) ||
          (Settings.TaskDevicePluginConfig[event->TaskIndex][3] != UserVar[event->BaseVarIndex+1])) {
            Settings.TaskDevicePluginConfig[event->TaskIndex][2] = UserVar[event->BaseVarIndex];  // Save new idx.
            Settings.TaskDevicePluginConfig[event->TaskIndex][3] = UserVar[event->BaseVarIndex+1]; // Save new value.
            success = true;
        } 
        else {
            success = false;
        }

        break;
    }


    case PLUGIN_WRITE: {

        String tmpString = string;
        int argIndex = tmpString.indexOf(',');
        if (argIndex) tmpString = tmpString.substring(0, argIndex);

        String log = F("Nextion TaskDeviceName : ");
        log += ExtraTaskSettings.TaskDeviceName;
        addLog(LOG_LEVEL_INFO, log);

        if (tmpString.equalsIgnoreCase(F("NEXTION"))) {     // Use hard coded name, patch for TaskDeviceName bug.
//      if (tmpString.equalsIgnoreCase(ExtraTaskSettings.TaskDeviceName)) { // Buggy! Plugin's TaskDeviceName not working correctly.
            argIndex = string.indexOf(',');
            tmpString = string.substring(argIndex + 1);
            sendCommand(tmpString.c_str(), HwSerial);

            String log = F("NEXTION075 : WRITE, ");
            log += F("Command is ");
            log += (tmpString.c_str());
            addLog(LOG_LEVEL_INFO, log);

            success = true;                             // Set true only if plugin found a command to execute.
        }
        break;
    }


    case PLUGIN_EXIT: {
        if (SoftSerial) {
            delete SoftSerial;
            SoftSerial=NULL;
        }

        if(HwSerial) {
            HwSerial = false;
            Settings.UseSerial      = DEFAULT_USE_SERIAL;
            Settings.BaudRate       = DEFAULT_SERIAL_BAUD;
            Serial.flush();
            Serial.begin(DEFAULT_SERIAL_BAUD);          // Restart Serial Logging with default baud.
        }
        break;
    }


    case PLUGIN_ONCE_A_SECOND: {
        success = true;
        break;
    }


    case PLUGIN_TEN_PER_SECOND: {
      uint16_t i;
      uint8_t c;
      uint8_t charCount;
      String log;
      String Vidx;
      String Nvalue;
      String Svalue;
      String Nswitch;
      char __buffer[RXBUFFSZ+1];

      if(HwSerial) charCount = Serial.available();      // Prime the Hardware Serial engine.
      else charCount = SoftSerial->available();         // Prime the Soft Serial engine.

      while (charCount) {                               // This is the serial engine. It processes the serial Rx stream.
        if(HwSerial) c = Serial.read();
        else c = SoftSerial->read();

        if (c == 0x65) {
          if (charCount < 6) delay((5/(AdvHwBaud/9600))+1); // Let's wait for a few more chars to arrive.

          if (HwSerial) charCount = Serial.available();
          else charCount = SoftSerial->available();
          if (charCount >= 6) {
            __buffer[0] = c;
            for (i = 1; i < 7; i++) {
                if(HwSerial) __buffer[i] = Serial.read();
                else __buffer[i] = SoftSerial->read();
            }

            __buffer[i] = 0x00;

            if (0xFF == __buffer[4] && 0xFF == __buffer[5] && 0xFF == __buffer[6]) {
              UserVar[event->BaseVarIndex] = (__buffer[1] * 256) + __buffer[2] + TOUCH_BASE;
              UserVar[event->BaseVarIndex + 1] = __buffer[3];
              log = F("NEXTION075 : code: ");
              log += __buffer[1];
              log += ",";
              log += __buffer[2];
              log += ",";
              log += __buffer[3];
              addLog(LOG_LEVEL_INFO, log);

              sendData(event);
            }
          }
        } 
        else {
          if (c == '|') {
            __buffer[0] = c;

            if (charCount < 8) delay((9/(AdvHwBaud/9600))+1); // Let's wait for more chars to arrive.
            else delay((3/(AdvHwBaud/9600))+1);               // Short wait for tardy chars.
            if (HwSerial) charCount = Serial.available();
            else charCount = SoftSerial->available();

            if(HwSerial) {
                i = 1;            
                while (Serial.available() > 0 && i<RXBUFFSZ) {  // Copy global serial buffer to local buffer.
                  __buffer[i] = Serial.read();
                  if (__buffer[i]==0x0a || __buffer[i]==0x0d) break;
                  i++;
                }
            }
            else {
                i = 1;            
                while (SoftSerial->available() > 0 && i<RXBUFFSZ) {  // Copy global serial buffer to local buffer.
                  __buffer[i] = SoftSerial->read();
                  if (__buffer[i]==0x0a || __buffer[i]==0x0d) break;
                  i++;
                }
            }

            __buffer[i] = 0x00;

            String tmpString = __buffer;
            log = F("NEXTION075 : code: ");
            log += tmpString;
            addLog(LOG_LEVEL_INFO, log);

            int argIndex = tmpString.indexOf(F(",i"));
            int argEnd = tmpString.indexOf(',', argIndex + 1);
            if (argIndex) Vidx = tmpString.substring(argIndex + 2,argEnd);

            boolean GotPipeCmd = false;
            switch (__buffer[1]){
              case 'u':
                GotPipeCmd = true;
                argIndex = argEnd;
                argEnd = tmpString.indexOf(',',argIndex + 1);
                if (argIndex) Nvalue = tmpString.substring(argIndex + 2,argEnd);
                argIndex = argEnd;
                argEnd = tmpString.indexOf(0x0a);
                if (argIndex) Svalue = tmpString.substring(argIndex + 2,argEnd);
                break;
              case 's':
                GotPipeCmd = true;
                argIndex = argEnd;
                argEnd = tmpString.indexOf(0x0a);
                if (argIndex) Nvalue = tmpString.substring(argIndex + 2,argEnd);
                if (Nvalue == F("On")) Svalue='1';
                if (Nvalue == F("Off")) Svalue='0';
                break;
            }

            if (GotPipeCmd) {
                UserVar[event->BaseVarIndex] = Vidx.toFloat();
                UserVar[event->BaseVarIndex+1] = Svalue.toFloat();
                sendData(event);
                log = F("NEXTION075 : Pipe Command Sent: ");
                log += __buffer;
                log += UserVar[event->BaseVarIndex];
            }
            else {
                log = F("NEXTION075 : Unknown Pipe Command, skipped");
            }
            addLog(LOG_LEVEL_INFO, log);
          }
        }
        if(HwSerial) charCount = Serial.available();
        else charCount = SoftSerial->available();
      }

      success = true;
      break;
    }
  }
  return success;
}


void sendCommand(const char *cmd, boolean SerialHW) 
{
    if(SerialHW) {
        Serial.print(cmd);
        Serial.write(0xff);
        Serial.write(0xff);
        Serial.write(0xff);
    }
    else {
        SoftSerial->print(cmd);
        SoftSerial->write(0xff);
        SoftSerial->write(0xff);
        SoftSerial->write(0xff);
    }
}

#endif // USES_P075
  • Thomas
thomastech picture thomastech on 19 Jul 2018

For example in case PLUGIN_TEN_PER_SECOND: {
There you only check for either hw serial or software serial.
But when entering that switch-case statement you can also check if software serial is set and if so, check if it is NULL.
If it is NULL, you will now dereference the null pointer which is a crash.
So add some check like if (SoftSerial) {, or my preferred way:
if (SoftSerial == NULL) break;

TD-er picture TD-er on 19 Jul 2018

Good catch, thanks for finding that! I've looked over the code again (hoping to find more such nuggets), and made the edit. ESPeasy still hates me because the crash/brick got me again. So the evil is still lurking in the shadows.

After struggling with this problem for a couple weeks I am certain the issue is with the presence of the hard-coded device name in the PLUGIN_WRITE.

As a reference point, I can edit my rule file and delete or change all the NEXTION names (in the rule statements) , then delete the entire plugin, and the crash/brick never occurs. I can also remove the hard coded name from the PLUGIN_WRITE and use ExtraTaskSettings.TaskDeviceName var instead and the nasty crash/brick is averted.

Perhaps a different way to approach this is to change the event process so that it never allows a rule to be passed into a deleted or disabled plugin.

I apologize for overloading the original reported issue with the crash/brick problem. But assuming I'm not dazed and confused, fixing the device name bug should fix the crash/brick. Two birds, one stoney patch.

  • Thomas
thomastech picture thomastech on 19 Jul 2018

I will certainly look into the enabled/disabled case for plugins and rule-commands.

TD-er picture TD-er on 20 Jul 2018

I hate reporting problems without offering solutions. But this issue is above my pay grade. For sure, if you need anything else from me then just ask.

  • Thomas
thomastech picture thomastech on 20 Jul 2018

I've dug into the code and it has been an interesting journey. Painful too, since my only debugging tool is addLog().

What I've found is that there are many calls in the core to LoadTaskSettings(). This reads the SPIFFS data and changes the device name and other device data. Some functions perform a fresh reload to stay in sync. But not all functions have done this and that is why the ExtraTaskSettings.TaskDeviceName value becomes incorrect.

I've identified one area that invites the device name sync problem. It's the PluginCall() function the _Plugin,ino file. In the PLUGIN_WRITE case, I've added this after START_TIMER;
LoadTaskSettings(y); // Load SPIFF settings data to refresh device name.

This has solved my observed device name problem and my plugin now works correctly. But my syslog debugging messages show another random device name problem. I need to hunt that one down too.

But I think using LoadTaskSettings() to restore the device name is a bit heavy handed and it won't protect against the problem in future releases. So rather than make changes to the system modules, I would like to determine TaskDeviceName from the *event structure that is passed into a plugin.

I expect that this is a bit tricky to do. So I could use some code examples on how the pros would do it. Any help on this would be fantastic!

  • Thomas
thomastech picture thomastech on 28 Jul 2018

Maybe this helps a bit:
ESPeasy wiki - Settings storage layout

I made it a while ago, since I got stuck in this mess of settings too.
Creating a proper settings layout is one thing. Using proper (short and describing) names is the most difficult part of programming.

I will make a 'quick' fix for this today.
I will look into merging all loads together in a staged manner and hope it will not affect performance too much.

TD-er picture TD-er on 28 Jul 2018

OK, I am now looking into uses of ExtraTaskSettings and since it is a single struct and not an array of these, you have to call LoadTaskSettings(event->TaskIndex); (or whatever variable holds your task index you're interested in) every time you plan on using it.

But a simple search does indicate this is simply not done in a lot of cases.
So I will take some coffee now, take a deep breath and start thinking about either adding a load on every use of ExtraTaskSettings or create functions (and some cache) to get those values needed.
This is going to be a big change in code, so I will try to split it into several commits to make sure it can be verified.

TD-er picture TD-er on 28 Jul 2018
👍1

Maybe this helps a bit:
ESPeasy wiki - Settings storage layout

Very helpful. Thanks!

you have to call LoadTaskSettings(event->TaskIndex); (or whatever variable holds your task index you're interested in) every time you plan on using it.

Unfortunately the event->TaskIndex var can't be trusted either since it suffers the same issue as ExtraTaskSettings.TaskDeviceName. But I should be able to get the event number from the struct that is passed by PluginCall() and use it to LoadTaskSettings(). If this works then it can be my short term workaround for use in my plugin.

So I will take some coffee now, take a deep breath and start thinking about either adding a load on every use of ExtraTaskSettings or create functions (and some cache) to get those values needed.

You will need a lot of coffee to go down that rabbit hole.

While you are looking at the code it would be great if you also kept an eye out for PluginCall()'s tendency to access deleted plugins. It seems to suggest that the plugin delete function doesn't fully null out everything in the deleted device's structures. Or maybe some functions using PluginCall() simply ignore all the helpful indications that a plugin is disabled/deleted.

  • Thomas
thomastech picture thomastech on 28 Jul 2018

One note for future propose: have changed offsets for data structures in config file and now it is possible to store 24 tasks, but not sure if we are ready to bump config version?

uzi18 picture uzi18 on 28 Jul 2018

@thomastech I was working on it this afternoon and if you like, you can already have a peek here: https://github.com/TD-er/ESPEasy/commit/28831be000a497fb990969bc412bab631450f66e

The main changes are in the handling of storage and a first attempt to make a single function for getting values from ExtraTaskSettings.
This handling function should also manage loading the settings and I will add some cache later in these functions.
Also the settings will be properly cleared now.

@uzi18 I added some lookup-functions to compute the offset.
These functions can be used to add a new offset for later to add more task settings.
But first let's make sure no storage blocks overlap.

TD-er picture TD-er on 28 Jul 2018

but I have rearrange data, now first are Settings, next controllers and extra controllers, next task.
So it is possible to extend task to more than 12...

uzi18 picture uzi18 on 28 Jul 2018

It is also strange how. often espeasy load and save data into spiffs.

uzi18 picture uzi18 on 28 Jul 2018

@uzi18 It is not really strange if you look at how it is using the ExtraTaskSettings
That single struct is the only cache of those values and these are needed a lot, since they contain:

  • char TaskDeviceName[NAME_FORMULA_LENGTH_MAX + 1];
  • char TaskDeviceFormula[VARS_PER_TASK][NAME_FORMULA_LENGTH_MAX + 1];
  • char TaskDeviceValueNames[VARS_PER_TASK][NAME_FORMULA_LENGTH_MAX + 1];
  • long TaskDevicePluginConfigLong[PLUGIN_EXTRACONFIGVAR_MAX];
  • byte TaskDeviceValueDecimals[VARS_PER_TASK];
  • int16_t TaskDevicePluginConfig[PLUGIN_EXTRACONFIGVAR_MAX];
TD-er picture TD-er on 28 Jul 2018

Thanks for the link, I checked it out.

Along with all the new goodness, I noticed this:

define DAT_TASKS_CUSTOM_SIZE 1024

But the wiki says its only 512.
https://www.letscontrolit.com/wiki/index.php/ESPEasyDevelopment
See "Configuration storage" section:
_A controller plugin can have custom configuration up to 4096 bytes. A device plugin can only use 512 bytes for it's custom configuration._

Because of the wiki I reduced the size of the custom configuration in my customized plugin to be less than 512 bytes. If 1024 is allowed I can changed it back to what it used to be. Can you confirm which is correct, 512 or 1024?

  • Thomas
thomastech picture thomastech on 28 Jul 2018

@thomastech
The wiki information is for sure outdated, but I am not sure how outdated.
At least it states we are not using SPIFF, but we currently are.

I will try to make a dump of the current storage and see if there is some overlap in the blocks.
Also including how much we are actually using.

TD-er picture TD-er on 29 Jul 2018

@TD-er
Thanks, I appreciate your helpfulness.

  • Thomas
thomastech picture thomastech on 29 Jul 2018

I just made some dump of how the data is stored (on ESP8266)

Struct | sizeof()
--- | ---
SecurityStruct | 593
SettingsStruct | 1220
ExtraTaskSettingsStruct| 472
DeviceStruct| 714

Notes:

  • ExtraTaskSettings is stored as TaskSettings
  • DeviceStruct is stored in memory as an array: Device[DEVICES_MAX + 1]; (ran at "normal" build)
  • SettingsStruct is stored at the start of the "config.dat" file.
  • Almost all settings below are stored in the "config.dat" file.
  • struct_size in the tables below is 0 when it may vary per use case.

TaskSettings | start| end| max_size | struct_size
--- | --- | --- | --- | ---
0|4096|5120|1024|472
1|6144|7168|1024|472
2|8192|9216|1024|472
3|10240|11264|1024|472
4|12288|13312|1024|472
5|14336|15360|1024|472
6|16384|17408|1024|472
7|18432|19456|1024|472
8|20480|21504|1024|472
9|22528|23552|1024|472
10|24576|25600|1024|472
11|26624|27648|1024|472

CustomTaskSettings | start| end| max_size | struct_size
--- | --- | --- | --- | ---
0|5120|6144|1024|0
1|7168|8192|1024|0
2|9216|10240|1024|0
3|11264|12288|1024|0
4|13312|14336|1024|0
5|15360|16384|1024|0
6|17408|18432|1024|0
7|19456|20480|1024|0
8|21504|22528|1024|0
9|23552|24576|1024|0
10|25600|26624|1024|0
11|27648|28672|1024|0

ControllerSettings | start| end| max_size | struct_size
--- | --- | --- | --- | ---
0|28672|29696|1024|724
1|29696|30720|1024|724
2|30720|31744|1024|724

CustomControllerSettings| start| end| max_size | struct_size
--- | --- | --- | --- | ---
0|32768|33792|1024|0
1|33792|34816|1024|0
2|34816|35840|1024|0

And in a different file (Notifications):

NotificationSettings | start| end| max_size | struct_size
--- | --- | --- | --- | ---
0|0|1024|1024|996
1|1024|2048|1024|996
2|2048|3072|1024|996

TD-er picture TD-er on 29 Jul 2018
👍1

@TD-er:
Thanks for the data storage memory map. Since it shows 1KB is available I will increase the CustomTaskSettings used in the plugin I am working on.

  • Thomas
thomastech picture thomastech on 29 Jul 2018 
diff --git a/src/ESPEasy-Globals.h b/src/ESPEasy-Globals.h
index 5cab97b..f0a2a66 100644
--- a/src/ESPEasy-Globals.h
+++ b/src/ESPEasy-Globals.h
@@ -279,7 +279,7 @@
 #endif

 #if defined(ESP8266)
-  #define TASKS_MAX                          12 // max 12!
+  #define TASKS_MAX                          24 // max 12!
 #endif
 #if defined(ESP32)
   #define TASKS_MAX                          32
@@ -296,7 +296,9 @@
 #define CPLUGIN_MAX                        16
 #define NPLUGIN_MAX                         4
 #define UNIT_MAX                           32 // Only relevant for UDP unicast message 'sweeps' and the nodelist.
-#define RULES_TIMER_MAX                     8
+#define RULES_TIMER_MAX                    24
+#define SYSTEM_TIMER_MAX                    8
+#define SYSTEM_CMD_TIMER_MAX                2
 #define PINSTATE_TABLE_MAX                 32
 #define RULES_MAX_SIZE                   2048
 #define RULES_MAX_NESTING_LEVEL             3
@@ -344,17 +346,20 @@
 #define BOOT_CAUSE_DEEP_SLEEP               2
 #define BOOT_CAUSE_EXT_WD                  10

-#define DAT_TASKS_SIZE                   2048
-#define DAT_TASKS_CUSTOM_OFFSET          1024
-#define DAT_TASKS_CUSTOM_SIZE            1024
+#define DAT_TASKS_SIZE                   2048 // 0x0800
+#define DAT_TASKS_CUSTOM_OFFSET          1024 // 0x0400
+#define DAT_TASKS_CUSTOM_SIZE            1024 // 0x0400
 #define DAT_CUSTOM_CONTROLLER_SIZE       1024
 #define DAT_CONTROLLER_SIZE              1024
 #define DAT_NOTIFICATION_SIZE            1024

 #if defined(ESP8266)
-  #define DAT_OFFSET_TASKS                 4096  // each task = 2k, (1024 basic + 1024 bytes custom), 12 max
-  #define DAT_OFFSET_CONTROLLER           28672  // each controller = 1k, 4 max
-  #define DAT_OFFSET_CUSTOM_CONTROLLER    32768  // each custom controller config = 1k, 4 max.
+//  #define DAT_OFFSET_TASKS                 4096  // 0x1000 each task = 2k, (1024 basic + 1024 bytes custom), 12 max
+//  #define DAT_OFFSET_CONTROLLER           28672  // 0x7000 each controller = 1k, 4 max
+//  #define DAT_OFFSET_CUSTOM_CONTROLLER    32768  // 0x8000 each custom controller config = 1k, 4 max.
+  #define DAT_OFFSET_CONTROLLER            4096    // 0x1000 each controller = 1k, 4 max
+  #define DAT_OFFSET_CUSTOM_CONTROLLER     8192    // 0x2000 each custom controller config = 1k, 4 max.
+  #define DAT_OFFSET_TASKS                16384    // 0x4000 each task = 2k, (1024 basic + 1024 bytes custom), 12 max
   #define CONFIG_FILE_SIZE                65536
 #endif
 #if defined(ESP32)
uzi18 picture uzi18 on 29 Jul 2018

The *_TIMER_MAX values will be removed, since the new scheduler doesn't have a maximum for timers.
And I want to get this fixed first before breaking all compatibility with other stored settings.
Increasing the number of tasks also affects the struct with normal settings and has effect on memory consumption.

So I would like to do that (increasing number of tasks) as a separate issue, after things get stable again.
We could also think about other ways to store the settings.

TD-er picture TD-er on 29 Jul 2018

@TD-er so my map is different and task are at the end.

0----------------4k------------------8k------------------------16k-----------64k
|SettingsStruct |ControllerSettings |CustomControllerSettings |TaskSettings |
0----------------------------------------------------------------------------64k
uzi18 picture uzi18 on 29 Jul 2018
👍1

So I need to look into your new implementation of timers :)

uzi18 picture uzi18 on 29 Jul 2018
👍1

@TD-er:
A question about the proposed changes. In the previous releases we could do something like this:

case PLUGIN_WEBFORM_SAVE: {
  if (ExtraTaskSettings.TaskDeviceName[0] == 0) {          // User forgot to enter device name!
     strcpy(ExtraTaskSettings.TaskDeviceName,"DEFAULT_NAME_HERE");  // Give standard name.
  }
//more code here..
}

With your new method, is the equivalent something like this?

case PLUGIN_WEBFORM_SAVE: {
  if (getTaskDeviceName(event->TaskIndex) == "") {  
      strcpy(getTaskDeviceName(event->TaskIndex),"DEFAULT_NAME_HERE");  // Need correct syntax for this!
  }
//more code here..
}

Or should a companion function be added to your change log that safely sets the Device Name? A function prototype like this: setTaskDeviceName(byte event, String& string);

Thomas
thomastech picture thomastech on 29 Jul 2018

I have not changed all access to the TaskDeviceName.
It is still a bit in between, since a function like that is no guarantee.
Also a lot of plugin calls are now verified to have the correct ExtraTaskSettings active. (unless it is changed by some plugin call)

TD-er picture TD-er on 29 Jul 2018

I added a pull request for this: https://github.com/letscontrolit/ESPEasy/pull/1618
If you could test it, would be great.

TD-er picture TD-er on 29 Jul 2018

@TD-er:
I will test it out. Please let me know when the revised mega package is available to download from github.

  • Thomas
thomastech picture thomastech on 29 Jul 2018

@TD-er:
A couple hours ago I built the latest commit (Build 20102-Mega) with my modified Nextion plugin using the PLUGIN_BUILD_TESTING compile directive. Furthermore, I updated my Nextion plugin to use your new string methods.

Some interesting observations to report.

  1. Increasing the custom configuration to use 768 bytes causes the WiFi connection to be dropped every few seconds. But decreasing it to 510 bytes works Ok. I've tried a clean flash, problem remains. I believe this issue haunted me with the older releases too. The 510 byte usage has been problem free.

  2. I'm still seeing a "zombie" device name appearing in my logs when Nextion does a PLUGIN_WRITE. Specifically TaskIndex 0 is appearing. It's a completely disabled Switch Input device with no interval (0 secs) and "send to controller" is unchecked.

A screenshot of my device list is shown below.

  • Thomas

esp2

thomastech picture thomastech on 31 Jul 2018

I can have a look at your code of the Nextion to see if I can see what's causing this.
Maybe you can also send me the Nextion config + config of your setup. (dropbox link via private message on forum or via email is also fine)
I do have a Nextion here, but never connected it, so that's one thing to setup too.

TD-er picture TD-er on 31 Jul 2018

I zipped it all up and emailed it to you.

  • Thomas
thomastech picture thomastech on 1 Aug 2018

@TD-er:
I thought it would be wise to move the email discussion back here so the conversation is easier to track. I have more information that may help pinpoint the problem with the incorrect TaskDeviceName.

First, it appears your recent test commit has improved the TaskDeviceName issues, since only one specific occurrence of the problem remains. As I mentioned in the email, the zombie TaskDeviceName is from Task#1 (index 0), which is a disabled Switch Input named DRYER.

The expected TaskDeviceName is NEXTION. Here is what a good log entry looks like:
Nextion TaskDeviceName: NEXTION, TaskIndex: 4, event->TaskIndex: 4, cmd str: Publish /ESPEZ_Laundry/NEXTION/idx,26

Here is an example of a zombie TaskDeviceName log entry:
Nextion TaskDeviceName: DRYER, TaskIndex: 0, event->TaskIndex: 0, cmd str: Publish /ESPEZ_Laundry/NEXTION/idx,26

In the Nextion.ino I emailed to you will find that the PLUGIN_WRITE has some debugging code in it that does a LoadTaskSettings(). Rather than rely on the globally defined data, it uses the event structure passed into the plugin to obtain the device name using a local TempEvent *event structure; The TaskIndex is extracted from TempEvent and then the device name is obtained from that using LoadTaskSettings(TempEvent.TaskIndex);

The results of adding the extra debugging code does not affect the problem (works the same if I remove the debug stuff).

This clue seems important to me because it suggests that the remaining TaskDeviceName problem is caused by a PluginCall() that has an incorrect task index in the passed *event structure. So even if the calling function did the required LoadTaskSettings() before PluginCall() to refresh everything, it would also have the incorrect index value. This will cause it to retrieve the wrong event structure data as well.

It also seems to be a useful clue that the zombie index belongs to a TaskDeviceName that is the first device (index 0).

EDIT: Also, the zombie's log entry always occurs immediately after the initial write's valid entry. They come in pairs, one good, followed by one bad.

I haven't put these pieces together yet, but I think these little details are trying to tell us something.

  • Thomas
thomastech picture thomastech on 1 Aug 2018

Have you tried to change the name of the plugin and save it again?
It could be the stored setting is corrupt.

TD-er picture TD-er on 1 Aug 2018

I had that thought too, but a name change/save on the Switch and Nextion plugins does not help. I've even deleted the Nextion plugin and re-installed it.

The problem only affects Publish statements. So the source of the bug may be related to the openhab MQTT controller plugin. EDIT: Or the rules parser.

  • Thomas
thomastech picture thomastech on 1 Aug 2018

Also, if I delete the Task 1 DRYER Switch Input device it still creates the same zombie log entry. Except the name is blank (null).

  • Thomas
thomastech picture thomastech on 1 Aug 2018

@thomastech You may be on to something.
See this code segment: (C006 PidomeMQTT does the same)

https://github.com/letscontrolit/ESPEasy/blob/402de69fd59b96b3a7dc3f43a1f1a98227de16e8/src/_C005.ino#L41-L82

There is a TempEvent created which is calling a PluginCall
But this TempEvent is not initialized, so the TaskIndex is 0.

TD-er picture TD-er on 1 Aug 2018

Sounds like settings garbage to me. Didn't you extend the settings allocation to a 64 but string per plugin @TD-er ? The Nextion plugin is the only plugin that is nibbling on that whole string. The rest isn't close to it's limits.

Grovkillen picture Grovkillen on 1 Aug 2018

PS. Forgive me if I'm wrong ;)

Grovkillen picture Grovkillen on 1 Aug 2018

@Grovkillen: I had to abandon that idea. I've reduced the custom config to use only 510. Any attempt to use its original 768 bytes ends up biting me. Causes very odd reboots.

@TD-er: That info sounds like it could lead to a holy grail fix. :)

thomastech picture thomastech on 1 Aug 2018

@thomastech
Working on it as we speak ;)

TD-er picture TD-er on 1 Aug 2018

Hmm, I will add some invalid initial value in the EventStruct (TaskIndex set to TASKS_MAX).
If somewhere else this value is not set, we will detect it.
So I may create a Pull request, but not yet merge it. Just to get all of these bugs out.
Things will cause issues then, but rather that, since the alternative is corrupt silently.

TD-er picture TD-er on 1 Aug 2018

If you email the revised file I can try it out ASAP. Or post the code here. I suspect its just a small change to ESPEasy-Globals.h

  • Thomas
thomastech picture thomastech on 1 Aug 2018

Nope it is not :(

TD-er picture TD-er on 1 Aug 2018

Ok, I'll be patient and wait for the release. If you need anything else from me then let me know.

  • Thomas
thomastech picture thomastech on 1 Aug 2018

I hate to add another distraction to your work, but I did an experiment. I changed struct EventStruct's default TaskIndex to 1 instead of 0. {ESPEasy-Globals.h}

That indeed changed the zombie Task name to the second device task. So this confirms that your discovery of the uninitialized *event struct is spot on.

  • Thomas
thomastech picture thomastech on 1 Aug 2018

I am now looking into ExecuteCommand, since it creates a temp event.
Only thing is that one cannot set the task number, since commands can originate from anywhere.
Nextion plugin is also calling that function. Not sure about where the TaskIndex is really needed in commands, but we may open another can of worms there....

TD-er picture TD-er on 1 Aug 2018

Ah found a big Catch-22 :(

MQTTClient does use a callback function to handle responses.
This callback function does create a temp event, but it hasn't got the information about the TaskIndex.
So with an unset TaskIndex in the event, it does call controllers with CPLUGIN_PROTOCOL_RECV
And the controllers responding to that function are:

  • 002: Domoticz MQTT
  • 005: OpenHAB MQTT
  • 006: PiDome MQTT

Domoticz MQTT does seem to handle it somewhat OK, as long as you don't use some string replacement of uservar formatting that use the ExtraTaskSettings.

C005 and C006 call PluginCall(PLUGIN_WRITE.... That one will try all TaskIndices, so that seems to work.
Then they call remoteConfig and that one does try to find an index based on a task name (which should be unique if all goes well, users behave, etc.

So maybe we have a loophole here around the Catch-22...

TD-er picture TD-er on 2 Aug 2018

@thomastech See https://github.com/letscontrolit/ESPEasy/pull/1623

Good luck testing... My only test was "does it compile?" and the compiler agrees :)

Now I'm off to bed. Tomorrow is going to be a very exhausting day (30+ degree C at 50% humidity here in the Netherlands)
So not sure if I can stand to be near a computer tomorrow evening, but if you thing it works, please say so then I can press [merge] on my mobile phone ;)

TD-er picture TD-er on 2 Aug 2018

I'll try out your commit. I have air conditioning.

I'm mentally circling back to my observation that that the zombie appears immediately after the a valid publish. As mentioned, they come in pairs, one good and one bad.

But there should only be one publish. It is triggered by a Nextion timer action that occurs every 10 secs. Nextion sends a unique idx code, the ESPEasy rules detect the code, and its assigned MQTT publish command is sent to my home automation system. But only one publish should be sent, not two.

This is another possible clue. That is to say, it seems like there's an extra PluginCall(PLUGIN_WRITE....) in the deep blue pool of code.

  • Thomas
thomastech picture thomastech on 2 Aug 2018

Air conditioning turned on. Ice tea glass filled. New code compiled and flashed. ESPEasy booted.

Now I see in my debugging logs that the zombie's TaskDeviceName is empty (NULL) and that you've defaulted the index to 12 (TASKS_MAX).

The extra publish (the zombie) seems to be a compelling clue. I feel it's telling us something important. My gut feeling is that it is screaming that there's an unnecessary second call to PluginCall(PLUGIN_WRITE...).

  • Thomas
thomastech picture thomastech on 2 Aug 2018

@TD-er: The duplicate publish log entries were indeed a clue. But not like I expected. And my apologies since I'm the one to blame for today's frustrations.

The previous commit (July 31) did indeed fix my Nextion Plugin TaskDeviceName issue. And the "zombie" publish problem I reported after installing it is not a problem at all.

That's because the Nextion Plugin's PLUGIN_WRITE isn't meant to process the "publish" commands. Although they needlessly get passed into the Nextion plugin's PLUGIN_WRITE, they are ignored. The only reason I saw them there was because of the addLog messages I added to debug the original TaskDeviceName problem.

So assuming my latest conclusions are correct (need to do more validation), I have to apologize for roping you into working on today's commit. I'm not sure it is needed, but before you kill it please give me another day to retest the July 31 release.

Stay cool!

  • Thomas
thomastech picture thomastech on 2 Aug 2018

Please do test with the latest pr, since that does fix some other possible issues.

About the temperature.
I really don't like those 30+ C days in Holland when humidity is high.

Last week we had 35+C but with about 20-30%H (almost no rain for weeks) that was do-able.
Now the humidity in the air is somewhat back to normal and then 30+C is horrible

TD-er picture TD-er on 2 Aug 2018

Not sure if that's related, but I'm seeing in a (self built, with standard 12 tasks and only 3 plugins and 1 controller enabled) on a sonoff-dual 1M node that setting/deleting task Nr. 7 invalidates/deletes IP & Port of my controller 1 settings and vice versa.... I'm trying to debug this, but probably it helps to point to some direction...
goin to try your PR if that still happens afterward...

EDIT:
MEA CULPA!!! (or so)

the issue seems to be the following definitions:
`#define DAT_OFFSET_CONTROLLER DAT_OFFSET_TASKS + (DAT_TASKS_SIZE * TASKS_MAX) // each controller = 1k, 4 max

define DAT_OFFSET_CUSTOM_CONTROLLER DAT_OFFSET_CONTROLLER + (DAT_CUSTOM_CONTROLLER_SIZE * CONTROLLER_MAX) // each custom controller config = 1k, 4 max

`

which were mentioned in the forum somewhere if you wnat to go beyond 12 tasks, but since the commit from july 31 this does not work anymore and causes strange effects that have not been tehre before (as well with 12 and 24 tasks)....

sorry for any confusion caused... but still my problem remains, how to get more than 12 tasks...

@thomastech: what's your defines to use 24 tasks?

EDIT2: DAT_TASKS_SIZE chnaged in a recent commit, so above calculation will not work anymore!! need to take DAT_TASKS_DISTANCE now...

clumsy-stefan picture clumsy-stefan on 2 Aug 2018

@clumsy-stefan You should try the new meminfodetail command and view the output on the serial port.
It may help you understand how the data is aligned in the settings file.

One thing we could do is to make a "jump" in the lookup function when task ID > 12.
That would keep the file compatible with older versions.

TD-er picture TD-er on 2 Aug 2018

@TD-er: yes I know, I do and I will.. but currentl don't have physical access to the nodes... ;)

again, sorry for confusion, I somehow missed that change... adapting above formulas (SIZE->DISTANCE) rendered all my nodes working flawlessly again... including tonights commit...

one more thing: I wanted to try with the standard builds today, but even the 1M builds are too large to use it with the two-step update process, so without physical access, it's not possible to try them...

and again, thanks for all the work and help!! really appreciate it!!

clumsy-stefan picture clumsy-stefan on 2 Aug 2018

@TD-er: I will rebuild with the latest PR and do a sanity validation.

I did an overnight run with the July 31 release. It seemed to be working Ok. But this morning I noticed that the uptime was wrong. Seems a reboot or something else reset the uptime. I haven't been monitoring uptime so I don't know if it is a new problem.

Thanks for mentioning meminfodetail. To get more insight on my custom configuration I ran it. I see the same memory map you posted a few days ago. However, I notice that the memory start & end points on TaskSettings and CustomTaskSettings step on each other. The memory map on the other structs overlap too. Wouldn't this invite problems? Or perhaps meminfodetail is reporting it wrong and overlap is not occurring?

@clumsy-stefan: My builds use the default 12 tasks.

  • Thomas
thomastech picture thomastech on 2 Aug 2018

@thomastech If there is some overlap, then we should definitely look into it.
That's the main reason I added this command.
And it would also explain your issues when increasing the area for Nextion.

Edit:
By overlap, you mean the last/first address?
That's a 1-off-error in the view. But just to make sure, I will check if this is not an error too in writing to the flash.

TD-er picture TD-er on 2 Aug 2018

@TD-er: Yes, I mean that meminfodetail reports the first and last address values overlap with the adjacent structs's memory map. 1-off-error.

  • Thomas
thomastech picture thomastech on 2 Aug 2018

@TD-er: LoadFromFile() and SaveToFile() look OK to me. No overwriting issues.

meminfodetail is reporting the offset + max_size for the end address. The actual end address should be offset + max_size - 1 (because start address begins at a zeroth location).

-Thomas

thomastech picture thomastech on 2 Aug 2018

Maybe this is more clear?
ESP8266 with 12 Tasks as max:
image

ESP32:
image

See latest commit on this PR: https://github.com/letscontrolit/ESPEasy/pull/1623

I still have to add the normal settings offset.

TD-er picture TD-er on 2 Aug 2018

@TD-er: The graphic memory map looks very nice. I recommend dropping the decimals on the memory values since they would be implied.

Yesterday's PR release is running OK. But the incorrect uptime occurred again. It's due to a panic reboot. Happened after a MQTT publish to my Openhab home automation system. Here's the log:

11514348 : NEXTION075 : code: |s,i16,sOff
11514353 : EVENT: NEXTION#idx=16.00
11514394 : ACT  : Publish /ESPEZ_Laundry/NEXTION/idx,16
11514408 : Command: publish

Panic C:\Users\Tom\AppData\Local\Arduino15\packages\esp8266\
hardware\esp8266\2.4.1\cores\esp8266\core_esp8266_main.cpp:99 __yield

ctx: cont 
sp: 3fff3e00 end: 3fff4750 offset: 01b0

>>>stack>>>
3fff3fb0:  00000001 00000001 3fff40f4 3fff2419  
3fff3fc0:  00000001 00000001 3fff899c 4025fab8  
3fff3fd0:  00000001 00000001 3fff899c 40241651  
3fff3fe0:  00000c01 00000000 00000000 40100200  
3fff3ff0:  00000000 00000010 00000000 00000000  
3fff4000:  00000000 00000000 3fff8b14 0000000f  
3fff4010:  00000000 3fff8b2c 0000000f 00000000  
3fff4020:  3fff8b44 0000000f 00000000 3fff8b5c  
3fff4030:  0000000f 00000000 3fff8b74 0000000f  
3fff4040:  00000000 00000000 6c627550 00687369  
3fff4050:  00000046 3fff4110 0000000d 402173d7  
3fff4060:  00000600 4c5f1800 00000000 4e2f7901  
3fff4070:  00000000 00000010 00000000 00000000  
3fff4080:  00000000 00000000 00000000 00000000  
3fff4090:  00000000 00000000 50003600 4c5f5a45  
3fff40a0:  646e7561 4e2f7972 49545845 692f4e4f  
3fff40b0:  00007864 00000000 00000000 00000000  
3fff40c0:  00000000 00000000 0000032d 4025ea7b  
3fff40d0:  312c7864 00000036 3fff41dc 4025eaad  
3fff40e0:  3fff41dc 00000000 3fff26ae 00000006  
3fff40f0:  00000001 3fff8afc 0000000f 00000000  
3fff4100:  00000001 00000001 00000001 40241dd5  
3fff4110:  00000c00 00000000 00000000 3fff5300  
3fff4120:  00000000 00000010 00000000 00000000  
3fff4130:  00000000 00000000 3fff8a34 0000000f  
3fff4140:  00000000 3fff8a4c 0000000f 00000000  
3fff4150:  3fff8a7c 0000000f 00000000 3fff8a94  
3fff4160:  0000000f 00000000 3fff8aac 0000000f  
3fff4170:  00000000 00000000 4e5b2c78 49545845  
3fff4180:  69234e4f 0d5d7864 6520200a 6669646e  
3fff4190:  20200a0d 5b206669 5458454e 234e4f49  
3fff41a0:  5d786469 30353d3e 2f202030 6f54202f  
3fff41b0:  20686375 6e657645 40107b10 00000000  
3fff41c0:  000003e8 00000000 3fff87fc 3fff883c  
3fff41d0:  00000000 00000000 00000000 3fff8ac4  
3fff41e0:  0000002f 00000025 3fff89d4 0000002f  
3fff41f0:  00000025 3fff899c 0000002f 00000025  
3fff4200:  3fff87e4 0000000f 00000000 3fff8964  
3fff4210:  0000002f 00000025 3fff891c 0000003f  
3fff4220:  00000025 3fff8a0c 0000001f 0000001b  
3fff4230:  00000001 00000001 00000001 00000001  
3fff4240:  3fff42d0 00000010 3fff4280 00000040  
3fff4250:  00000000 3fff2419 3fff428c 4025ebb0  
3fff4260:  3fff0031 3fff2419 00000001 00afb1f1  
3fff4270:  3fff42d0 3fff2419 00000001 40241f93  
3fff4280:  00000000 00000000 00000000 3fff87cc  
3fff4290:  0000000f 0000000a 3fff879c 0000000f  
3fff42a0:  0000000a 00000030 3fff42d0 4025f058  
3fff42b0:  3fff259e 00000000 00000000 00000005  
3fff42c0:  00000010 00000004 00000000 40242b55  
3fff42d0:  3fff8774 0000001f 00000011 4010020c  
3fff42e0:  3fff23c4 3fff0e58 3fff4310 40100690  
3fff42f0:  00000000 00000000 00000000 00000011  
3fff4300:  3fff1b68 3fff26a8 3fff4610 40244435  
3fff4310:  00000000 00000000 00000000 00000011  
3fff4320:  3fff1b68 3fff0e58 3fff4610 40264db4  
3fff4330:  ae4f0759 3fff0e58 3fff4560 40262844  
3fff4340:  3fff0030 3fff0e58 3fff45a8 00000011  
3fff4350:  3fff1b68 3fff0e58 3fff4610 4024a0c7  
3fff4360:  692c737c 732c3631 0066664f 0083126e  
3fff4370:  3fff5ccc 3fff4670 3fff4478 00000030  
3fff4380:  4021515a 3fff4670 3fff64b4 3fff645c  
3fff4390:  0000072f 0000002f 00000000 00000089  
3fff43a0:  00000000 00000000 0000001f 40104ec9  
3fff43b0:  4000050c 3fff1b00 3fff4470 00000030  
3fff43c0:  40000f83 00000030 00000000 ffffffff  
3fff43d0:  40285942 00000023 3fffbff8 00000e71  
3fff43e0:  000006b0 00000e70 00000001 3fff8a78  
3fff43f0:  000000c2 3fff64dc 00000001 3fff4b5c  
3fff4400:  3fff3640 00000000 00007388 00000030  
3fff4410:  40103e13 3ffef740 50800000 4000050c  
3fff4420:  3fffc278 40101f8c 3fffc200 00000022  
3fff4430:  3fff4440 00000023 3fff84fc 40214afa  
3fff4440:  ffffe660 00000020 00000000 4010053d  
3fff4450:  401004ba 00000023 00000003 3fff4490  
3fff4460:  3fff1b00 3fff4468 3fff4670 40214bbc  
3fff4470:  3fff4470 3fff4470 3fff4478 3fff4478  
3fff4480:  3fff4480 3fff4480 3fff4488 3fff4488  
3fff4490:  3fff4490 3fff4490 3fff4498 3fff4498  
3fff44a0:  3fff44a0 3fff44a0 3fff44a8 3fff44a8  
3fff44b0:  3fff44b0 3fff44b0 3fff44b8 401071ee  
3fff44c0:  3fff44c0 3fff44c0 3fff4610 40210069  
3fff44d0:  3fff44d0 3fff44d0 3fff44d8 40244d60  
3fff44e0:  3fff44e0 3fff44e0 3fff44e8 3fff44e8  
3fff44f0:  3fff44f0 3fff44f0 3fff44f8 3fff44f8  
3fff4500:  00000000 000006c0 000006c0 4010020c  
3fff4510:  3fff3640 000006c0 000006c0 00000046  
3fff4520:  3fff3648 00000000 3fff16d0 40285968  
3fff4530:  00000014 000035d8 000006bb 40207b90  
3fff4540:  00000010 00000010 3fff178c 4028f34f  
3fff4550:  4025d1f4 3fff4610 3fff178c 4025ebb0  
3fff4560:  00000000 00000000 00000000 4020b47c  
3fff4570:  3fff3640 000006bb 000006bb 4010020c  
3fff4580:  00000004 3fff4610 3fff45b0 40100690  
3fff4590:  3fff875c 0000000f 0000000b 3fff8744  
3fff45a0:  0000000f 00000000 3fff8r‚)ŠLþ
ôŒ
JE0¸
†´ˆˆ9!9¯„ù„!ÁÎì)ÿ18Áù526 : INIT : Free RAM:21328
527 : INIT : I2C
527 : INIT : SPI not enabled
541 : NEXTION075 : serial pin config RX:13, TX:15, Plugin Enabled
543 : NEXTION075 : Using software serial
599 : INFO : Plugins: 71 [Normal] [Testing] (ESP82xx Core 2_4_1, NONOS SDK 2.2.1(cfd48f3), LWIP: 2.0.3)
696 : EVENT: System#Wake
745 : WIFI : Set WiFi to STA
  • Thomas
thomastech picture thomastech on 2 Aug 2018

Maybe something is calling delay() or yield() in the MQTT callback function?

TD-er picture TD-er on 2 Aug 2018

Controller.ino has a yield when initializing the MQTT connection, but nothing obvious beyond that. I'm not experienced with the callbacks, so it would be great if you took a peek at that magic code.

  • Thomas
thomastech picture thomastech on 2 Aug 2018

@thomastech I think the MQTT crash is another issue.
And since it will involve some changes, which may reflect all MQTT, I guess we should add another issue for it.

TD-er picture TD-er on 3 Aug 2018

@TD-er: You are right, the MQTT panic crash/reboot needs its own ticket. I have a feeling it may also be involved with my custom configuration memory problem (>512 bytes bug) since it also has a lot of panic exception crashes.

So let's finish up the TaskDeviceName issue. Here's my final findings:

I ran a second NodeMCU board with your Aug 1 PR. I removed the MQTT actions from it and it ran all night without a reboot. So I think your latest PR is Ok to commit.

So I think this ticket can be closed. Thanks for all your help on fixing TaskDeviceName, I appreciate it.

Parting words:
I have learned that intensely monitoring ESPEasy's logs is a lot like watching sausage being made. Besides finding the MQTT reboots, I also saw a wifi beacon timeout. I searched and found some discussion on this, including a comment from you that beacon timeout isn't a concern if it doesn't occur too often. So I will ignore it. And I'll try to avoid looking in the logs, since I might not like what I see.

  • Thomas
thomastech picture thomastech on 3 Aug 2018

The logs are indeed quite verbose and may need some cleanup.
We were also discussing to make log topics selectable as an extra filter on top of severity levels.
Also a lot of logs use Info, perhaps a bit more than needed.

TD-er picture TD-er on 3 Aug 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tedenda picture tedenda  ·  6Comments
ghtester picture ghtester  ·  3Comments
TD-er picture TD-er  ·  4Comments
thehijjt picture thehijjt  ·  4Comments
Barracuda09 picture Barracuda09  ·  5Comments