Irremoteesp8266: Extend auto_analyse_raw_data.py -g with generic Receive function

[NiKiZe]

Version git master e12e194127 after v2.6.5

First mandatory thank you for an up2date and maintained repo!

I have a few remotes that I would like to add, hopefully with full decoding of bits
But how to create the basic functions are not clear to me (not working when trying), might be helpful if that could be made easier, for example by having templates, och use the python script as a template generator.

when running auto_analyse_raw_data.py with -g we get a nice base for sending:

// WARNING: This probably isn't directly usable. It's a guide only.
const uint16_t kHdrMark = 3380;
const uint16_t kBitMark = 441;
const uint16_t kHdrSpace = 1760;
const uint16_t kOneSpace = 1294;
const uint16_t kZeroSpace = 433;
const uint16_t kXyzBits = 40;
// Function should be safe up to 64 bits.
void IRsend::sendXyz(const uint64_t data, const uint16_t nbits, const uint16_t repeat) {
  enableIROut(38);  // A guess. Most common frequency.
  for (uint16_t r = 0; r <= repeat; r++) {
    // Header
    mark(kHdrMark);
    space(kHdrSpace);
    // Data
    // e.g. data = 0x800B3048C0, nbits = 40
    sendData(kBitMark, kOneSpace, kBitMark, kZeroSpace, data, nbits, true);
    // Footer
    mark(kBitMark);
    space(100000);  // A 100% made up guess of the gap between messages.
  }
}

I get a felling that this is before the existence of the generic functions?
So the send function could be updated to use generic base functions instead?
And adding a receive function should also be possible?

I would also like to suggest that -g can take a name of the protocol and use that in constants and function names

Also add comments on what else needs to be updated when adding new protocols, more or less like a checklist for IRSend.h/cpp IRRecv.h/cpp etc

And one last thing, would it be helpful if we could provide multiple samples in the same run to get better identification?
I noticed that if there is a large enough gap it can decode multiple samples in one go?

[crankyoldgit]

First off, I'll draw your attention to this line. :-)

// WARNING: This probably isn't directly usable. It's a guide only.

I get a felling that this is before the existence of the generic functions?

Correct.

So the send function could be updated to use generic base functions instead?

TL;DR: Optimising is a hard problem. This is really just a generic diagnostic tool.

The generic base functions don't apply in all cases, and to have that script have the intelligence to work it out whether or not it can use them is far beyond the scope and usefulness IMHO.
Hence it uses more primitive functions/methods to be better able to handle more cases.

And adding a receive function should also be possible?

In very simple cases, it could be done. However, it's hard to tell when it's a simple case, and most often it isn't simple.

I'm not saying No, but it's hard to do, and way way down on my priority list.

I would also like to suggest that -g can take a name of the protocol and use that in constants and function names

This I can probably do easily. I'll look into it.

Also add comments on what else needs to be updated when adding new protocols, more or less like a checklist for IRSend.h/cpp IRRecv.h/cpp etc

I can probably add some documentation on the wiki and add a link in the comments.

And one last thing, would it be helpful if we could provide multiple samples in the same run to get better identification?

Not sure what you mean by this. Can you elaborate please?

I noticed that if there is a large enough gap it can decode multiple samples in one go?

Ditto. Not sure what you mean by this. Can you elaborate please? e.g. by way of examples perhaps?

[NiKiZe]

Sorry if it sounded like i was expecting miracles, I'm not.
Not expecting or suggesting it to be a tool to automatically fix everything.
Just for it to be a little better than it is now, in terms of using it as a template for newcomers as myself to make it easier to get started.
This is also based on that from what I have seen, if someone posts a sample as below with brand and model numbers you (@crankyoldgit) are very quick with generating a basic working Receiver and Sender for it, from that I think that the python tool could be used to do more of that work.
Might be a bad idea, but maybe even generate testcase(s) would be possible.

in regards of multiple samples, consider

uint16_t rawData[83] = {+3440,1696, +442, 1296, +440, 436,  +462, 386,  +462, 410,  +462, 412,  +492, 414,  +470, 414,  +448, 442,  +442, 434,  +468, 418,  +468, 410,  +472, 416,  +468, 1270, +440, 440,  +442, 1290, +440, 1292, +440, 444,  +440, 412,  +460, 1286, +442, 1296, +442, 416,  +456, 414,  +460, 416,  +456, 418,  +458, 414,  +490, 1272, +442, 436,  +444, 430,  +440, 1288, +418, 462,  +438, 412,  +456, 420,  +458, 1286, +440, 1290, +440, 414,  +464, 428,  +442, 412,  +464, 412,  +486, 414,  +474, 410,  +442 };  // UNKNOWN 7CB2F103
uint16_t rawData[83] = {+3386,1756, +442, 1294, +420, 456,  +440, 410,  +438, 434,  +464, 414,  +464, 440,  +470, 414,  +470, 420,  +440, 410,  +468, 436,  +448, 438,  +468, 416,  +472, 1268, +442, 440,  +442, 1288, +468, 1268, +468, 394,  +462, 414,  +432, 1312, +442, 1296, +418, 458,  +440, 412,  +436, 458,  +440, 438,  +438, 412,  +490, 1272, +442, 442,  +440, 410,  +460, 1294, +442, 414,  +458, 410,  +462, 414,  +464, 1286, +414, 1318, +416, 458,  +418, 434,  +464, 408,  +464, 412,  +490, 414,  +448, 438,  +442 };  // UNKNOWN 7CB2F103

Here we have 2 samples, it would be good if these could be given to the tool and use both of them to determine what to do.
we can emulate that by adding a fictional gap (using 20000 in this case), such as:

{+3408,1722, +442, 1286, +440, 420,  +466, 440,  +446, 438,  +472, 412,  +448, 438,  +442, 418,  +466, 442,  +416, 438,  +466, 440,  +442, 440,  +446, 440,  +442, 1296, +416, 432,  +438, 1310, +418, 1318, +446, 438,  +416, 454,  +414, 1310, +416, 1316, +444, 408,  +438, 454,  +416, 458,  +414, 438,  +442, 454,  +386, 1356, +412, 470,  +412, 436,  +460, 1288, +420, 456,  +414, 460,  +412, 440,  +438, 1308, +416, 1318, +416, 458,  +414, 438,  +436, 432,  +438, 460,  +440, 442,  +440, 440,  +418, 20000, +3380,1756, +446, 1290, +416, 460,  +440, 432,  +444, 426,  +418, 454,  +448, 440,  +468, 414,  +524, 364,  +444, 428,  +472, 414,  +466, 418,  +440, 438,  +472, 1270, +418, 462,  +444, 1286, +442, 1292, +416, 466,  +444, 430,  +440, 1284, +438, 1302, +440, 432,  +414, 458,  +442, 406,  +440, 458,  +442, 434,  +466, 1272, +418, 464,  +446, 426,  +420, 1312, +416, 440,  +460, 412,  +460, 436,  +418, 1306, +444, 1292, +416, 436,  +460, 432,  +414, 458,  +438, 410,  +492, 414,  +474, 410,  +418, 20000, +3380,1760, +440, 1290, +438, 442,  +446, 438,  +446, 438,  +446, 438,  +444, 438,  +448, 438,  +442, 446,  +438, 414,  +466, 438,  +446, 440,  +440, 440,  +470, 1270, +442, 406,  +464, 1286, +446, 1292, +494, 392,  +438, 430,  +420, 1306, +444, 1292, +412, 456,  +418, 458,  +440, 432,  +414, 458,  +418, 458,  +444, 1300, +438, 442,  +440, 412,  +460, 1288, +442, 410,  +462, 410,  +462, 436,  +416, 1310, +418, 1312, +416, 460,  +412, 438,  +460, 410,  +464, 432,  +442, 444,  +466, 418,  +412, 20000, +3384,1758, +412, 1320, +442, 412,  +438, 436,  +436, 458,  +438, 412,  +464, 440,  +446, 440,  +444, 444,  +420, 456,  +442, 442,  +468, 414,  +444, 440,  +444, 1298, +440, 442,  +414, 1314, +418, 1318, +446, 414,  +462, 414,  +462, 1280, +420, 1326, +440, 412,  +460, 412,  +436, 456,  +442, 434,  +418, 438,  +488, 1274, +440, 438,  +418, 434,  +464, 1286, +442, 434,  +472, 404,  +440, 432,  +418, 1306, +442, 1292, +442, 434,  +414, 456,  +418, 454,  +444, 408,  +488, 416,  +470, 414,  +446, 20000, +3440,1696, +442, 1296, +440, 436,  +462, 386,  +462, 410,  +462, 412,  +492, 414,  +470, 414,  +448, 442,  +442, 434,  +468, 418,  +468, 410,  +472, 416,  +468, 1270, +440, 440,  +442, 1290, +440, 1292, +440, 444,  +440, 412,  +460, 1286, +442, 1296, +442, 416,  +456, 414,  +460, 416,  +456, 418,  +458, 414,  +490, 1272, +442, 436,  +444, 430,  +440, 1288, +418, 462,  +438, 412,  +456, 420,  +458, 1286, +440, 1290, +440, 414,  +464, 428,  +442, 412,  +464, 412,  +486, 414,  +474, 410,  +442, 20000, +3386,1756, +442, 1294, +420, 456,  +440, 410,  +438, 434,  +464, 414,  +464, 440,  +470, 414,  +470, 420,  +440, 410,  +468, 436,  +448, 438,  +468, 416,  +472, 1268, +442, 440,  +442, 1288, +468, 1268, +468, 394,  +462, 414,  +432, 1312, +442, 1296, +418, 458,  +440, 412,  +436, 458,  +440, 438,  +438, 412,  +490, 1272, +442, 442,  +440, 410,  +460, 1294, +442, 414,  +458, 410,  +462, 414,  +464, 1286, +414, 1318, +416, 458,  +418, 434,  +464, 408,  +464, 412,  +490, 414,  +448, 438,  +442, 20000, +3466,1670, +438, 1280, +460, 414,  +454, 418,  +458, 432,  +438, 416,  +486, 420,  +466, 420,  +466, 420,  +468, 406,  +470, 414,  +468, 416,  +468, 416,  +494, 1246, +442, 440,  +464, 1264, +442, 1298, +436, 442,  +440, 414,  +434, 1310, +440, 1302, +442, 414,  +488, 408,  +440, 408,  +462, 416,  +458, 414,  +518, 1246, +438, 440,  +446, 408,  +460, 1288, +446, 410,  +460, 412,  +466, 412,  +464, 1282, +442, 412,  +462, 1290, +440, 414,  +460, 412,  +458, 414,  +494, 414,  +470, 414,  +440, 20000, +3434,1694, +416, 1320, +418, 436,  +462, 414,  +462, 408,  +464, 434,  +464, 420,  +442, 440,  +444, 444,  +446, 430,  +442, 440,  +468, 414,  +496, 392,  +492, 1246, +442, 440,  +416, 1312, +442, 1296, +440, 440,  +438, 412,  +464, 1280, +440, 1300, +442, 434,  +440, 414,  +462, 406,  +464, 414,  +458, 414,  +464, 1298, +440, 442,  +440, 408,  +436, 1316, +440, 434,  +414, 438,  +462, 434,  +440, 1284, +446, 408,  +460, 1288, +418, 458,  +442, 434,  +442, 406,  +490, 416,  +468, 416,  +440, 20000, +3460,1674, +442, 1288, +442, 408,  +462, 410,  +438, 456,  +468, 420,  +468, 416,  +468, 416,  +468, 422,  +440, 434,  +466, 414,  +470, 414,  +446, 440,  +470, 1266, +414, 458,  +444, 1288, +442, 1294, +466, 412,  +442, 434,  +440, 1282, +446, 1288, +414, 440,  +458, 412,  +462, 414,  +460, 434,  +444, 410,  +488, 1272, +442, 440,  +418, 456,  +414, 1318, +440, 408,  +462, 414,  +460, 416,  +462, 1284, +440, 412,  +462, 1294, +440, 434,  +440, 410,  +460, 412,  +492, 412,  +444, 440,  +442, 20000, +3440,1700, +442, 1284, +416, 462,  +412, 436,  +436, 460,  +440, 442,  +442, 442,  +442, 442,  +388, 506,  +410, 462,  +442, 440,  +386, 498,  +440, 442,  +440, 1296, +416, 460,  +360, 1368, +414, 1322, +390, 494,  +362, 488,  +438, 1310, +360, 1374, +358, 492,  +438, 436,  +440, 456,  +414, 442,  +434, 460,  +384, 1354, +360, 522,  +414, 436,  +380, 1372, +414, 440,  +436, 460,  +410, 440,  +380, 1368, +358, 512,  +414, 1316, +358, 516,  +418, 456,  +414, 440,  +408, 494,  +386, 498,  +358, 20000}

Or maybe there is a good reason for this being a bad idea as well.

[crankyoldgit]

Sorry if it sounded like i was expecting miracles, I'm not.

Na. You didn't come across that way. No worries. I'm just letting you know the reality from my perspective. ;-)

for it, from that I think that the python tool could be used to do more of that work.

I use that tool when analysing the protocol. It's basically a tool for me to get an idea of the structure of the message so I can code it up easier. The tool could probably do more of the work, but I haven't really needed. That is NOT to say others couldn't benefit from it. Clearly they could.

Might be a bad idea, but maybe even generate testcase(s) would be possible.

You're not alone in thinking that. I really do need to make it easier/more obvious how to add stuff.

in regards of multiple samples, consider

First off, where is that data from? the +'s are not normal in the rawData[] output lines. What generated that?

While it isn't obvious to new folk. If the library detects it as two messages, they are NOT the same as a single message with a small (20000) gap between them. That "analysis" would lead to different results than what you were expecting. However, it would give you a better "average" timing values because of more data. However, those "values" are just a good first place to start when tuning for matching a protocol. That's the harder part.

[NiKiZe]

First off, where is that data from? the +'s are not normal in the rawData[] output lines. What generated that?

Based on standard dumpV2, with the code dump function copied and modified to add '+' on the mark part to make it easier to see what is what when reading it manually, similar to how the timing dumper function does it. - it is the same normal dump data, just with + added.

While it isn't obvious to new folk. If the library detects it as two messages, they are NOT the same as a single message with a small (20000) gap between them. That "analysis" would lead to different results than what you were expecting. However, it would give you a better "average" timing values because of more data. However, those "values" are just a good first place to start when tuning for matching a protocol. That's the harder part.

What I noticed during another remote where i expected 115-ish long values (based on previous work), but got 3-hundred here instead, I might be wrong but expect it to be 3 messages, not 1 with 2 gaps.
The idea was, as you say, to get better timings. but also to analyse bit data for multiple messages easier, without having ha working receive function yet.

Thanks!

[crankyoldgit]

Based on standard dumpV2, with the code dump function copied and modified to add '+' on the mark part to make it easier to see what is what when reading it manually, similar to how the timing dumper function does it. - it is the same normal dump data, just with + added.

Phew. Just making sure it wasn't my code that was now doing that.

What I noticed during another remote where i expected 115-ish long values (based on previous work), but got 3-hundred here instead, I might be wrong but expect it to be 3 messages, not 1 with 2 gaps.
The idea was, as you say, to get better timings. but also to analyse bit data for multiple messages easier, without having ha working receive function yet.

Yeah, there are some protocols like that. Sometimes they have a "repeat" built in. I think (I used to) have code in the analyse prog to see if the same code was used. ie. a true repeat.

Some protocols send a code, then send it inverted, and then inverted again (i.e. normal) .. only experience and human-smarts picks that up. It's hard(er) to code that in to a quick python code tool. ;-)

FYI the "does it detect a gap or treat it as separate messages" is influenced by the _timeout_ controls in the library. Some protocols have multiple sub-messages. e.g. Kelvinator has 20000usec and a 40000usec gap between message segments. Some simpler protocols have gap between their distinct messages of less than 40000usec etc. So it is quite possible to have two messages in the one capture.
Again, experience and "wtfbbq is going on here" analysis can tell what is what. Again. Hard to add to the script. ;-)

[crankyoldgit]

@NiKiZe Can you have a read over this wiki entry to see if it helps you add sending of the protocols you are playing with?

https://github.com/crankyoldgit/IRremoteESP8266/wiki/Adding-support-for-a-new-IR-protocol

Feedback/mistakes/omissions etc welcome. It was done off the top of my head, so there will be mistakes.

[NiKiZe]

This is awesome!
Right now I'm mostly working with the receive path.
I will however follow the new page to create another remote, and hopefully the receive on that one will be more friendly.
I might have to give up on the first one, and open an issue for the pros here to help me get a receive working for it, and than i can add settings etc on that. (I did play around with timeouts to tackle the gaps but no dice)

[crankyoldgit]

Using the data from your first rawData[83] line and the updated code in https://github.com/crankyoldgit/IRremoteESP8266/tree/analyse_add_decode:

$ cat /tmp/j
uint16_t rawData[83] = {3440, 1696, 442, 1296, 440, 436,  462, 386,  462, 410,  462, 412,  492, 414,  470, 414,  448, 442,  442, 434,  468, 418,  468, 410,  472, 416,  468, 1270, 440, 440,  442, 1290, 440, 1292, 440, 444,  440, 412,  460, 1286, 442, 1296, 442, 416,  456, 414,  460, 416,  456, 418,  458, 414,  490, 1272, 442, 436,  444, 430,  440, 1288, 418, 462,  438, 412,  456, 420,  458, 1286, 440, 1290, 440, 414,  464, 428,  442, 412,  464, 412,  486, 414,  474, 410,  442 };  // UNKNOWN 7CB2F103
$ ./tools/auto_analyse_raw_data.py -f /tmp/j -g -n Nikize
Found 83 timing entries.
Potential Mark Candidates:
[3440, 492]
Potential Space Candidates:
[1696, 1296, 462]

Guessing encoding type:
Looks like it uses space encoding. Yay!

Guessing key value:
kNikizeHdrMark   = 3440
kNikizeHdrSpace  = 1696
kNikizeBitMark   = 453
kNikizeOneSpace  = 1286
kNikizeZeroSpace = 420

Decoding protocol based on analysis so far:

kNikizeHdrMark+kNikizeHdrSpace+1000000000001011001100000100100011000000
  Bits: 40
  Hex:  0x800B3048C0 (MSB first)
        0x03120CD001 (LSB first)
  Dec:  549943527616 (MSB first)
        13187731457 (LSB first)
  Bin:  0b1000000000001011001100000100100011000000 (MSB first)
        0b0000001100010010000011001101000000000001 (LSB first)

Total Nr. of suspected bits: 40

Generating a VERY rough code outline:

// Copyright 2019 David Conran (crankyoldgit)
// Support for Nikize protocol

#include "IRrecv.h"
#include "IRsend.h"
#include "IRutils.h"

// WARNING: This probably isn't directly usable. It's a guide only.

// See https://github.com/crankyoldgit/IRremoteESP8266/wiki/Adding-support-for-a-new-IR-protocol
// for details of how to include this in the library.
const uint16_t kNikizeHdrMark = 3440;
const uint16_t kNikizeBitMark = 453;
const uint16_t kNikizeHdrSpace = 1696;
const uint16_t kNikizeOneSpace = 1286;
const uint16_t kNikizeZeroSpace = 420;
const uint16_t kNikizeBits = 40;  // Move to IRremoteESP8266.h
#if SEND_NIKIZE
// Function should be safe up to 64 bits.
void IRsend::sendNikize(const uint64_t data, const uint16_t nbits, const uint16_t repeat) {
  enableIROut(38);  // A guess. Most common frequency.
  for (uint16_t r = 0; r <= repeat; r++) {
    // Header
    mark(kNikizeHdrMark);
    space(kNikizeHdrSpace);
    // Data
    // e.g. data = 0x800B3048C0, nbits = 40
    sendData(kNikizeBitMark, kNikizeOneSpace, kNikizeBitMark, kNikizeZeroSpace, data, nbits, true);
    // Footer
    mark(kNikizeBitMark);
    space(kDefaultMessageGap);  // A 100% made up guess of the gap between messages.
  }
}
#endif  // SEND_NIKIZE

#if DECODE_NIKIZE
// Function should be safe up to 64 bits.
bool IRrecv::decodeNikize(decode_results *results, const uint16_t nbits, const bool strict) {
  if (strict && nbits != kNikizeBits)
    return false;

  uint16_t offset = kStartOffset;
  uint64_t data = 0;
  match_result_t data_result;

  // Header
  if (!matchMark(results->rawbuf[offset++], kNikizeHdrMark))
    return false;
  if (!matchSpace(results->rawbuf[offset++], kNikizeHdrSpace))
    return false;

  // Data
  // e.g. data_result.data = 0x800B3048C0, nbits = 40
  data_result = matchData(&(results->rawbuf[offset]), nbits,
                          kNikizeBitMark, kNikizeOneSpace,
                          kNikizeBitMark, kNikizeZeroSpace,
                          _tolerance, kMarkExcess, true);
  offset += data_result.used;
  if (data_result.success == false) return false;  // Fail
  data = data_result.data;

  // Footer
  if (!matchMark(results->rawbuf[offset++], kNikizeBitMark))
    return false;

  // Success
  results->decode_type = decode_type_t::NIKIZE;
  results->bits = nbits;
  results->value = data;
  results->command = 0;
  results->address = 0;
  return true;
}
#endif  // DECODE_NIKIZE

I hope it helps!

[NiKiZe]

Hi,
Sorry for late response here, other things came in the way.
This makes things easier, it's just the "normal" testcase part left than, (but I don't have a *nix shell avaiable on the dev machine ATM so don't help me),
I hope to add PR based on this in a few days.

[crankyoldgit]

Looking forward to it!

[crankyoldgit]

FYI, the changes mentioned above are included in the newly released version of the library (v2.6.6).