Zigbee2mqtt: OTA Update with local files for arbitrary devices?

[felixstorm]

Hi Koen,

do you know of any plans to allow OTA updating of arbitrary devices using local files? Sorry for asking if I should have overlooked something...

Current use case is that I have some Jung ZigBee switches (compatible to Gira 2430-100) lying around that need to be updated to become ZigBee 3.0 compatible, but since there is no public download link available for the update it does not really seem meaningful to especially code a "Jung" OTA update routine and attach it to the device.

Thanks and kind regards,
Felix

[Koenkk]

For these cases I plan to make a github repo containing the ota files together with a json file having meta data. (Similar to IKEA OTA works). What do you think?

[sjorge]

Sounds like a good idea, we can probably set it as the 'default' OTA provider, although hosting them on github might get us into copyright strike territory? Phillips Hue for example is not very willing to share info and seem rather hostile towards the idea of 3rd party software being able to update there stuff :(

[Koenkk]

Repo is live: https://github.com/Koenkk/zigbee-OTA will start on integrating it into zigbee-herdsman-converters now.

[Koenkk]

Cool, I've just managed to update my Hue bulb with this:

zigbee2mqtt:info  2020-03-15 21:50:23: Finished update of 'hue_bulb'
zigbee2mqtt:info  2020-03-15 21:50:23: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"type":"ota_update","message":"Finished update of 'hue_bulb'","meta":{"status":"update_succeeded","device":"hue_bulb","from":{"softwareBuildID":"1.46.13_r26312","dateCode":"20181115"},"to":null}}'

[sjorge]

Nice, looks like it also has the same null version issue that half my trådfri updates had :(
Also were are we getting the firmwares from? I was only aware of one hue firmware for E14 CWS bulbs.

[Koenkk]

Firmwares are downloaded from https://github.com/Koenkk/zigbee-OTA, regarding the null issue, it reads after 30 seconds now, but the bulb took 40 seconds to restart. I've increased this wait period to 60 seconds so should be fixed now.

[sjorge]

Yeah, I noticed the repo, but were are they sourced from?

[Koenkk]

https://github.com/dresden-elektronik/deconz-rest-plugin/issues/270#issuecomment-598630290

[Koenkk]

@felixstorm you can now easily add OTA support for this device by following https://github.com/Koenkk/zigbee-OTA#adding-new-and-updating-existing-ota-files

and adding ota: ota.zigbeeOTA for this device (https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/devices.js#L1482)

[felixstorm]

Great, thanks a lot (and sorry for not replying earlier)! And good idea to include add.js to do the parsing! Will add the firmware tomorrow or in the next few days and file the respective PRs...

Thanks for all your efforts with Z2M and kind regards,
Felix

[sjorge]

@Koenkk is there a way aside from repairing to update the version info?

    {
      "ieeeAddr": "0x0017880104259333",
      "type": "Router",
      "networkAddress": 27321,
      "model": "9290012573A",
      "vendor": "Philips",
      "description": "Hue white and color ambiance E26/E27/E14",
      "friendly_name": "bedroom/desk_lamp/bulb",
      "manufacturerID": 4107,
      "manufacturerName": "Philips",
      "powerSource": "Mains (single phase)",
      "modelID": "LCT012",
      "hardwareVersion": 1,
      "softwareBuildID": "1.29.0_r21169",
      "dateCode": "20170908",
      "lastSeen": 1584526392945
    },

Is still returned by the devices topic, but it did update succesfully using the zigbee-OTA ota provider, and now update_available is false when doing a check.

[Koenkk]

@sjorge unfortunately not, but I've increased delay between update completed and read to 90 seconds, so next time you update it should be fine.

[sjorge]

So the only solution for now would be repairing? Can we perhaps reread the version everytime a /check is executed for a device? Or are those too network intensive?

[copystring]

@Koenkk @sjorge
I believe you mean this? https://github.com/Koenkk/zigbee2mqtt/issues/2921#issuecomment-590107444
I asked this before. I could not be bothered to make a PR for this but it is possible to have this updated.
Or am I wrong here?

[peterforeman]

According to the deconz wiki I should be able to update some of my hue devices, but get an error:

zigbee2mqtt:error 2020-03-19 21:08:25: Failed to call 'OTAUpdate' 'onZigbeeEvent' (AssertionError [ERR_ASSERTION]: No image available for imageType '270'

And also for 274 and 276. These indeed are not listed in the index.json of zigbee-OTA repo. Probably not updated yet? Upgrading my Hue Outdoor motion sensor worked fine! (although it took about one hour)

[Koenkk]

@peterforeman I don't have the firmware files for these devices, if someone helps can finding it would be great!

[sjorge]

@Koenkk so I removed and rejoined the hue bulb... it does indeed have the correct version info afterwards...

But now it fails to setup on/off reporting!

zigbee2mqtt:error2020-03-20 12:26:10: Failed to setup reporting for '0x0017880104259333' - Error: ConfigureReporting 0x0017880104259333/11 genOnOff([{"attribute":"onOff","minimumReportInterval":0,"maximumReportInterval":300,"reportableChange":0}], {"timeout":6000,"manufacturerCode":null,"disableDefaultResponse":true}) failed (Error: Timeout - 27321 - 11 - 3 - 6 - 7 after 6000ms) at Endpoint.<anonymous> (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/controller/model/endpoint.js:316:23) at Generator.throw (<anonymous>) at rejected (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/controller/model/endpoint.js:6:65

I think this was reported a few times before, oddly enough the bulb (before rejoining) did still report on/off correctly 🤔

It also consistantly gives this error once it comes back after a physical power toggle:

zigbee2mqtt:error 2020-03-20 12:29:46: Failed to read state of 'bedroom/desk_lamp/bulb' after reconnect

Before the update this was never an issue... Ok the last one is a bit odd, if only happens if I power off the bulb for less than ~10 seconds

[Koenkk]

@sjorge could you sniff the traffic? We need to figure out if its a zigbee2mqtt issue or a bulb firmware issue.

[sjorge]

So remove it again, then start a capture and rejoin it?

[Koenkk]

You can also try that now (I assume it still responds to on/off commands)

[sjorge]

OK, good news... after I restarted z2m after the update I got the timeout again!

zigbee2mqtt:error 2020-03-21 20:27:00: Failed to setup reporting for '0x0017880104259333' - Error: ConfigureReporting 0x0017880104259333/11 genOnOff([{"attribute":"onOff","minimumReportInterval":0,"maximumReportInterval":300,"reportableChange":0}], {"timeout":6000,"manufacturerCode":null,"disableDefaultResponse":true}) failed (Error: Timeout - 27321 - 11 - 2 - 6 - 7 after 6000ms)
    at Endpoint.<anonymous> (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/controller/model/endpoint.js:316:23)
    at Generator.throw (<anonymous>)
    at rejected (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/controller/model/endpoint.js:6:65)

I will start a capture and restart z2m

[sjorge]

The pcapng is here: https://pkg.blackdot.be/extras/zigbee/hue_genOnOff.pcapng
It happened somewhere around packet nr 90 (I stopped the capture about 10 sec after the message, not much was happening after the fail it seems)

Then again it was a timeout... so the packet probably was send way before, I think packet nr 46... or maybe 16?

{"id":4,"type":"Router","ieeeAddr":"0x0017880104259333","nwkAddr":27321,"manufId":4107,"manufName":"Philips","powerSource":"Mains (single phase)","modelId":"LCT012","epList":[11,242],"endpoints":{"11":{"profId":49246,"epId":11,"devId":528,"inClusterList":[0,3,4,5,6,8,4096,768,64513],"outClusterList":[25],"clusters":{"genBasic":{"attributes":{"modelId":"LCT012","manufacturerName":"Philips","powerSource":1,"zclVersion":1,"appVersion":2,"stackVersion":1,"hwVersion":1,"dateCode":"20191218","swBuildId":"1.50.2_r30933"}},"genOnOff":{"attributes":{"onOff":1}},"lightingColorCtrl":{"attributes":{"currentX":29972,"currentY":26871,"colorTemperature":366}},"genLevelCtrl":{"attributes":{"currentLevel":254}}},"binds":[{"cluster":6,"type":"endpoint","deviceIeeeAddress":"0x00124b001938a7e5","endpointID":1}]},"242":{"profId":41440,"epId":242,"devId":97,"inClusterList":[33],"outClusterList":[33],"clusters":{},"binds":[]}},"appVersion":2,"stackVersion":1,"hwVersion":1,"dateCode":"20191218","swBuildId":"1.50.2_r30933","zclVersion":1,"interviewCompleted":true,"meta":{},"lastSeen":1584823046183}

[Koenkk]

It seems that the bulb doesn't respond anymore the the configure reporting commands, binds are OK (giving a bind response) but the configure reporting only give a default response (where it should do a configure reporting response). Maybe Philips removed it in the new firmware version?

[sjorge]

@Koenkk I guess we need someone else that has update a bulb, and then removed/joined it again to confirm.

It was still reporting fine after the upgrade until I removed it and joined it again to update the firmware version.

I had a peak at deconz there stuff but I can't figure out if they setup reporting or it... it seems to poll hue bulbs 🤔

[Kryzek]

Would it be possible that reporting setup changed on Hue environment somehow due later updates? Could it help if we sniff the traffic with Hue bridge to check if it does something differently (or does not try do reporting configuration at all)?

[sjorge]

@Kryzek I only have one bulb withnout the bridge.
After the OTA it worked fine, including reporting but it had the wrong version info.

Then I removed and joined it and now reporting is broken, sadly on/off is like the bare minimum IMHO. I also tried after resetting the bulb with touchlink, same result.

[Kryzek]

I have Hue bridge and couple of LCT010's which also do give this at startup:
Failed to setup reporting for '0x001788010278b40a' - Error: ConfigureReporting 0x001788010278b40a/11 genOnOff([{"attribute":"onOff","minimumReportInterval":0,"maximumReportInterval":300,"reportableChange":0}], {"timeout":6000,"manufacturerCode":null,"disableDefaultResponse":true}) failed (Error: Timeout - 54154 - 11 - 2 - 6 - 7 after 6000ms)

These are not new firmware devices ("dateCode":"20181115","swBuildId":"1.46.13_r26312") but if sniffing these ones with bridge would help I could give it a shot.

[Koenkk]

The thing is that the hue bridge doesnt do reporting, they use polling.

[sjorge]

These are not new firmware devices ("dateCode":"20181115","swBuildId":"1.46.13_r26312") but if sniffing these ones with bridge would help I could give it a shot.

Mine was older, it had a 20170908 date code.

The thing is that the hue bridge doesnt do reporting, they use polling.
Maybe we (I could give it a shot) should look at devicePoll that can be enabled globally and then an optional interval override per device.

We'd need to update zigbee-herdsman-converters to add which fields to poll though, but that doesn't seem like a horrible thing to do and it can be rolled out slowly. (And only use it for device that don't report, to keep traffic down)

Let me think about this for a few days during my walks.

[Kryzek]

That seems pretty counter-productive way to do it. Reporting would be most accurate and fastest way to get infoirmation to me, if something changes - report. Wonder why they wouldn't use reporting if they had it in place earlier in the firmware. Oh well.. 🤷‍♂

[sjorge]

That seems pretty counter-productive way to do it. Reporting would be most accurate and fastest way to get infoirmation to me, if something changes - report. Wonder why they wouldn't use reporting if they had it in place earlier in the firmware. Oh well.. 🤷‍♂

Yeah it sucks, guess I have to update https://www.zigbee2mqtt.io/information/report.html for hue now. (PR koenkk/zigbee2mqtt.io#244)

Might be time to switch to Innr instead of Hue, the colors are pretty good too an they seem to report everything just like trådfri.

[vke]

Is it possible to add OTA upgrade by sending local file path to firmware to /zigbee2mqtt/bridge/ota_update/update/FRIENDLY_DEVICE_NAME or something like this?

I'm talking about firmware updates for custom devices when developing Zigbee device firmware, so there is no reason to upload firmware to github.

[Koenkk]

@vke this is not possible, but it's quite easy to setup an OTA server yourself

• Create a private fork of https://github.com/koenkk/zigbee-ota (or you anything else to host e.g. a python webserver)
• Add your image with instruction of https://github.com/koenkk/zigbee-ota#adding-new-and-updating-existing-ota-files
• Point https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/ota/zigbeeOTA.js#L2 to your own server
• Add OTA for your device (example: https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/devices.js#L1457)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.