esp-homekit-devices 🚀 - OTA Updates broken: Manual update

https://github.community/t5/GitHub-API-Development-and/GitHub-changed-the-capitalisation-of-the-HTTP-headers-and-OTA/m-p/48247

RavenSystem on 29 Feb 2020

If everyone follows this link and kudos this request, it might actually help convince the GitHub team...

HomeACcessoryKid on 29 Feb 2020

@RavenSystem @HomeACcessoryKid
Thank you ... just do it

GPL71 on 29 Feb 2020

https://github.com/HomeACcessoryKid/life-cycle-manager/pull/22

RavenSystem on 29 Feb 2020

👍1

New installations have been fixed and they will work. Ensure that you are using the last haaboot.bin

RavenSystem on 29 Feb 2020

do I have to make a re-flash of all the accessories?

gnoto on 1 Mar 2020

👀10

New installations have been fixed and they will work. Ensure that you are using the last haaboot.bin

How do i change the haaboot.bin on a 1.7-Device? is there a way without FTDI-flash?

archiolochos on 3 Mar 2020

👍1

@RavenSystem can you update us please?
Do we need to flash all devices physically ? Or u will release update?

sblitzer on 3 Mar 2020

All,

There is a request outstanding with the GitHub support team.

IF they roll back the change they did on the HTTP header “Location:” then all will work again and you can download the next version of haaboot or lcm.

So, I recommend some patience since this kind of organisations do not act in a 2 workdays notice.

I still have a reasonable hope this will be fixed in the next two weeks.

Go to the page https://github.community/t5/GitHub-API-Development-and/GitHub-changed-the-capitalisation-of-the-HTTP-headers-and-OTA/td-p/48247 and give a kudos to the repair request.

Meanwhile, cross your fingers,
my 2 cents,
HacK

On 3 Mar 2020, at 20:11, sblitzer notifications@github.com wrote:

@RavenSystem https://github.com/RavenSystem can you update us please?
Do we need to flash all devices physically ? Or u will release update?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/RavenSystem/esp-homekit-devices/issues/825?email_source=notifications&email_token=AFOQOQSFNT7VZHMAP5A3USTRFVI7JA5CNFSM4K6X5GH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENUYLII#issuecomment-594118049, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFOQOQSKHXBPAQIB6PMZNULRFVI7JANCNFSM4K6X5GHQ.

HomeACcessoryKid on 3 Mar 2020

We are still waiting on github to reverse the change (temporarily). Until then we have to wait. Flashing by wire is always an option but it will mean all HomeKit ID’s will be reset. There is a way to do it without resetting HomeKit ID and @RavenSystem has informed us in the discord that he will make a step by step of how to do so if github is not willing to reverse the change. For now just don’t panic and wait patiently 😊.

tonysprenk on 3 Mar 2020

👍4

I’m planning on flashing a new device.

@RavenSystem, how about you release a new HAA OTA version, which tries handling both old and new HTTP Headers? Something based on latest LCM Beta.
Who knows how much time GitHub can stall on this issue and what final decision they may do.

UPD: My fault, HAA OTA v2.2+ already has new headers.
Guess latest v2.2.2 won’t support old ones, if they would be reverted back by GitHub.

i3laze on 15 Mar 2020

@i3laze For new installations its been fixed.

New installations have been fixed and they will work. Ensure that you are using the last haaboot.bin

Kienz on 15 Mar 2020

👍1

@RavenSystem so what's the way out of this? Will you release instructions to flash the OTA without resetting the HomeKit id/settings? What should we do?

morphix9 on 18 Mar 2020

👍1

I have updated my LCM with several features, including an emergency mode.
This can download a new otaboot from any http server.
Since it verifies the signature it is still protected against hijacking the device.
I suggest HAA gets updated with these features as well…

I for one have tested my new LCM to work by uploading only to sector 0x2000 and this preserves the HomeKit settings.

HomeACcessoryKid on 18 Mar 2020

👍7 🎉2 🚀1

Is there a way to fake the GitHub servers to update to the last firmware?
Is the update process checking the validity of the GitHub certificate when updating?

macjl on 23 Mar 2020

No and Yes, the entire point of LCM is to protect against malware injection. Afterward the emergency mode should have been there from the beginning, but this is how we learn from our mistakes.

HomeACcessoryKid on 24 Mar 2020

I have updated my LCM with several features, including an emergency mode. This can download a new otaboot from any http server. Since it verifies the signature it is still protected against hijacking the device. I suggest HAA gets updated with these features as well… I for one have tested my new LCM to work by uploading only to sector 0x2000 and this preserves the HomeKit settings.

@HomeACcessoryKid Do you have an example?

Kienz on 24 Mar 2020

esptool.py -p /dev/cu.usbserial-* --baud 230400 write_flash -fs 8m -fm dout -ff 40m
0x2000 versions/1.9.5v/otaboot.bin

HomeACcessoryKid on 24 Mar 2020

Better do this, it then preserves the bootflags as well:

esptool.py -p /dev/cu.usbserial-* write_flash 0x2000 versions/1.9.5v/otaboot.bin

HomeACcessoryKid on 24 Mar 2020

👍1

But this can't be done if you have the HAA OTA currently installed, right?

morphix9 on 24 Mar 2020

no, the example was given for LCM otaboot
I for one do not use HAA ota (yet)

HomeACcessoryKid on 24 Mar 2020

@RavenSystem are you planning to do an update on the OTA? Can you share some info?

morphix9 on 24 Mar 2020

I managed to update without a reset.

xrust83 on 31 Mar 2020

share your secret

hillaliy on 4 Apr 2020

share your secret

@hillaliy read first comment.

RavenSystem on 4 Apr 2020

I managed to update without a reset.

it would be better, update without flash. 😉

claire078 on 4 Apr 2020

I don't understand,
How can I update without flash mode

hillaliy on 4 Apr 2020

Guys, the link found in the original post (https://github.community/t5/GitHub-API-Development-and/GitHub-changed-the-capitalisation-of-the-HTTP-headers-and-OTA/m-p/48247) no longer works.

Any progress with Github?

peros550 on 5 Apr 2020

😕1

Wow that’s strange. Just a few hours ago I contacted github support asking them what the status was of this issue. I referenced the above link in my request. I verified that the link was operational. Now it seems the issue has been removed from the support pages.

tonysprenk on 5 Apr 2020

I had made post with a "vote" to see if people were in favour of making a fuz or giving up.
When I updated the main post with a link to post#15 the article was marked as spam

I have used the feedback form for this, indicating it is not spam. And asking for a reply, any reply.
If we are lucky, on Monday they might un-spam it.
Anyway, LCM version 2.0.0 is almost ready and Jose is looking at a backup server mode...
So either way, progress might be made...

HomeACcessoryKid on 5 Apr 2020

I had made post with a "vote" to see if people were in favour of making a fuz or giving up.

When I updated the main post with a link to post#15 the article was marked as spam

I have used the feedback form for this, indicating it is not spam. And asking for a reply, any reply.

If we are lucky, on Monday they might un-spam it.

Anyway, LCM version 2.0.0 is almost ready and Jose is looking at a backup server mode...

So either way, progress might be made...

Thanks allot for the clarification! As I understand it the solutions you mention are only for newly flashed devices right? Not for devices that are already flashed OTA?

tonysprenk on 5 Apr 2020

Correct. Unless GitHub changes that header, all those devices cannot be updated without serial flashing. I those cases, if you use HAA, you can also switch to HAA_OTA if you estimate it to be sufficiently able to protect you in the future for ever having to go through this again.
I still have to evaluate HAA_OTA so cannot tell if it is up to it already.

HomeACcessoryKid on 5 Apr 2020

@HomeACcessoryKid Supposing Github will not help, isn't any other way to save those devices running already a previous version of LCM? I have a few smart bulbs which were originally flashed with Tuya convert (OTA) hence having to flash by wire is quite difficult

peros550 on 5 Apr 2020

👍1

If there would have been the remotest way of achieving anything, we would have found it by now.
Key observation is that your device is a state machine that acts on what it gets.
Since you cannot ADD anything to and it is therefor driving to a crash, there is no way around.

HomeACcessoryKid on 5 Apr 2020

I had made post with a "vote" to see if people were in favour of making a fuz or giving up.
When I updated the main post with a link to post#15 the article was marked as spam

I have used the feedback form for this, indicating it is not spam. And asking for a reply, any reply.
If we are lucky, on Monday they might un-spam it.
Anyway, LCM version 2.0.0 is almost ready and Jose is looking at a backup server mode...
So either way, progress might be made...

Keep us updated plz.

Thnx!

seritos on 6 Apr 2020

LCM 2.0.1 is now the latest release. The readme has instructions to upgrade using solder iron.
No news from GitHub

HomeACcessoryKid on 12 Apr 2020

Tried to upgrade a SonoffBasic from RavenCore 0.5.15 by using LCM 2.0.1 and it turns out that HAA overwrites LCM otamain code by using a very big 8 sector sysparam definition.

Considering that HAA_OTA still doesn't have an emergency mode, I do not want to switch to HAA_OTA, at least not till that is done

What should be do here?
Maybe fork HAA and bring sysparam back to normal sized 2 sectors which will not clobber LCM?

HomeACcessoryKid on 13 Apr 2020

IMPORTANT UPDATE:
if you upgrade from RC or HAA version without dummy switch to go in setup mode, this procedure create itself dummy switch!!! It isn't necessary to do any reset of Homekit ID, so you haven't to touch/modify any existing homekit accessory and automation; you have only to move dummy switch to righe room...

this is very useful to speed up migration ta vastest version of HAA with working OTA features!!!!

You have only to add again accessory to the right scenes...

have a good migration task!

pilot1981 on 13 Apr 2020

Tried to upgrade a SonoffBasic from RavenCore 0.5.15 by using LCM 2.0.1 and it turns out that HAA overwrites LCM otamain code by using a very big 8 sector sysparam definition.

Considering that HAA_OTA still doesn't have an emergency mode, I do not want to switch to HAA_OTA, at least not till that is done

What should be do here?
Maybe fork HAA and bring sysparam back to normal sized 2 sectors which will not clobber LCM?

Am I get it right? Newest versions of HAA are not compatible with LCM v2?
If yes, that would be a big drawback for me because I liked LCM new features (emergency mode & repo reset tool)

peros550 on 13 Apr 2020

Jose and i are working on that

BR,
HacK

HomeACcessoryKid on 13 Apr 2020

Thank you guys! I think it's good thing that Github hasn't fixed their issue yet.

peros550 on 13 Apr 2020

@peros550 you can use dummy switch to enter in setup mode?!

pilot1981 on 13 Apr 2020

Hi @pilot1981

I have a few HAA devices running LCM old version and HAA 1.6.4. I can always go into setup mode using either the emergency mode or switching on-off (8x).

I am not fun of the idea of having an additional dummy switch, unless there is a good reason for that....?

peros550 on 14 Apr 2020

I would find those virtual switches quite useful for some devices, where it's hard to do the reset. But can't find something in the wiki on that. Are those possible with HAA or was this only possible with the old RavenCore?

morphix9 on 14 Apr 2020

I have no Idea how far your security is implemented but ... is it maybe possible to modify the headers with a proxy?

"Requestly" or "Charles" for example can do such things.

Really hope there will be ANY other solution than flash the devices over the pin headers again 🤦‍♂️

Shimakuchi on 16 Apr 2020

I don't know if you can tell LCM to use an http proxy. Is there a way to do that?

macjl on 16 Apr 2020

A transparent proxy maybe? Is it encrypted or unencrypted traffic?

uDuCkV on 16 Apr 2020

Encrypted. That's why transparent proxy won't be able to change headers.

macjl on 16 Apr 2020

and not only encrypted but actively checking the server certificates!

HomeACcessoryKid on 16 Apr 2020

If we could fake the github server than anyone could and none of our devices would be safe. Maybe it’s possible to create a proxy that has the right URL that the HAA device can connect to which can relay the information from the actual github server. Not sure if that’s even physically possible with the certificates etc. And again it would mean that anyone could use that method to gain access to all our devices.

tonysprenk on 16 Apr 2020

@HomeACcessoryKid In general LCM check validity of certificates, which is a highly secure thing to do, but haven't you designed it in a way to trust certificates signed by you?

If yes, maybe we can fix this

peros550 on 16 Apr 2020

The 1.0.0 version did not contemplate the scenario where GitHub would make http headers incompatible. It does handle if GitHub will update their certificates (bound to happen one day, e.g. if they change their certificate provider). But without the basics of http working, it is broken and no way around it (unless you want to break GitHubs certificates, which would be breaking news around the world.
Anyhow, only with version 1.9.1+ will you be able to use GitHub again. And versions 2.0.0+ have an emergency mode that can save a device without needing GitHub.

HomeACcessoryKid on 18 Apr 2020

Am I get it right? Newest versions of HAA are not compatible with LCM v2?
If yes, that would be a big drawback for me because I liked LCM new features (emergency mode & repo reset tool)

The version 2.0.2 uses a special version created by @RavenSystem that is compatible with LCM.
@peros550, although already tested by myself, please check it out?

HomeACcessoryKid on 18 Apr 2020

@RavenSystem @HomeACcessoryKid Many thanks for your works guys.

I just flashed the 2.0.2 otaboot.bin and watched the magic happening. LCM updated to latest version, then rboot, then HAA updated.

The Sonoff S20 kept Homekit settings. As soon as the process completed, the accessory was available again in Home app.

But without the basics of http working, it is broken and no way around it
@HomeACcessoryKid I'm not a security expert and don't try to pretend one. I was trying to think how we could alter http headers and came across one tool, it's called mitmproxy. This tool is designed to intercept http/https and also has an API to make changes on the fly [1] [2] and alter the http responses in the way we want.

I actually followed this guide to install it on a raspberry and do some tests.

Assuming the http header issue can be solved, the next problem with this approach is trust. As you said, we can't brake security of Github certificates. On the other hand, this tools seems to provide some options for using Custom certificates, or CAs in the Certificates section

Do you think it's possible to setup a system like that and try to OTA devices that can't be flashed using soldering method?

peros550 on 18 Apr 2020

Unless the device trusts MitM certificate representing the GitHub, there’s no way faking it.
What happened with support ticket at Github portal? Any news from the support?

i3laze on 30 Apr 2020

Just received this message from support:

Apparently, the GitHub Community Forum entry has since been deleted. As such I have no insight into what exactly changed which broke this project's update procedure. As far as I can tell, they seem to mention a URL change of some sort.

I'm not sure if the following is applicable to the current situation, but some projects rely on undocumented URLs or query data in a way that is not via GitHub's API. In those cases, we can't provide support for any problems they might encounter.

API changes are usually communicated well ahead of time and come with deprecation warnings, unless an endpoint is part of a preview phase for which there's always a disclaimer that syntax and behavior can change without prior notice.
Let me know if you have any further questions.

tonysprenk on 11 May 2020

Google cache has no copies of the original issue.
However, Yandex cache does :)

I have a few RC devices left not flashed to HAA, however any resolution from GitHub is still welcome.

@HomeACcessoryKid, @tonysprenk,
Please, use text from the cache URL above to reproduce the ticket.

i3laze on 11 May 2020

or, you can use the ticket number of the original request for support.

Thank you for contacting GitHub Developer Support. We wanted to let you know that we've received your message and will get to it as quickly as possible.

Ticket ID: 582776

And the issue is not with the API but with the HTTP of the server, so that is below this API.
Although the RFC of HTTP allows using any capitalisation, the point is that changing this serves no use but creates this kind of nuisance

Below the text of the original support ticket request of 29 Feb 2020

Since today GitHub has changed their HTTP headers and has chosen to use lowercase instead of the normal capital first letter.
This results in the header "Location: "(RFC2616:14.30) to have become "location: "
MANY amateur coders have no code in place for case-insensitive parsing because they are not aware!
While I have now found out that the RFC allows for the capitalization to be insensitive, the reasonable thing to do for GitHub is to stick to the text as presented in the RFC and use "Location: "

In my case there are over 20.000 devices out there in the world that depend on my code to update via OTA
HomeACcessoryKid/life-cycle-manager and they cannot anymore.
Of course the self-update of OTA will become smarter, but for the sake of helping the community I really really hope that GitHub will fix this back to the old RFC suggested format. It doesn't hurt them and helps amateurs around the world.
Thanks,
HacK
PS. this has also been posted to the forum for API but I feel there it is a bit out of place...
https://github.community/t5/GitHub-API-Development-and/GitHub-changed-the-capitalisation-of-the-HTTP-headers-and-OTA/m-p/48247 https://github.community/t5/GitHub-API-Development-and/GitHub-changed-the-capitalisation-of-the-HTTP-headers-and-OTA/m-p/48247
PPS. I hope this doesn't happen to AWS as well, but for now it is OK
PPPS. There are still 'normal' headers in the reply, just not the top ones anymore
HTTP/1.1 302 Found
date: Sat, 29 Feb 2020 14:47:10 GMT
content-type: text/html; charset=utf-8
server: GitHub.com http://github.com/
status: 302 Found
vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
location: https://github.com/HomeACcessoryKid/life-cycle-manager/releases/tag/1.0.0 https://github.com/HomeACcessoryKid/life-cycle-manager/releases/tag/1.0.0
cache-control: no-cache
Age: 0
Set-Cookie: Set-Cookie: logged_in=no; Path=/; Domain=github.com http://github.com/; Expires=Mon, 01 Mar 2021 14:47:10 GMT; HttpOnly; Secure
Content-Length: 139
X-GitHub-Request-Id: DF95:2F739:3CBF7FE:57D6ABE:5E5A796E

HomeACcessoryKid on 11 May 2020

@HomeACcessoryKid just got this response from github support. I also sent you a private message on discord.

tonysprenk on 13 May 2020

👀1 👍1

@tonysprenk did they ever answer?
I believe its good news an answer for them... isnt it?

seritos on 13 May 2020

@seritos yes the answer from today is above but I don’t really know what it means so I cannot write back. Any ideas would be greatly appreciated.

tonysprenk on 13 May 2020

Looks like GitHub support tested wrong URL api.github.com, which returned them old fashioned “Location”.
I believe @HomeACcessoryKid will point them in right direction.

i3laze on 13 May 2020

@i3laze what should be the right url?

tonysprenk on 13 May 2020

That’s a question for @HomeACcessoryKid, however it seems everything on github.com itself (not api.github.com), like:
curl -v https://github.com/HomeACcessoryKid/ota/releases/download/0.1.0/certs.sector.sig

It gets redirected to AWS with “l”ocation header:
```
GET /HomeACcessoryKid/ota/releases/download/0.1.0/certs.sector.sig HTTP/1.1

Host: github.com
User-Agent: curl/7.55.1
Accept: /
< HTTP/1.1 302 Found
< date: Thu, 14 May 2020 04:52:52 GMT
< content-type: text/html; charset=utf-8
< server: GitHub.com
< status: 302 Found
< vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
< location: https://github-production-release-asset-2e65be.s3.amazonaws.com/118357494/
```

i3laze on 14 May 2020

correct, the right URL for LCM and HAA starts with https://github.com/
They are a but stupid because the cURL output is part of the original request and it clearly states that the server is GitHub.com

But Kudos for insisting and getting their attention !!!!!

HomeACcessoryKid on 14 May 2020

@tonysprenk While you are at it, can you ask them what is the MOST EFFICIENT way to collect a file from the latest release and a file from a specific release using the API system.
I didn't use that for two reasons:

It didn't exist when I started development.
It looks horribly complicated

HomeACcessoryKid on 14 May 2020

Thanks for the responses guys. I figured it out last night with help of @RavenSystem and this was my response to them:

Thanks allot Dominic for taking the time.

Yes that is the issue I’m talking about, I wasn’t clear in my initial explanation.

When I do:

curl -v github.com/RavenSystem/haa/releases/latest/download/otamain.bin

it returns:

*   Trying 140.82.118.3...
* TCP_NODELAY set
* Connected to github.com (140.82.118.3) port 80 (#0)
> GET /RavenSystem/haa/releases/latest/download/otamain.bin HTTP/1.1
> Host: github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-length: 0
< Location: https://github.com/RavenSystem/haa/releases/latest/download/otamain.bin
< 
* Connection #0 to host github.com left intact
* Closing connection 0

But the code was written with location: in mind not Location:.

New code has been updated to deal with this capitalization change but thousands of IOT devices are still left without a way of updating OTA.

If this change could be reverted back to location: even just for a few days it would give al of us the chance to update all these devices.

Thanks again for all your support.

When I get a response back I will ask about the API.

tonysprenk on 14 May 2020

👍2 👀1

Better correct this answer, you flipped the L and l one time more !!!

Also, LCM is using https, not http ! and it uses a search for the version and then direct reference to the files within that version

--- ota_get_version
--- ota_connect LocalPort=e550 DNS IP:140.82.118.3 local..OK remote..OK SSL..OK set_fd to github.com port 443..OK
GET /HomeACcessoryKid/life-cycle-manager/releases/latest HTTP/1.1
Host: github.com

sent OK
HTTP returns 302 for HomeACcessoryKid/life-cycle-manager@version:"1.0.0" according to latest release
--- ota_get_file_ex
GET /HomeACcessoryKid/life-cycle-manager/releases/download/1.0.0/latest-pre-release HTTP/1.1
Host: github.com

--- ota_connect LocalPort=e551 DNS IP:140.82.118.3 local..OK remote..OK SSL..OK set_fd to github.com port 443..OK
sent OK
HTTP/1.1 302 Found
date: Sat, 04 Apr 2020 18:40:23 GMT
content-type: text/html; charset=utf-8
server: GitHub.com
status: 302 Found
vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
location: https://github-production-release-asset-2e65be.s3.amazonaws.com/127610504/d8fe7600-76b3-11ea-9ed4-fdf3e9751c87?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200404%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200404T184023Z&X-Amz-Expires=300&X-Amz-Signature=9e77f13566e5ced1a05700cb037f025c28b42a7734572dd3dac4ac4f384d7f10&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlatest-pre-release&response-content-type=application%2Foctet-stream
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
content-security-policy: default
HTTP returns 302
--- ota_connect LocalPort=e552 DNS IP:52.217.47.28 local..OK remote..OK SSL..OK set_fd to github-production-release-asset-2e65be.s3.amazonaws.com port 443..OK
send request......OK
312e39...392e38  so far collected 5 bytes
--- ota_boot...1
HomeACcessoryKid/life-cycle-manager@version:"1.9.8"
--- ota_get_hash
--- ota_get_file_ex
GET /HomeACcessoryKid/life-cycle-manager/releases/download/1.9.8/certs.sector.sig HTTP/1.1
Host: github.com

--- ota_connect LocalPort=e553 DNS IP:140.82.118.3 local..OK remote..OK SSL..OK set_fd to github.com port 443..OK
sent OK
HTTP/1.1 302 Found
date: Sat, 04 Apr 2020 18:40:28 GMT
content-type: text/html; charset=utf-8
server: GitHub.com
status: 302 Found
vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
location: https://github-production-release-asset-2e65be.s3.amazonaws.com/127610504/b40a0300-76b3-11ea-9188-02ab6b820bfb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200404%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200404T184028Z&X-Amz-Expires=300&X-Amz-Signature=55d1deda80957e1da19ca1d3f51283129a8efede9c4ce050ae237858d46c9e91&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dcerts.sector.sig&response-content-type=application%2Foctet-stream
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
content-security-policy: default-s
HTTP returns 302
--- ota_connect LocalPort=e554 DNS IP:52.217.36.220 local..OK remote..OK SSL..OK set_fd to github-production-release-asset-2e65be.s3.amazonaws.com port 443..OK
send request......OK
8c1d63...c6ef10  so far collected 155 bytes

HomeACcessoryKid on 14 May 2020

I think I’m misunderstanding. Sorry.

This is the response I just got:

Hi Tony,

I'm not aware of a recent change which would result in the behavior you describe, but since I can't be sure I also looked through old support requests where people sent us their console output. Apparently it's always been Location with a capital L. I can't rule out the possibility that for some reason there was a brief period during which we sent out location with a lowercase l, but I can't find any indication for that being the case.

Currently, there are no plans to change GitHub's API headers going forward, not even temporarily.

Regards,
Dominik

@HomeACcessoryKid could you please formulate a clear answer for me to send back?

tonysprenk on 14 May 2020

Please add me in that email loop with Github and I'll participate there, OK?

HomeACcessoryKid on 14 May 2020

please find the mail we sent to GitHub.

Hi Dominik,

The issue has been misrepresented since the reply of May 13th.

In 2017 I started the code to do over the air updates and then the API was not well
known or even existing. I used the web pages of https://GitHub.com instead so
api.github.com is not involved in this request .

github.com used to reply like this (actual log files, not made up!) and note the date!

======================
GET /HomeACcessoryKid/life-cycle-manager/releases/latest HTTP/1.1
Host: github.com

sent OK
HTTP returns 302 for HomeACcessoryKid/life-cycle-manager@version:"1.0.0" according to latest release
--- ota_get_file_ex
GET /HomeACcessoryKid/life-cycle-manager/releases/download/1.0.0/latest-pre-release HTTP/1.1
Host: github.com

--- ota_connect LocalPort=ffe4 DNS IP:140.82.118.4 local..OK remote..OK SSL..OK set_fd to github.com port 443..OK
sent OK
HTTP/1.1 302 Found
Date: Fri, 14 Feb 2020 13:48:52 GMT
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Status: 302 Found
Vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/127610504/9207c600-4f37-11ea-841c-3a6383191e25?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200214T134851Z&X-Amz-Expires=300&X-Amz-Signature=9e4f9c40873135374a62839229ddcb1e636922fccc528d459693ec94160b0b19&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlatest-pre-release&response-content-type=application%2Foctet-stream
Cache-Control: no-cache
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
====================

Note that the 302 redirect uses the Header field “Location” with capital L

All our code prior to 29 February 22:00 2020 is written with case sensitive parsing
of this header field And this is where 10s of thousands of devices are at, parsing
this field case sensitive


Then, Saturday morning Feb 29 2020 we noticed that https://github.com/ started
to reply with “location” with lower case “l"

======================
GET /HomeACcessoryKid/life-cycle-manager/releases/latest HTTP/1.1
Host: github.com

sent OK
HTTP returns 302 for HomeACcessoryKid/life-cycle-manager@version:"1.0.0" according to latest release
--- ota_get_file_ex
GET /HomeACcessoryKid/life-cycle-manager/releases/download/1.0.0/latest-pre-release HTTP/1.1
Host: github.com

--- ota_connect LocalPort=d2c1 DNS IP:140.82.118.4 local..OK remote..OK SSL..OK set_fd to github.com port 443..OK
sent OK
HTTP/1.1 302 Found
date: Sat, 29 Feb 2020 22:07:20 GMT
content-type: text/html; charset=utf-8
server: GitHub.com
status: 302 Found
vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
location: https://github-production-release-asset-2e65be.s3.amazonaws.com/127610504/0f425600-5b48-11ea-8846-85333ccb4101?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200229%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200229T220720Z&X-Amz-Expires=300&X-Amz-Signature=ead5fb7179c78b6e0f1dc881a68113bd90f0a864b0440793f70f4ea81059f907&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlatest-pre-release&response-content-type=application%2Foctet-stream
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
===================

Note that this log is the same except the “l", but was made using a version that has
been modified to parse this field case insensitive The versions before (1.9.0) would
have crashed by now.

The point is that those 10s of thousands of devices can either not at all or with very
big efforts be updated manually.

The ask  is that the change in the header field of “location: “ be reversed back to “Location: “

The new code is much better in recovering from external factors, but the thousands of
owners with these devices that either need to be considered “bricked” or else have
to be pried out of connection boxes in walls etc. will be very grateful !!
They can then  apply the security fixes waiting for them (cyber war doesn’t stop)
and start using new features.

I hope this extensive explanation made it all clear.
And to answer your request explicit about the cURL output:
Request this: 
curl -v https://github.com/HomeACcessoryKid/life-cycle-manager/releases/download/1.0.0/latest-pre-release

Thanks in advance,
HomeACcessoryKid

The mail of 13 May below was very much wrong for the issue of LCM at hand

When I do: curl -v github.com/RavenSystem/haa/releases/latest/download/otamain.bin it returns:
```* Trying 140.82.118.3...

TCP_NODELAY set
Connected to github.com (140.82.118.3) port 80 (#0)

GET /RavenSystem/haa/releases/latest/download/otamain.bin HTTP/1.1
Host: github.com> User-Agent: curl/7.64.1
Accept: /

< HTTP/1.1 301 Moved Permanently
< Content-length: 0
< Location: https://github.com/RavenSystem/haa/releases/latest/download/otamain.bin
<

Connection #0 to host github.com left intact

Closing connection 0
``But the code was written with location: in mind not Location .New code has been updated to deal with this capitalization change but thousands of IOT devices are still left without a way of updating OTA. If this change could be reverted back to location: ` even just for a few days it would give al of us the chance to update all these devices.

HomeACcessoryKid on 17 May 2020

Good news! We got the helpdesk person Dominik inside GitHub understanding the issue.
He will try to get it fixed, but cannot promise. This is as expected, but it is progress!

Thanks for elaborating. I understand the situation you are in and will try to find
the reason why GitHub's response has changed for requests like the following:

curl -v https://github.com/HomeACcessoryKid/life-cycle-anager/releases/download/1.0.0/latest-pre-release

If it's at all possible to use capitalized headers like Location, I'll make a case for
it. However, I can't promise anything, as querying GitHub's web interface is not
a valid alternative to using GitHub's API, because URLs, headers, and even
content may change without prior notice.

Regardless of whether my investigation is successful or not, going forward I
highly recommend using the following API endpoint to retrieve a repository's
latest release's URL:

https://developer.github.com/v3/repos/releases/#get-the-latest-release

I am going to mark this ticket as solved, but if you still need any information from
us, please respond to this message and the ticket will be reopened.

Cheers,
Dominik

HomeACcessoryKid on 18 May 2020

👍4

Some bad news on this issue. The only option now to update the stuck devices is flashing by wire unfortunately.

Response from github support:

Hi,

I have news for you, albeit not what you were hoping for.

While the HTTP/1.1 specification simply states that headers are supposed to be case-insensitive, some of the tools we are using already switched to lowercase headers, following HTTP/2 specification which states that header should still be "compared in a case-insensitive fashion. However, header field names MUST be converted to lowercase prior to their encoding in HTTP/2." There are quite a few tools, proxies etc. that have decided to build a generalized HTTP processing that unifies both HTTP/1.1 & HTTP/2. Since the latter has the requirement for how headers are encoded, it means that HTTP/1.1 headers also end up encoded in all lowercase in these systems.

This is the reason that some responses you see already have (some) lowercase headers, depending on which tools are used to serve the respective request.
We're in the process of making people aware of these changes, as some of them are depending on the infrastructure we use to run GitHub.com.

I'm sorry that this broke your devices' update path, but I don't think there's anything I can do for you at this point.

Thanks,
Dominik

tonysprenk on 20 May 2020

@tonysprenk Many thanks for your efforts! Now, it is time to wire and flash.

RavenSystem on 20 May 2020

Btw. is the updated OTA still parsing some HTTP responses or are you using GitHub's API?

morphix9 on 20 May 2020

I have considered using the API, but it turns out the response of the API is very big, like 12k. This is a challenge to parse for a simple device like a ESP8266.
If you want to have a look at the one of LCM, try this URL

I had to reduce the HTTP collection of the binary by using blocks of 4k at a time, else the https didn't even survive a simple get.

So, even though I would like to, do not expect that such a switch will happen easily.

HomeACcessoryKid on 21 May 2020

What's wrong with parsing http headers? The issue is: headers are case insensitive, but (old) code was written to parse them case sensitive.

nikbyte on 21 May 2020

HAA OTA supports any http/https web server that supports partial content. Headers and responses are processed as case insensitive, and firmware can manage up to 5 redirections. It has been tested with Apache Web Server and Nginx.

RavenSystem on 21 May 2020

Yes. now it works perfectly except sometimes it can't connect even to local server. It writes "connection failed" and that's all. However the server is ok. The same issue with github.com too.

nikbyte on 21 May 2020

Hi, the url to haaboot.bin in the first comment is not working. @RavenSystem can you update?

pandarek on 21 Dec 2020

Esp-homekit-devices: OTA Updates broken: Manual update

Most helpful comment

All 80 comments

Related issues