Openpilot: Model lagging after update to revision a6e8778ae6ee6ef4a8ea6805a5457a827b8c0355

[briskspirit]

Describe the bug

Latest master-ci (revision a6e8778ae6ee6ef4a8ea6805a5457a827b8c0355) totally broke 0.8.0 . Constantly getting model lagging error.
Updated from revision 3328cf50b49a9ba429f249394190728fdbb640ab

How to reproduce or log data

Update to git revision a6e8778ae6ee6ef4a8ea6805a5457a827b8c0355

Expected behavior

No lagging

Device/Version information (please complete the following information):

• Device: Comma Two
• Dongle ID: f66032c2b5aa18ac
• Route: f66032c2b5aa18ac
• Timestamp: 2020-11-20--09-27-38, 2020-11-20--09-23-54, 2020-11-20--09-22-33
• Version: a6e8778ae6ee6ef4a8ea6805a5457a827b8c0355
• Car make/model: Toyota Corolla 2021 SE

[geohot]

Tracking in #2546, there was an issue in the lag detector that was fixed recently.

[briskspirit]

Model lagging unfortunately was not fixed with your commit...

Device: f66032c2b5aa18ac
Route: f66032c2b5aa18ac|2020-11-20--17-39-47
Exact segments:
f66032c2b5aa18ac|2020-11-20--17-39-47--4
f66032c2b5aa18ac|2020-11-20--17-39-47--5
f66032c2b5aa18ac|2020-11-20--17-39-47--6
f66032c2b5aa18ac|2020-11-20--17-39-47--14

Now it was definitely better than in the morning (when I had panics after 5-10 sec of driving)
Commit I was on : f10783b57501e123857502de8358f04a20095bf5

[geohot]

Can confirm model lags on "f66032c2b5aa18ac/2020-11-20--17-39-47/4"

But also, dmonitoring at 14ms. So that commit seemed to fix that issue, but not enough for the main model lags. Perhaps the governors will fix it.

[briskspirit]

Could you also check f66032c2b5aa18ac|2020-11-19--17-35-53 ? My point is that on previous commit there was no lagging panics. Maybe something got broken by another commit? Just to compare on the same device/car/etc.

[geohot]

The lag detector was tightened yesterday. The lag was always there, you just didn't know about it.

[adam-nelson]

I can comment that still getting driver lag on comma two with 18e8ac91b805db11 on latest master-ci

I left my comma two offline in the meantime (in glovebox) and defaulted to factory tss2 for tonight.

Let me know when there a new update to test

[Martint1980]

Had a couple of issues today
693d6317215d8dfa|2020-11-21--13-18-22--4
693d6317215d8dfa|2020-11-21--13-18-22--8
693d6317215d8dfa|2020-11-21--13-18-22--9
693d6317215d8dfa|2020-11-21--11-52-14--21
693d6317215d8dfa|2020-11-21--11-52-14--23
693d6317215d8dfa|2020-11-21--12-25-28--7
693d6317215d8dfa|2020-11-21--12-25-28--11

[briskspirit]

Can confirm that commit d703e88f11e9a9ea3dc9f8f375fb334222d6bebb made it better but didn't fixed model lagging problem.
Segments:
f66032c2b5aa18ac|2020-11-21--13-02-06--6
f66032c2b5aa18ac|2020-11-21--13-02-06--10

[udiram]

commit d703e88 caused almost non stop model lag errors, wouldn't even let me activate op, reboot seemed to help a little.
1ea81d35a5792cef|2020-11-21--17-53-54--22

[briskspirit]

Latest commit for GPU priority and governors fixed the bug.