Godot: 3.2 RC1 hangs on Windows, worked in beta 6 [GLES3] (regression from 796d35d)

[NieznanywInternetach]

Godot version: 3.2 RC1 official
(3.2 beta 6 works, is without the problem)

OS/device including version: Windows 8.1 (64bit)
AMD A8-6410 APU with AMD Radeon R5 Graphics

Driver package version - 13.302.1501-140320a-169666C-Toshiba
2D Driver Version - 8.1.1.1634
OpenGL® Version - 25.20.15000.13547
OpenCL™ Version - 10.0.2766.5
AMD Mantle Version - 9.1.10.0295
AMD Mantle API Version - 102400
(Wasn't sure which are relevant, so copying more)

Issue description:
Whenever i try to open a project, i get a popup about AMD display driver that crashed and was recovered while in the meantime the editor during splash screen stops responding (when i try to close it i get the info visible below). The prompt shows basically nothing.


Steps to reproduce:
Try to open any project while using AMD?

[clayjohn]

Can you copy the text from "problem details" and paste here. Also, in the meantime try updating your GPU drivers.

[akien-mga]

Does the Windows error dialog provide any useful logs?

Does it hang the same with GLES2 and GLES3?

[NieznanywInternetach]

Can you copy the text from "problem details" and paste here. Also, in the meantime try updating your GPU drivers.

I'm afraid i can't copy that, i can't even mark it

i'll try it manually, last time i checked it with Radeon's app it said everything is updated

[NieznanywInternetach]

Does the Windows error dialog provide any useful logs?

Does it hang the same with GLES2 and GLES3?

I'll try to find something, it doesn't provide anything with the isn't responding popup (if i can call it so)
Also, no, GLES2 works normally
So it's about GLES3

[NieznanywInternetach]

Does the Windows error dialog provide any useful logs?

I found something (in Windows' report archive)
I can't send it via browser though (as i check it can be opened with notepad)
https://cdn.discordapp.com/attachments/561684643313614867/667764011663884288/Report.wer

[NieznanywInternetach]

Can you copy the text from "problem details" and paste here. Also, in the meantime try updating your GPU drivers.

I just updated it, and the problem still remains
And here's the newest log (if that's what you meant, at least that's what i manage to find)
https://cdn.discordapp.com/attachments/561684643313614867/667773396917944321/Report.wer

Edit:
I've tried again to make a dump file of the process and i accidentally made Godot crash
Crash:
https://cdn.discordapp.com/attachments/561684643313614867/667785014104031281/Report.wer
Dump file: (over 330 MB, .DMP)
https://mega.nz/#!MgR0XIgI!HQlhqGGv1VDyTJXpwhw1Ln5pJ5XbV_T3_FWYfQy1elU

[clayjohn]

Between last running beta 6 and running RC1 did you have any major updates to your GPU drivers or OS? I searched up the "Hang Type" from your above report and it seems to be a common crash relating to OS updates/BIOS updates.

_Edit:_ Are you by any chance setting "force vsync" on your GPU settings?

[akien-mga]

Between last running beta 6 and running RC1 did you have any major updates to your GPU drivers or OS?

I don't think so, I asked them to retry beta 6 on Discord and they confirm that it was still working on the same system.

[akien-mga]

@NieznanywInternetach Could you test this build? It's the same as 3.2 RC 1 but with one commit reverted, which we have a hunch might be the cause for the issue.
https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot-3.2-rc1-test1.zip

[NieznanywInternetach]

Between last running beta 6 and running RC1 did you have any major updates to your GPU drivers or OS? I searched up the "Hang Type" from your above report and it seems to be a common crash relating to OS updates/BIOS updates.

_Edit:_ Are you by any chance setting "force vsync" on your GPU settings?

I didn't have, it's possible though i don't update OS as often as i should....
Anyway, i found something when i was looking for the force vsync setting - i opened the device's profile and manually updated drivers. It didn't solved the problem but triggered Godot's error lines

[NieznanywInternetach]

@NieznanywInternetach Could you test this build? It's the same as 3.2 RC 1 but with one commit reverted, which we have a hunch might be the cause for the issue.
https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot-3.2-rc1-test1.zip

sorry, it doesn't change anything.

[akien-mga]

Thanks for testing. And just to be 100% clear, if you run 3.2 beta 6 now on that system, it works without printing those initialize errors?

[NieznanywInternetach]

Thanks for testing. And just to be 100% clear, if you run 3.2 beta 6 now on that system, it works without printing those initialize errors?

Em... it's actually printed. The numbers of lines are different though. But GLES3's working

[akien-mga]

I can't explain why beta 6 would "work" (though those errors are fairly critical, chances are that it would not really work properly in a GLES3 game with meshes or lights) when RC 1 fails to start...

On the other hand those initialization errors do point at drivers that do not properly support OpenGL 3.3, which is surprising for AMD A8-6410 APU, which should support 4.3 or even 4.5 with latest drivers.

The drivers you had originally from Toshiba were quite old (2014). I'm not sure what version you got now which made things more verbose, but I would suggest that you try the latest drivers for this APU provided by AMD themselves: https://www.amd.com/en/support/apu/amd-series-processors/amd-a8-series-apu-for-laptops/a8-6410-radeon-r5-graphics

Windows might complain that those are not "made" for your hardware since usually laptop OEMs put some locks to force users to use their tested drivers from their own support website, but in my experience it's always safe and recommended to take the newer versions directly from the hardware vendor (here AMD).

[NieznanywInternetach]

I can't explain why beta 6 would "work" (though those errors are fairly critical, chances are that it would not really work properly in a GLES3 game with meshes or lights) when RC 1 fails to start...

On the other hand those initialization errors do point at drivers that do not properly support OpenGL 3.3, which is surprising for AMD A8-6410 APU, which should support 4.3 or even 4.5 with latest drivers.

The drivers you had originally from Toshiba were quite old (2014). I'm not sure what version you got now which made things more verbose, but I would suggest that you try the latest drivers for this APU provided by AMD themselves: https://www.amd.com/en/support/apu/amd-series-processors/amd-a8-series-apu-for-laptops/a8-6410-radeon-r5-graphics

Windows might complain that those are not "made" for your hardware since usually laptop OEMs put some locks to force users to use their tested drivers from their own support website, but in my experience it's always safe and recommended to take the newer versions directly from the hardware vendor (here AMD).

Well, i was working with 2D so far so didn't have a chance to experience that
also, when i said i updated my drivers first time

I meant downloading the app you just provide url to
And the error lines start showing when i "updated" it via my Windows

[akien-mga]

And the error lines start showing when i "updated" it via my Windows

It "updated" you back to a version from 2014 though, instead of the one from 2017 on AMD's website. That's because Toshiba stopped supporting your laptop and did not register newer versions as tested and compatible for their Windows 8 users. That Windows feature can't really be trusted, as it can only install the most recent version it knows about, which in this case is 13.302.1501.0 provided by Toshiba.

So I'd suggest to update manually back to AMD's 2017 version, which should bring you back to a better supported version.

Apparently it won't solve this bug, but it will likely fix those initialization issues which are a sign of a broken driver.

[Zireael07]

As someone who used to suffer from those init issues, GLES3 used to work (mostly, in practice I didn't see issues) with them (but still, there's the problem of RC1 not starting...)
How many commits between the two versions touched rendering? I seem to remember a PR to do with vsync and stutter mitigation?

[NieznanywInternetach]

And the error lines start showing when i "updated" it via my Windows

It "updated" you back to a version from 2014 though, instead of the one from 2017 on AMD's website. That's because Toshiba stopped supporting your laptop and did not register newer versions as tested and compatible for their Windows 8 users. That Windows feature can't really be trusted, as it can only install the most recent version it knows about, which in this case is 13.302.1501.0 provided by Toshiba.

So I'd suggest to update manually back to AMD's 2017 version, which should bring you back to a better supported version.

Apparently it won't solve this bug, but it will likely fix those initialization issues which are a sign of a broken driver.

Good to know

Yes, i've done it and no init error's printed
But Godot still doesn't work on GLES3, as expected

I also checked vsync in RC1 (in Project Settings) but both sets don't make any difference

[akien-mga]

Are you able to compile Godot from source? http://docs.godotengine.org/en/latest/development/compiling/compiling_for_windows.html

If so, our best shot now would probably that you run a git bisect between the beta 6 and rc 1 commits to try to pinpoint what causes the regression.

You can do that with:

// This is RC 1
git checkout ba7aca4199019529dec60555a5ff005f6692d281
scons p=windows -j4
// Check that you still reproduce the bug as with official binaries, otherwise the rest won't be useful.
git bisect start
git bisect bad
// This is beta 6
git checkout 0ab1726b43dbe81c96d208a41a582435b76fd058
git bisect good

Then it will automatically check out a commit in the middle to bisect, which you can compile, and issue either git bisect good if it works or git bisect bad if it doesn't. Rinse and repeat, eventually it should bring you to what caused the regression. Fair warning, that can take some time since each compilation may last between 5 and 20 min depending on how fast your CPU is.

[NieznanywInternetach]

Are you able to compile Godot from source? http://docs.godotengine.org/en/latest/development/compiling/compiling_for_windows.html

If so, our best shot now would probably that you run a git bisect between the beta 6 and rc 1 commits to try to pinpoint what causes the regression.

You can do that with:

// This is RC 1
git checkout ba7aca4199019529dec60555a5ff005f6692d281
scons p=windows -j4
// Check that you still reproduce the bug as with official binaries, otherwise the rest won't be useful.
git bisect start
git bisect bad
// This is beta 6
git checkout 0ab1726b43dbe81c96d208a41a582435b76fd058
git bisect good

Then it will automatically check out a commit in the middle to bisect, which you can compile, and issue either git bisect good if it works or git bisect bad if it doesn't. Rinse and repeat, eventually it should bring you to what caused the regression. Fair warning, that can take some time since each compilation may last between 5 and 20 min depending on how fast your CPU is.

I've never compiled anything more complicated than a mini-console program while learning C++, but sure, i can try. And as far as i know my thing, it'll take minimum 30 min though
But we'll see
i'll be updating this comment over time

[akien-mga]

Yeah it's a bit cumbersome when you're not familiar with the process, I wish we had saved builds for each commit to test with.

I can make a couple builds quickly to already start to reduce the range of possible problematic commits (basically doing the first couple of steps of git bisect manually).

[akien-mga]

Here's the first one to test (halfway between b6 and rc1):

• build 0.50: https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.20190114.7971ec74f9.exe.zip

If (0.50) is bad, then you can test:

• build 0.25: https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.20190113.04a1ae90df.exe.zip

If (0.50) is good, then you can test:

• build 0.75: https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.20190116.6135037551.exe.zip

[NieznanywInternetach]

Here's the first one to test (halfway between b6 and rc1):
https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.20190114.7971ec74f9.exe.zip

I'll check it
also, it seems i'll be able to compile it tommorow after 15 (21:23 over here) based i need to update VS (4.30~ GB) and my download speed is around 100kbps now (only at daytime achieving 2-3Mbps)

Edit:
0.50 bad, i'll check 0.25

0.25 is bad too

Edit2:
With @Calinou we've found the first working commit, now we'll be calibrating this debugging gun :^)

[akien-mga]

@Calinou and @NieznanywInternetach finished the git bisect started above (thanks!):

796d35d8b3af2ea78767149c07a2aae4bd0510cf is the first bad commit
commit 796d35d8b3af2ea78767149c07a2aae4bd0510cf
Author: clayjohn <[email protected]>
Date:   Sun Jan 12 14:45:31 2020 -0800

    Fix generation of irradiance map

 drivers/gles3/shaders/cubemap_filter.glsl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

That's 796d35d8b3af2ea78767149c07a2aae4bd0510cf.

It's a bit surprising as there's already code using textureLod() similarly in the same shader, so it shouldn't be a problem of textureLod() not being supported a priori.

I'll make a build of RC 1 with that commit reverted to confirm that it's indeed the problematic change.

[akien-mga]

@NieznanywInternetach @katoneko Could you try this build?
https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.3.2.rc1.796d35d-reverted.exe.zip

It's the same as 3.2 RC 1, but with commit 796d35d reverted as per https://github.com/godotengine/godot/issues/35242#issuecomment-575884889.
If it works, that would confirm that this commit indeed causes the issue.

[katoneko]

I was just about to jump on the Godot compiling bandwagon when I saw your comment @akien-mga :)
Your build with 796d35d reverted works, no hanging, this must've been the first bad commit.

[Calinou]

@NieznanywInternetach also told me a build of the latest commit with the irradiance size changed to 32 on this line worked. According to reduz, this is because the textureLod() generation takes too much time with the irradiance size set to 64.

[NieznanywInternetach]

@NieznanywInternetach @katoneko Could you try this build?
https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.3.2.rc1.796d35d-reverted.exe.zip

It's the same as 3.2 RC 1, but with commit 796d35d reverted as per #35242 (comment).
If it works, that would confirm that this commit indeed causes the issue.

Yup, it works. The same like the one sent by Calinou

So i suppose the problem's solved and the issue can be closed

[akien-mga]

So i suppose the problem's solved and the issue can be closed

We still need to write a fix :)
Reverting that commit introduces back another bug, so it would only be a last resort solution.

[NieznanywInternetach]

So i suppose the problem's solved and the issue can be closed

We still need to write a fix :)
Reverting that commit introduces back another bug, so it would only be a last resort solution.

Oh, right. I had on mind what i could do to help is done. So good luck!

[akien-mga]

I made test builds of the current master branch (e8dc581) + PR #35302 which should fix this issue. Could you test them to confirm?

• Windows (@NieznanywInternetach @katoneko @jgodfrey): https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.3.2.rc.e8dc581%2bGH-35302.64.exe.zip
• Linux (@Berkru): https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.3.2.rc.e8dc581%2bGH-35302.x86_64.zip

[katoneko]

The build works but it takes longer (~8 seconds) to load a project compared to ~4 secs on 3.1. Not sure whether that's natural or connected to the issue, just letting you know in case it's important.

[NieznanywInternetach]

I made test builds of the current master branch (e8dc581) + PR #35302 which should fix this issue. Could you test them to confirm?

* Windows (@NieznanywInternetach @katoneko @jgodfrey): https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.3.2.rc.e8dc581%2bGH-35302.64.exe.zip

* Linux (@Berkru): https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/godot.3.2.rc.e8dc581%2bGH-35302.x86_64.zip

It works for me as well
And i tested the increase of loading time:

Your build:
for GLES 3 it's 8 (+/- 0.3) seconds no matter of size of the project,
for empty GLES2 it's 6.5 (+/- 0.3) seconds

3.2 beta 6:
for full project in GLES3 it's 9.5 (+/- 0.5) seconds (but i may increased it a bit by my surprice),
empty GLES3: 9 (+/- 0.3) seconds
for empty GLES2 it's 6.5 (+/- 0.3) seconds

3.1.1 (27.04.2019)
full project GLES3 6.2 (+/- 0.3)
empty GLES3 5.5 (+/- 0.3)
empty GLES2 5.9 (+/- 0.3)

[akien-mga]

Thanks! I think the increased start times are expected due to the increased complexity of the shaders to support more rendering features. Some of the new additions could likely be optimized to reduce the overhead a bit, but the increase shown in above comments seems acceptable (as long as performance is then also fine).