Gliden64: Super lag in HLE mode, at least with r600/radeonsi Mesa drivers

[Jj0YzL5nvJ]

From the moment of implementation a625225323c902b614ed9601143df3bc51550fc4, this generates some kind of delay in radeon Mesa driver (r600g).

Xubuntu 16.04.3 LTS
glxinfo | grep OpenGL

OpenGL vendor string: X.Org
OpenGL renderer string: AMD JUNIPER (DRM 2.43.0 / 4.4.0-93-generic, LLVM 6.0.0)
OpenGL core profile version string: 3.3 (Core Profile) Mesa 17.3.0-devel - padoka PPA
OpenGL core profile shading language version string: 3.30
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.0 Mesa 17.3.0-devel - padoka PPA
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.0 Mesa 17.3.0-devel - padoka PPA
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.00
OpenGL ES profile extensions:

The CPU consumption is not higher when compared to previous versions. And compiling with -DCRC_OPT=On changes almost nothing when it comes to FPS. Examples:

Running SM64 until the moment the white star appears:

dddb3ae1f71afa85ef782d5f5edb9661a1b1b5bd
cmake -DMUPENPLUSAPI=On ../../src/
LIBGL_SHOW_FPS=1 MESA_GL_VERSION_OVERRIDE=3.3COMPAT MESA_GLSL_VERSION_OVERRIDE=420

libGL: FPS = 0.5
libGL: FPS = 40.0
libGL: FPS = 40.0
libGL: FPS = 51.0
libGL: FPS = 60.0
libGL: FPS = 4.6
libGL: FPS = 54.0
libGL: FPS = 59.0
libGL: FPS = 59.0
libGL: FPS = 60.0
libGL: FPS = 60.0
libGL: FPS = 60.0
libGL: FPS = 60.0

a625225323c902b614ed9601143df3bc51550fc4
cmake -DMUPENPLUSAPI=On ../../src/
LIBGL_SHOW_FPS=1 MESA_GL_VERSION_OVERRIDE=3.3COMPAT MESA_GLSL_VERSION_OVERRIDE=420

libGL: FPS = 0.5
libGL: FPS = 20.6
libGL: FPS = 15.0
libGL: FPS = 15.2
libGL: FPS = 15.0
libGL: FPS = 15.0
libGL: FPS = 15.2
libGL: FPS = 16.4
libGL: FPS = 16.4
libGL: FPS = 16.6
libGL: FPS = 16.4
libGL: FPS = 16.4
libGL: FPS = 4.1
libGL: FPS = 18.1
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.5
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 9.0
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 9.2
libGL: FPS = 10.7
libGL: FPS = 8.5
libGL: FPS = 8.3
libGL: FPS = 9.0
libGL: FPS = 8.2
libGL: FPS = 8.1
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 9.0
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 9.0
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 9.0
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 9.2
libGL: FPS = 11.0
libGL: FPS = 8.9

bbc7131655a78ae887cee481f0a67674385fc2d2
cmake -DCRC_OPT=On -DMUPENPLUSAPI=On ../../src/
LIBGL_SHOW_FPS=1

libGL: FPS = 0.3
libGL: FPS = 21.8
libGL: FPS = 16.3
libGL: FPS = 16.4
libGL: FPS = 16.5
libGL: FPS = 16.6
libGL: FPS = 16.3
libGL: FPS = 16.6
libGL: FPS = 16.5
libGL: FPS = 16.4
libGL: FPS = 16.4
libGL: FPS = 6.9
libGL: FPS = 18.4
libGL: FPS = 8.3
libGL: FPS = 8.4
libGL: FPS = 8.4
libGL: FPS = 8.3
libGL: FPS = 8.4
libGL: FPS = 8.2
libGL: FPS = 8.4
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.4
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 9.2
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 9.0
libGL: FPS = 10.7
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 9.1
libGL: FPS = 8.2
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 9.1
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 8.3
libGL: FPS = 9.0
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 8.2
libGL: FPS = 9.3
libGL: FPS = 8.2
libGL: FPS = 8.3
libGL: FPS = 9.2
libGL: FPS = 10.6

If anyone knows how to put spoilers, let me know how.

[loganmc10]

Can you post the full output of glxinfo? Maybe put it on pastebin or upload a txt file or something so it's not a huge post

[Jj0YzL5nvJ]

http://sprunge.us/DehV

[Jj0YzL5nvJ]

I've been doing more testing and apparently VBO just makes the problem even more evident, so this problem reside in other place.

Just as note, the use of the GPU is almost nil in lag times and CPU usage does not exceed 33%. Any mesa-utils uses more GPU... (tested with radeontop). But if LIBGL_ALWAYS_SOFTWARE=1 is used, the CPU usage is stupidly high (89% average).

[loganmc10]

Do you have anything non-default set in the config? If so, can you post your mupen64plus.cfg?

[Jj0YzL5nvJ]

http://sprunge.us/cAII

I modify VideoPlugin and RspPlugin constantly, delete everything related to Video-GLideN64 every time I change the version.
In theory I'm always using defaults, the only custom configuration that I remember doing is in Input-SDL-Control1.
I have been finding multiple bugs in mupen64plus itself, so I'm going to have to do tests with old versions too.

Edit:
I even tested with modesetting driver and r600g with EXA acceleration (DRI2), all the same, nothing changed.

[Jj0YzL5nvJ]

I found the origin! 313741d8276f0e018ba0c137892d67794468cc28

In my tests, that comment causes much lag in:
Harvest Moon 64
Perfect Dark
Paper Mario

And partial lag (only in certain scenes) in:
Space Station Silicon Valley
Super Smash Bros.
Bomberman 64
Donkey Kong 64
Kirby 64: The Crystal Shards

Further comments increase the lag in other games, but a625225323c902b614ed9601143df3bc51550fc4 generalizes the lag in all games, "it was the straw that broke the camel's back". At least in my hardware, I deduce.

[loganmc10]

It seems like your machine has an issue with gl_arb_buffer_storage

Using the latest master, can you try forcing this variable to false:

https://github.com/gonetz/GLideN64/blob/master/src/Graphics/OpenGLContext/opengl_GLInfo.cpp#L58-L59

Just get rid of that whole statement and replace it with bufferStorage = false

[Jj0YzL5nvJ]

Nope, not is it. With that change the lag is slightly worse (between 0.1 to 1.2 FPS less in SM64).

[Jj0YzL5nvJ]

I think I found something interesting. See this first: https://imgur.com/a/rEMEu

In some parts of Carrington Institute, the lag disappears completely (No. 1, No. 6 and No. 7). Usually, places with poor lighting or being very close to any wall, lit up or not.
But are some exceptions (No. 2, No. 4 and No. 5), the FPS drop relatively little when compared to most of the other lights (No. 3).

In the case of image No. 7 and No. 8, both are exactly the same spot. The only difference is that No. 8 was taken after activating Hi-Res.
With Hi-Res enabled, the FPS are maintained very similar to No. 8 in places where they were previously perfect.

Save state: https://0x0.st/RpV.zip
Current cfg: http://sprunge.us/Pdia
gliden64.log: http://sprunge.us/MNCH

P.S: The gliden64.log only is generated when GLideN64 is compiled with Clang...

[Jj0YzL5nvJ]

I've been "playing a lot", updating and recompiling many dependencies in my PC...
After doing test after test, in different games and messing with the configuration file. I found some mitigators for the symptoms, unfortunately none such universal solution.

The first one is use cxd4-sse2 in "HLE mode" instead of the HLE plugin.
The second one is use DisableFBInfo = False or/and EnableCopyColorToRDRAM = 0. #1559
The last one is useing LIBGL_ALWAYS_SOFTWARE=1. But in many games this will become an backfire.

Examples: https://imgur.com/a/HAqJZ

Edit:
Added more images differentiating LLE and HLE in cxd4-sse2.
It's like if GLideN64 forces an LLE mode into "mupen64plus-rsp-hle.so".

[Jj0YzL5nvJ]

I've been testing angrylion-plus... and I have noticed that in my previous post I confused HLE and LLE configuration in cxd4-sse2 completely (thanks to this). So the images description are inverted... cxd4-sse2 (HLE) in reality is cxd4-sse2 (LLE) and vice versa.

My lag problem in HLE persists ... only now I know that cxd4 is also affected, Even angrylion-plus works for me faster than GLideN64 in HLE mode (DisplayListToGraphicsPlugin = True).
I gonna stick to bc00985f33c8b1e2e63cebabe53d01f9cf3708a6, the last commit that works well for me u.u

[fzurita]

So the one that made things slow for you is this?

313741d8276f0e018ba0c137892d67794468cc28

Can you try disabling Copy color buffer to RDRAM and check if performance improves? It sounds like AMD VESA drivers don't like buffer storage either.

Edit: While you are at it, can you test this branch?
https://github.com/fzurita/GLideN64/tree/threaded_GLideN64

I'm curious how threaded GLideN64 performs with AMD hardware.

[loganmc10]

Can you try disabling Copy color buffer to RDRAM and check if performance improves? It sounds like AMD VESA drivers don't like buffer storage either.

I had him try disabling buffer storage (https://github.com/gonetz/GLideN64/issues/1561#issuecomment-326972673) it didn't seem to make a difference

[fzurita]

Let's double check by turning off "Copy color buffer to RDRAM".

[Jj0YzL5nvJ]

The second one is use DisableFBInfo = False or/and EnableCopyColorToRDRAM = 0.

Disabling those two things is the only thing that helped me a bit. But I don't remember trying to disable that with the modification suggested by loganmc10 (https://github.com/gonetz/GLideN64/issues/1561#issuecomment-326972673), I'll try again.

Edit: While you are at it, can you test this branch?
https://github.com/fzurita/GLideN64/tree/threaded_GLideN64

Okay, I don't have much to do anyway. I'll try it when I'm at home.

[Jj0YzL5nvJ]

No. I don't see much difference between disabling bufferStorage or by using EnableCopyColorToRDRAM = 0 in the cfg file. Or by using the two, at most there will be an average difference of 1 FPS, 2 VI/S and 3 % in the counters (in Banjo-Kazooie). At least with HLE.

@fzurita, the same history is with thethreaded_GLideN64 branch, I can test in LLE tomorrow.

[fzurita]

So this is very odd. The commit that made things slow for you is https://github.com/gonetz/GLideN64/commit/313741d8276f0e018ba0c137892d67794468cc28 but that code is only invoked when EnableCopyColorToRDRAM is not zero.

I'm not sure what is going on.

Could it be that https://github.com/gonetz/GLideN64/blob/master/ini/GLideN64.custom.ini is overwriting your setting?

[Jj0YzL5nvJ]

I've been testing in my free time (which is very little) and I did some discoveries. Unfortunately, nothing directly related to my problem... apparently.

Could it be that https://github.com/gonetz/GLideN64/blob/master/ini/GLideN64.custom.ini is overwriting your setting?

I really doubt it. I don't use any GLideN64*.ini files. Or more precisely, I don't know where to put them to make them work with mupen64plus. And I can see the effects caused after edit my configuration file.

In respect to 313741d8276f0e018ba0c137892d67794468cc28 see this first https://github.com/gonetz/GLideN64/issues/1561#issuecomment-326910677
In addition to the ROM's already mentioned, I also tested this others without detect lag problems: Banjo-Kazooie, GoldenEye 007, The Legend of Zelda (OoT and MM), Mario Kart 64, Resident Evil 2, Super Mario 64, the both Castlevania's and Conker's Bad Fur Day
Something worth to mention is that I only test the intros without interaction at that time.
Recently it was tried again and in the case of Perfect Dark, the lag only appears in "the first boot". The lag only appear in the logos animation (RARE, Nintendo, N64 and Perfect Dark), in the second round the same logos move smoothly, the gameplay too. Interestingly, if I left START pressed in the game's boot time, when leaving the Controller Pak menu, the lag never occurs.

I know very little of programming (Turbo C, Turbo Pascal, Visual FoxPro, Delphi, etc.) and I'm more used to writing maintenance scripts (Batch, VBScript, AHK, etc.) for Windows.
This seems very AMD specific. So, I don't know if this is worth to refer to this, sorry if not.

I split the VRAM into 8×8 blocks. All hazards and dependencies are tracked at this level. I chose 8×8 simply because it fits neatly into 64 threads on blits (wavefront size on AMD), and the smallest texture window is 8 pixels large, nice little coincidence 🙂

https://www.libretro.com/index.php/introducing-vulkan-psx-renderer-for-beetlemednafen-psx/

In my last test I seen lag and freezing times in bc00985f33c8b1e2e63cebabe53d01f9cf3708a6 when GLideN64 try to fill more of 542M in VRAM. After that the VRAM usage reduces a few MB and start to fill again.
In 313741d8276f0e018ba0c137892d67794468cc28 this never occur, but is very difficult to fill more of 120M of VRAM, is like the code is doing more cleaning VRAM than using it.
I had to destroy many things in Perfect Dark to achieve to fill the VRAM and surpass the erasing VRAM code. But again this never use more of 542M or the 60% of VRAM (max 1024M).

Edit:
Forget to mention that I found specific places in Banjo-Kazooie which the lag is more intense with LLE than in HLE (with https://github.com/gonetz/GLideN64/commit/deaf61299f12168b775d7e2d448b9eca149c0e7e and 3cf7377 threaded_GLideN64). So this not HLE specific, but is much more notorious in it.

[Jj0YzL5nvJ]

@fzurita, I get this using your 'further_reduce_shader_logic' branch: https://0x0.st/sX2R.log
In my case https://github.com/gonetz/GLideN64/issues/1665#issuecomment-351760214 doesn’t make any difference.

[Jj0YzL5nvJ]

I have found the true origins (I believe). 313741d8276f0e018ba0c137892d67794468cc28 and 3aa365d24a6bd5f49e1205ea6338cddd7092e6f4

As I mentioned before, 313741d8276f0e018ba0c137892d67794468cc28 cause lag (and glitches) to me on the boot time and the first scenes of Perfect Dark and others games https://github.com/gonetz/GLideN64/issues/1561#issuecomment-326910677. But not in the gameplay.

On the other hand, 3aa365d24a6bd5f49e1205ea6338cddd7092e6f4 fixes the glitches caused by 313741d8276f0e018ba0c137892d67794468cc28, but the lag become permanent in Perfect Dark, in HLE with Hi-Res enabled. With Hi-Res disabled, the FPS only become unstable.
So the problem was very difficult to find... because Hi-Res enabled does not offer any benefit in emulation.

And I quote myself:

Further comments increase the lag in other games, but a625225323c902b614ed9601143df3bc51550fc4 generalizes the lag in all games, "it was the straw that broke the camel's back". At least in my hardware, I deduce.

[fzurita]

Do you have color buffer to RDRAM enabled? If you don't have it enabled, the first commit shouldn't make a difference.

[Jj0YzL5nvJ]

I tested with the default value EnableCopyColorToRDRAM = 2.
Yeah, with EnableCopyColorToRDRAM = 0, neither 313741d8276f0e018ba0c137892d67794468cc28 nor 3aa365d24a6bd5f49e1205ea6338cddd7092e6f4 manifests lag.

Now I have to find the comment that makes the lag manifest with EnableCopyColorToRDRAM = 0, I suppose. u.u

[fzurita]

Well before those commits, EnableCopyColorToRDRAM was always disabled for GLES 2.0 devices.

Edit: whoops, you have a AMD device. Too many issues and got things confused. It seems like buffer storage causes slow downs with AMD. At least with that specific driver.

[Jj0YzL5nvJ]

Ouroboros! Back to the beginning T.T

Starting from a625225323c902b614ed9601143df3bc51550fc4, the lag persist even with EnableCopyColorToRDRAM = 0. And it also affects LLE, but with less severity than in HLE.

Hmm... I'm on vacation, I will try to create a dual-boot installation with Windows and another distribution for testing.
...Just thinking about it, I lose all desire to do it.

[fzurita]

Same thing. That commit added more buffer storage usage. We enable buffer storage automatically when the driver reports that it supports it. We could make a special case against that specific driver to disable buffer storage utilization.

[Jj0YzL5nvJ]

Lets summarize the few workarounds that can be made to get GLideN64 working on r600/radeonsi Mesa drivers. From better to worse...

• Stick to dddb3ae1f71afa85ef782d5f5edb9661a1b1b5bd, the last "fixable version" if you launch the emulator with: MESA_GL_VERSION_OVERRIDE=3.3COMPAT MESA_EXTENSION_OVERRIDE="-GL_ARB_buffer_storage"
• Stick to bc00985f33c8b1e2e63cebabe53d01f9cf3708a6 and launch using: MESA_GL_VERSION_OVERRIDE=3.3COMPAT
• With a good CPU and LIBGL_ALWAYS_SOFTWARE=1, you can play dementia (that nothing happens)...
• Use GLideN64 in LLE mode (glitch mode)
• Use EnableCopyColorToRDRAM = 0 to gain a few FPS (hell no ¬¬)

That's all.

This is going to take some time ...I hope.

[fzurita]

Can you try one more thing? In here: https://github.com/gonetz/GLideN64/blob/master/src/Graphics/OpenGLContext/opengl_GLInfo.cpp#L61-L62

Replace:

    bufferStorage = (!isGLESX && (numericVersion >= 44)) || Utils::isExtensionSupported(*this, "GL_ARB_buffer_storage") ||
            Utils::isExtensionSupported(*this, "GL_EXT_buffer_storage");

with

    bufferStorage = false;

Also, this: https://github.com/gonetz/GLideN64/blob/master/src/Graphics/OpenGLContext/opengl_ContextImpl.cpp#L62-L67

Replace those lines with this:

m_graphicsDrawer.reset(new UnbufferedDrawer(m_glInfo, m_cachedFunctions->getCachedVertexAttribArray()));

[Jj0YzL5nvJ]

@fzurita
Eureka! That do the trick!
I need to use MESA_GL_VERSION_OVERRIDE=3.3COMPAT, but that doesn't matter.
Now we have a real workaround.

Thanks so much, really ^^

[BPaden]

Note for on a slightly newer GCN card, this helps for me but is only a partial fix.

For the Ocarina of Time intro, when link first appears riding on Epona, without the patch performance drops from ~50fps to about 15. It's pretty consistently within 1 FPS of 15 in this case.

With the patch, performance drops from ~60 FPS to somewhere between 20-40. It's less consistent here, and there are places (generally when there are no characters on the screen) where it runs at full speed, but I'd say average with characters on the screen is about 30 FPS. That's probably closer to a real n64 than 60 FPS, but the problem is the constant stuttering. :P

Also with the patch, I have to use 3.3COMPAT or I get a black screen. I was going to make an apitrace, but it seems that tool doesn't work with compatibility profiles, unless I'm missing something.

Probably there are multiple issues with GlideN64 and Mesa on older AMD hardware, including this one.

[fzurita]

It seems that mesa driver has poor support for some features. Even after disabling buffer storage and VBOs, there is still something it doesn't like.

[BPaden]

GLideN64 is definitely hitting a slow path in mesa that doesn't exist in the Windows drivers, where the plugin works fine. At the same time, I think GLideN64 must be doing something unusual (not unexpected for an emulator) because I'm not seeing these symptoms in any other GL application.

[loganmc10]

The unusual thing GLideN64 does with VBO's is called Buffer Object Streaming (https://www.khronos.org/opengl/wiki/Buffer_Object_Streaming)

It works well on my Intel Mesa laptop, but I don't have any AMD devices to test.

Mesa only officially supports the core profile, and the core profile requires the use of VBO's, which is why this is needed

[Jj0YzL5nvJ]

I've been quick testing some games. Until now I spotted three lag points, one in Bomberman 64 and two in GoldenEye 007.
I'm gonna bisect and apply "the patch" to discern the other "bad commit". That when finish my work, of course.

[Lithium64]

Same issue here, I'm running latest m64p linux build (ubuntu build don't works correctly here).

My specifications/settings are

i3 4150 3.50ghz
12 GB RAM
Radeon R7 260X 2GB VRAM
Ubuntu 17.10
Kernel 4.14.15
Mesa 17.3.2

[Jj0YzL5nvJ]

I made a small "benchmark" using GALLIUM_HUD, testing some branches and pull requests. All my tests include the changes to disable buffer storage.
@fzurita, @loganmc10 and for the few people who share my problem. I hope this at least can be useful for you.

Benchmark 01
Savestate: ge007_1.zip

Benchmark 02
Savestate: ge007_0.zip

Benchmark 03

Benchmark 04
Savestate: bomberman64.zip

P.S: For adapt https://github.com/gonetz/GLideN64/issues/1561#issuecomment-360246650 to threaded_GLideN64.

Replace this: https://github.com/fzurita/GLideN64/blob/threaded_GLideN64/src/Graphics/OpenGLContext/opengl_ContextImpl.cpp#L65-L71

For this:

        if (config.video.threadedVideo)
            m_graphicsDrawer.reset(new UnbufferedDrawerThreadSafe(m_glInfo, m_cachedFunctions->getCachedVertexAttribArray()));
        else
            m_graphicsDrawer.reset(new UnbufferedDrawer(m_glInfo, m_cachedFunctions->getCachedVertexAttribArray()));

[loganmc10]

I have a few ideas for this, I'll create some test commits tomorrow for you to try out. Just to confirm, you say that LLE mode is often faster than HLE is that correct?

[Jj0YzL5nvJ]

Just to confirm, you say that LLE mode is often faster than HLE is that correct?

Only with buffer storage enabled. If buffer storage is forcefully disabled, HLE is faster.

Edit:
This is how HLE mode performs with 'master' on my hardware

All previous benchmarks with the exception of 03, are laggy places that even persist with buffer storage disabled. But with buffer storage enabled, such places are even most affected by lag. How ironic, don't make any sense.

[loganmc10]

Maybe we'll have to chat on some other platform, I thought you previously said disabling buffer storage didn't make a difference?

[fzurita]

To make any difference, he had to disable both buffer storage and VBOs.

[loganmc10]

@Jj0YzL5nvJ This is going to seem like a silly shot in the dark, but can you try to apply this commit:

https://github.com/loganmc10/GLideN64/commit/881f58df4a5fdf539da5b8449d81c999e86ec0a4

Leave everything else like it is in the current master, and I would set EnableCopyColorToRDRAM = 0 for now in the config, just so we can work on one issue at a time.

From what I read, older ATI cards required the vertex data to be aligned to 32/64 bytes for best performance. Ours is currently at 44 bytes, so the padding brings it up to 64. I'm really not sure if it'll make a big difference, but it's worth a shot

[loganmc10]

Here is another test: https://github.com/loganmc10/GLideN64/commit/af5d5b8a1878d2e87a6a78dee25a0fd86b745266

Same rules as before, please leave everything else the same as master, and set EnableCopyColorToRDRAM = 0. This is testing a different way to handle the buffer storage

[loganmc10]

@fzurita I'm curious if https://github.com/loganmc10/GLideN64/commit/881f58df4a5fdf539da5b8449d81c999e86ec0a4 makes any difference on Adreno devices with GLES.3.2 (they'll support VBO + buffer storage). Some stuff I read says the vertex data alignment matters on some mobile chipsets but it's hard to find solid information

[fzurita]

I can do some quick testing on a slower Adreno device.

[BPaden]

Neither patch appears to effect my ~1.0 GCN card.

I mentioned earlier that I was interested in running this in a profiler. Caveat: I have no idea what I'm doing.

Using the apitrace protocol, it looks like gl calls are using no more than ~2 ms in the gpu.

In the CPU, however, the graph is skewed because some calls to glTexSubImage2D is occasionally take around 100 ms or more (!!). However, this happening enough for it to be the problem on a frame-by-frame basis. I am seeing calls to glDrawElementsBaseVertex taking ~15ms. When I zoom in the graph looks like this.

The pattern here looks like several calls to glDrawElementsBaseVertex taking around 10-15ms, followed by a call to glFlushMappedBufferRange also taking about 10ms.

Is there anything anyone would be interested in me looking at?

[Jj0YzL5nvJ]

@loganmc10, my tests without disabling buffer storage and VBOs, take them all with a grain of salt.
GL_ARB_buffer_storage is certainly broken in my drivers.

Test with EnableCopyColorToRDRAM = 0

Personally I did not notice any significant changes, much less by disabling buffer storage and VBOs.
But comparing results with the previous test, the differences are very significant, especially in the CPU activity, GPU activity and buffer wait time.

@BPaden, try to run using MESA_GL_VERSION_OVERRIDE=3.3COMPAT MESA_GLSL_VERSION_OVERRIDE=410 MESA_EXTENSION_OVERRIDE="-GL_ARB_buffer_storage"

Can you put the results of the following commands?

glxinfo | grep OpenGL
cat /var/log/Xorg.0.log | grep -i enabled
cat /var/log/Xorg.0.log | grep -i load
cat /var/log/Xorg.0.log | grep -i swap

[loganmc10]

@BPaden that trace is very helpful. My next test will be replacing glDrawElementsBaseVertex with glDrawArrays. I assumed this might be an issue since @Jj0YzL5nvJ mentioned that LLE works better, I believe LLE always uses glDrawArrays (I could be remembering wrong though).

Disabling VBO's is good for testing, but it can't be a long-term solution. Core OpenGL requires the use of VBO's (that's why you need the environment variable to get it to work). In a future version of Mesa, they could remove support altogether for non-VBO rendering if they wanted, so we can't count on that.

@BPaden the long glTexSubImage2D is unfortunate but not unexpected. That is when the emulator is uploading texture data to the GPU. In a normal game you would do that at the beginning, not during rendering, but the emulator doesn't know about the texture data until right before it's needed, so we have to upload it like that

[loganmc10]

Ok @BPaden @Jj0YzL5nvJ can you try this commit:

https://github.com/loganmc10/GLideN64/commit/9bcfa67d9550c7f1cd4ba72f657facd66a4d27e4

I tested this on my Nvidia laptop and saw no difference in performance, but it may make a difference for you. I'm also curious if this makes any difference on Adreno devices with buffer storage @fzurita

[BPaden]

libGL: FPS = 60.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 56.9
libGL: FPS = 60.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 60.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 60.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 60.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 60.9
libGL: FPS = 59.9
libGL: FPS = 59.9
libGL: FPS = 59.9

😁

[fzurita]

Sure, I'll try that. It will actually probably help performance with slower Android devices. I do know that VBOs and EBOs are slower with them. I remember in the past they were about 10% slower.

[loganmc10]

Yeah well it definitely looks like we found the bottleneck in the Mesa driver. I'm going to hop on their IRC channel and ask about this.

It's a little counterintuitive, the whole point of the elements (glDrawElements), is that you can reduce the amount of bandwidth used in uploading the vertex data. But maybe when used in conjunction with VBO streaming the benefits are negated, I'll be curious to hear if there is any difference on a mobile device.

[Jj0YzL5nvJ]

This time the improvement is huge:
Test with new arrays

This time I don't consider necessary a test by disabling buffer storage and VBOs. (The true is that I'm short of time =P)

[fzurita]

Copy color to RDRAM is still slow even with that change it seems.

[loganmc10]

Yeah I suspect that disabling buffer storage for the copies might fix that, we can test that in a bit.

Sorry to do this to you, but @Jj0YzL5nvJ and @BPaden can you do one more test for me?

https://github.com/loganmc10/GLideN64/commit/65275370b8cb0c29c944d74182f58978594fd59b

I've been looking at Dolphin's code, and it looks like they don't use buffer storage or any buffers for the element arrays, I suspect that may be the actual problem.

[loganmc10]

Also, can one or both of you add:

printf("%s\n", strRenderer);

Here:

https://github.com/gonetz/GLideN64/blob/master/src/Graphics/OpenGLContext/opengl_GLInfo.cpp#L29

And post the output.

We need to see how your card reports it's name so we can disable the buffer storage extension for Copy color to RDRAM

[loganmc10]

Another update: someone from Mesa responded and indicated that it is our use of GL_UNSIGNED_BYTE that is the issue, please also test this:

https://github.com/loganmc10/GLideN64/commit/fa0ab9d0aeb14f8760623aa0f6d3833bb824ad04

We are looking for whatever solution offers the best FPS really (although the use of glDrawArrays is probably out).

So I'm curious between these 2 solutions:

https://github.com/loganmc10/GLideN64/commit/65275370b8cb0c29c944d74182f58978594fd59b

https://github.com/loganmc10/GLideN64/commit/fa0ab9d0aeb14f8760623aa0f6d3833bb824ad04

Which is faster

[loganmc10]

@fzurita I wonder if that is the reason the Copy Color to RDRAM is slow as well, I believe glReadPixels is using GL_UNSIGNED_BYTE right?

[BPaden]

can you do one more test for me?

No problem. :)

printf("%s\n", strRenderer);

AMD Radeon HD 8500 series (OLAND / DRM 3.23.0 / 4.15.6-1-ARCH, LLVM 5.0.1)

https://github.com/loganmc10/GLideN64/commit/65275370b8cb0c29c944d74182f58978594fd59b

this is about 5-10 FPS faster than master, but still slow.

https://github.com/loganmc10/GLideN64/commit/fa0ab9d0aeb14f8760623aa0f6d3833bb824ad04

This is running at full speed for me.

[BPaden]

Oh also I have EnableCopyColorToRDRAM=2 for this test.

[fzurita]

@loganmc10 That could be the reason for your specific device.

All the devices I have in my possession are pretty fast when using glReadPixels in a async way, except for Adreno 530, which I had to use floats for the pixel buffer. GLES 2.0 devices can't do a glReadPixels in a async way due to lack PBOs, but a lot can use the EGL image extension which is REALLY fast.

Anyways, devices that have fast glReadPixels in my possession have PowerVR, Adreno, and Mali GPUs. What GPU that you have has slow glReadPixels?

[loganmc10]

Anyways, devices that have fast glReadPixels in my possession have PowerVR, Adreno, and Mali GPUs. What GPU that you have has slow glReadPixels?

Sorry I meant why @Jj0YzL5nvJ might be experiencing slow glReadPixels, you've looked into the format stuff a lot so I thought you might have some insight on that. Maybe I'll try using the FLOAT type for the Radeon cards like the Adreno 530 has to see if it works faster.

[LegendOfDragoon]

Any idea if Intel IGP's could benefit from any of these changes?

[loganmc10]

Any idea if Intel IGP's could benefit from any of these changes?

According to the person on the Mesa issue tracker (https://bugs.freedesktop.org/show_bug.cgi?id=105256#c4):

This should affect all amd/ati hw up to Sea Islands....
At a very quick glance at the driver code, it looks like newer nvidia chips can handle it (everything from g80). nv30/nv40 cannot, but there it looks like nv30 will emulate index buffers anyway, so the only chips which this might make a difference is nv40 family.
Looks like all intel chips can handle it fine.

But this is talking about the Mesa (Linux) drivers. This bug didn't affect the AMD Windows drivers. Basically there is no way to know what it might affect. The PR I submitted (https://github.com/gonetz/GLideN64/pull/1735) isn't GPU specific, so this fix will be applied to all GPU's, since it doesn't harm anything. This was a pretty crippling bug on the AMD/Linux driver, so I suspect if Intel was affected, you would have noticed it already.

[Jj0YzL5nvJ]

printf("%s\n", strRenderer);

AMD JUNIPER (DRM 2.50.0 / 4.13.0-36-generic, LLVM 5.0.1)

Crippled GL_ARB_buffer_storage? (EnableCopyColorToRDRAM = 2)

Is any nouveau user out there?...

[Jj0YzL5nvJ]

Benchmark 06746ac vs 06746ac+1735.patch (SHA256: ba4f4db46dd861a08cd8dec222e20714c391c3d682739d9bb04326c60e358e79)

EnableCopyColorToRDRAM = 2

EnableCopyColorToRDRAM = 0

P.S: Just in case this isn't clear to someone Revision P06746ac is 06746ac+1735.patch

[Jj0YzL5nvJ]

Adding the GL_CLIENT_STORAGE_BIT flag at here fixes the issue completely.

Info: https://bugs.freedesktop.org/show_bug.cgi?id=102204#c7

[fzurita]

You should make a pull request, but based on the wine link, it looks like we only want to use that flag on Mesa.

[Lithium64]

@loganmc10 @gonetz