Retroarch: Add anti-aliasing to the XMB Menu ribbon pipeline

[hyarsan]

This should be toggleable. I don't think the template is needed.

[RobLoach]

Use the simplified ribbon?

[hyarsan]

The simplified Ribbon doesn't have anti-aliasing? I am using Windows 10 64 bit.

[bparker06]

Both ribbons are jaggy-rific for me on Win8.1

[hyarsan]

I managed to add some anti-aliasing using the NVIDIA Control Panel.

[hizzlekizzle]

Just to be clear, I don't think there's any way to antialias the ribbons due to the way they're implemented. That is, most shaders use the vertex stage to create the canvas and then draw stuff to that canvas in the fragment stage, while the ribbons are created entirely in the vertex stage and the fragment just colors them. In other words, the ribbons are the canvas, if that makes sense. We can't add any particle effects, etc. around the ribbons for this same reason.

Post-proc anti-aliasing, like FXAA applied through the driver, is the only way to anti-alias it, though we could potentially add something like that ourselves if the menu supported multipass pipelines.

[bparker06]

It should be possible with MSAA. As a quick test I tried with the old school polygon smoothing:

diff --git a/menu/drivers_display/menu_display_gl.c b/menu/drivers_display/menu_display_gl.c
index d1d57b707..104556f3c 100644
--- a/menu/drivers_display/menu_display_gl.c
+++ b/menu/drivers_display/menu_display_gl.c
@@ -172,6 +172,7 @@ static void menu_display_gl_draw_pipeline(void *data)
       case VIDEO_SHADER_MENU:
       case VIDEO_SHADER_MENU_2:
          glBlendFunc(GL_ONE, GL_ONE);
+         glEnable(GL_POLYGON_SMOOTH);
          break;
       default:
          glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

And while there is some vertical line artifacts there, you can tell that it is anti-aliased nicely. Implementing proper multisampling where supported should solve the issue.

[hizzlekizzle]

Ah, yeah, good idea. I was only thinking from within the shaders themselves :)

[bparker06]

Here's a quick way to test 4xMSAA when running under X on Linux with OpenGL:

diff --git a/gfx/drivers_context/x_ctx.c b/gfx/drivers_context/x_ctx.c
index a5276d2..5ca3633 100644
--- a/gfx/drivers_context/x_ctx.c
+++ b/gfx/drivers_context/x_ctx.c
@@ -479,8 +479,8 @@ static void *gfx_ctx_x_init(video_frame_info_t *video_info, void *data)
       GLX_ALPHA_SIZE       , 8,
       GLX_DEPTH_SIZE       , 0,
       GLX_STENCIL_SIZE     , 0,
-      GLX_SAMPLE_BUFFERS   , 0,
-      GLX_SAMPLES          , 0,
+      GLX_SAMPLE_BUFFERS   , 1,
+      GLX_SAMPLES          , 4,
       None
    };
    GLXFBConfig *fbcs       = NULL;

This gets rid of the jaggies in the fancy ribbon for me (2x wasn't quite enough). The only issue is that this enables MSAA for the default framebuffer and hence runs all the time, including during the rendering of the core video, which may have a performance impact on less capable hardware. This is currently unavoidable (at least with the OpenGL driver) due to the fact that the menu/background is not drawn into its own FBO, but directly to the back buffer.