This appears not to be so bad at higher resolutions, but I'm testing at 720p on the lowest-of-the-low settings, including no AA. Whatever type of aliasing, or softening, is present in the display scaling on current Proton (3.7-8 and Beta) goes completely away in 3.16-1 Beta, causing the fonts, etc. to get extremely jagged. There is a definite, visible difference.
Also mentioned at https://github.com/ValveSoftware/Proton/issues/773#issuecomment-429589983.
Just cross-tested game on a Dell (A10/RX 560) Windows 10 stock box with similar-sized 1920x1080 monitor. Same "lowest" settings at 720p in DX11 confirm that Proton 3.7 is the correct 'look'.
I can confirm the issue. I am having the same issue with the games that use DXVK (Witcher 3, The Forest, etc). DirectX 9 games seems to be fine though.
I also confirm this happening on Dishonored demo. Looks like for 3.16-1 an nearest type of filtering is used, while for 3.7 a linear one, very visible on text.
3.7 on the left, 3.16 on the right:

We do request linear scaling when stretching the framebuffer up to the display, see https://github.com/ValveSoftware/wine/blob/proton_3.16/dlls/winevulkan/vulkan.c#L1911 . Nothing changed here between 3.7 and 3.16. @doitsujin, do you have any ideas why this might be happening?
Ah, wait, if we're building DXVK as a native linux .so, it's probably bypassing our fullscreen hack code entirely. I bet that's our problem.
This is a DXVK bug. For some reason it is now responsible for doing the scaling by itself, which is okay, but it doesn't handle the case where the window size differs from the Vulkan swap chain size correctly (previously they were always the same). Should be easy to fix though.
On the same Proton 3.16-1, I replaced the native dxvk libraries with the latest (dll) files from dxvk 0.90 and the scaling issue doesn't happen.
@dubigrasu Thanks for testing, that fits with the explanation. We're going to have to back out the native DXVK build anyway due to issues with the Steam Runtime (see #1730) so this should be fixed alongside that.
I'm sorry for the probably off-topic question, but can't be the steam-runtime just updated to avoid similar conflicts?
Yes, but that's a much larger task.
Anyway, doitsujin/dxvk@0d1fa17da289790a5cbe7d8af89f1565964957fc should fix this.
Hello @NeptNutz, @dubigrasu, please retest with Proton 3.16-2.
Looks good! Back to normal. [Tested on Nvidia 396.54.09 and 410.66.]
Yes, all good here.
Thanks for retesting, closing.
Most helpful comment
On the same Proton 3.16-1, I replaced the native dxvk libraries with the latest (dll) files from dxvk 0.90 and the scaling issue doesn't happen.