Dxvk: Triage documentation request

[ryao]

Could we get documentation on how to properly triage issues? In particular, I am interested in triaging VR titles like Waltz of the Wizard (missing objects and a misrendered skull) and The Lab (massive pipeline stalls). I recall trying to run one title without DXVK to get a log, but got a crash. My HMD broke while I was trying to figure it out and don’t have much data saved from my attempts. I thought the downtime until it is fixed would be a good time to ask for guidance.

Also, I have yet to get a winelib built DXVK working with Proton so that I can use Linux profiling tools. It crashes when trying to load the OpenVR DXVK dll. Using a Mingw tool chain built DLL works. Information on how to use the winelib version would be great. I likely could figure it out eventually, but I feel like this is something that could be documented easily by someone who already knows how to make it work.

[doitsujin]

I don't really get the point, what exactly is it that you want?

[ryao]

@doitsujin I would like a step by step instructions showing each of the following on one real world instance of a problem (it can be historical):

1. How to determine the cause of a pipeline stall.

2. How to determine the cause of a misrendering.

3. How to determine the cause of a crash.

4. How to install a winelib built DXVK into proton so that ftrace/perf/eBPF can be used on it.

I understand that the examples that I am requesting will not be cookie cutter solutions that can be applied to arbitrary problems, but I want to study how you triage issues / use debug tools. That way, I can figure out how to do triage asynchronously. Then I could ping the core graphics developers only after I have something that they should see. I imagine that other developers from other areas would find such examples to be similarly useful.

Does that make sense?

[Guy1524]

@ryao https://www.merriam-webster.com/dictionary/triage

Also, assuming you want to debug the problems you listed, and considering you have NVIDIA, those are very general issues and you should report the specific issue you are having with the appropriate information as laid out in the issue template that you ignored.

[ryao]

@Guy1524 I consider a description of what is wrong inside the software to be necessary for triaging bug reports. I like to file bug reports with such information. When applied to software, the term also includes bug report quality:

https://www.mediawiki.org/wiki/Bug_management/Triage

There are things that a reporter can do to localize and describe the cause of a problem. Then they can hand it off to someone familiar with the codebase so that the issue can be rapidly fixed. I want to learn how to do that. It is a little difficult to pick that up when everything is literally new to me.

To give an example, I consider the act of determining what stack trace allocated the leaked objects in #632 to be part of triage. With #632, I was able to lean on my existing kernel development knowledge, but without the winelib builds or any idea of where to look, I cannot do that in general.

Another example would be #514 where an excellent report was made showing what was wrong.

Anyway, the examples that I requested should be extremely useful for learning how to file good bug reports. The idea being that I don’t want to need much back and forth on issues. I want to file an issue to the relevant person with all of the information necessary to understand the issue and figure out how to fix it.

[Guy1524]

Ah, my bad, TIL. The other part of my comment still stands though. There is an issue template you deleted that you should follow when reporting a bug.

[ryao]

@Guy1524 I would like to be able to go deeper than that. Basically, I want to be able to serve the issue to the developers on a silver platter for them, provided that I do not fix it myself and instead serve them a patch. I am in the credits for Battle for Wesnoth because I had a habit of filing bug reports with precise descriptions of issues and patches fixing them when I was in college.

The interaction with the Battle for Wesnoth developers in IRC back then was priceless as I asked how to file bug reports, they told me to use bugzilla and I asked them if they wanted me to attach the patch fixing the issue along with the report. I was rather inexperienced back then, although my patches were taken as is without modification after some brief clarifications.

I doubt that I will be able to write patches for Proton/DXVK issues in any reasonable span of time that I could accommodate in my spare time due to my inexperience with graphics, but I would at least like to be able to triage the issues so that I can file reports that are more likely to result in quick resolutions.

To use Battle for Wesnoth as an example, there was graphical corruption occurring on certain systems. I identified the cause as a reverse memcpy and wrote a patch that addressed it by dynamically identifying the direction of the memcpy and switching to memmove() when the implementation ran in reverse. If I did not have time to write that patch, I would have at least filed a bug saying that the graphical corruption was caused by a combination of Intel’s implementation of a reverse memcpy in glibc 2.13 for certain processors and the improper use of memcpy() on overlapping memory address ranges in the code. The patch was written to keep abusing memcpy() where it worked because it was faster in those instances and I did not want to risk taking performance away from older systems. At the time, I had yet to learn how to do proper profiling to quantify the hit.

Anyway, I might be an oddball in terms of how much attention to detail that I want to include in issue reports, but I doubt that I am the only one. My inability to properly triage issues has kept me from filing several issues with both Proton and DXVK.

[ryao]

@Guy1524 Also, the issue template‘s request to do apitrace is not very useful to end users using proton. Doing it requires:

1. Setting PROTON_DUMP_DEBUG_COMMANDS=1 PROTON_USE_WINED3D11=1 %command% as their launch options.
2. Running the program to get steam to generate the scripts.
3. Running /tmp/proton_$USER/run /path/to/apitrace.exe /path/to/game/binary
4. Finding the game binary location. It is in /tmp/proton_$USER/run.
5. Finding where to get the apitrace binary. There are many clicks while they could be told to go to
   https://people.freedesktop.org/~jrfonseca/apitrace/
6. Guessing what to do if the game crashes when attempting to run it without DXVK, which actually happened to me (with The Lab if I recall correctly).

I just learned how to do these things over the past day or two. Then my HMD had to break.

I am unable to do that until my HMD is fixed. I figured that I could ask for examples of how to go deeper so that I have some idea of what to do when my HMD is fixed. I guess that you could describe my issue as a request for documentation on how to debug issues. There is a certain amount of debugging that I would like to do before posting the issues.

Also, there is no apitrace in #514, but it is an excellent issue report. If you can identify the cause of an issue, you can report that as the issue so that the developers need not spend time going through a mountain of trace data to diagnose the more trivial ones.

[Plagman]

Hey Richard,

Please keep in mind you're dealing with the upstream DXVK project here, which is standalone, and used in many other ways than just Proton. So the fact that the typical bug reporting pipeline here isn't 100% optimal for a Proton end-user in your opinion might not actually be relevant. Maybe it would be more appropriate to raise this process issue in the Proton bug tracker?

[ryao]

@Plagman I am asking for documentation of how to diagnose issues in DXVK and that is DXVK specific. I only mentioned the proton issue because of @Guy1524’s insistence that I ignored reporting guidelines when I tried to ask for documentation. I had no idea how to do that until right before my HMD broke.

However, what I wanted to do was debug the issues so that I could say what is wrong inside the software, which is what I had planned to at least try doing before filing issues. I cited a few broken games only because I thought that they might be helpful when generating documentation of how to triage issues. They had specific things wrong with them in different ways, which might have saved developer time if someone were willing to document what I had requested. In particular, I imagined it would keep them from having to hunt for examples.

[ryao]

If it helps, there is no need to actively pursue this. I just ask that when someone is doing one of the items on that list, that they write down what they did for others to study and learn. It would be helpful to those trying to learn how to contribute good reports.

[Guy1524]

To see what is wrong in the software, you'll need to have a good grasp on the Vulkan and D3D11 APIs, and studying DXVK code would help. Nobody is stopping you from doing that and finding a bug. Until you do that, you should use the issue report template.

[ryao]

@Guy1524 It would help if I had a description of a debugging session or could instrument the DXVK code to study it. Eyeballing code is a lousy way of learning how it works. Right now, I have found a vague description about using a winelib build, but the winelib build does not happily substitute for the native build when I try it. Some basic directions would be helpful. At the moment, I am still perplexed by how the thing even compiles for people because it references native_vkGetInstanceProcAddrWINE, which does not exist. I had to pass -Dnative_vkGetInstanceProcAddrWINE=wine_vkGetInstanceProcAddr to the preprocessor to make it happy.

Also, a number of these reports could belong elsewhere. Blindly using the issue report template (which I am unable to do in the things that matter to me until my HMD is fixed) would flood the bug tracker. Would you rather that I file dozens of bug reports here without understanding if they belong here? It is very easy to just run through a steam library, see what is broken, run apitrace and then post a report. I don’t think that is the right thing to do when it is unclear whose bug it is.

[ryao]

Maybe some examples would help to explain my request. Look at how debugging methodology is described here:

https://blog.cloudflare.com/the-story-of-one-latency-spike/
http://www.brendangregg.com/blog/2016-10-04/linux-bcc-mysqld-qslower.html
https://www.cyberciti.biz/tips/linux-core-dumps.html

These things tell other developers how to do things efficiently. I am a developer in another area. I would like to understand things before filing the bulk of my issue reports. I am asking for descriptions of how to do some basic things so that I can avoid posting bug reports to the wrong places, which would waste core developer time. Posting documentation of such things makes the community more efficient.

[K0bin]

There is no easy set path toward figuring out an issue besides what's in the issue template and I don't understand how you're trying to provide any meaningful information without any knowledge about graphics programming.

[ryao]

I can learn. However without knowing the debugging tricks, I cannot possibly try. Furthermore, you do not necessarily need to understand the graphics APIs to diagnose a pipeline stall like is happening in The Lab. You just need the tools to find out where things block. I do not understand the resistance to providing descriptions of how issues have been debugged to help others get started.

[pchome]

@ryao
A good point to start is to carefully read README and Wiki, then titles of opened/closed issues and opened/closed PRs, then contents or use search.

The answers for most of your questions are already there.

1. Minimum requirements
   
   
   
   • for MinGW: wine-3.10,
     
     (you can use Proton 3.7, because it have some Vulkan patches).
   
   

* for `winelib` builds: wine-3.15,
  (you **can't** use Proton 3.7, no lib32 fixes, no required Vulkan patches).

1. Debugging: #392
2. Tools and IDE: https://github.com/doitsujin/dxvk/issues/259#issuecomment-385373073
3. ...

[ryao]

It would have been nice to summarize things, but I’ll take that. Thanks. I’ll close this given that nobody seems to want things summarized.

[K0bin]

Here's an example of what the debugging process might look like:

• Take a look at screenshots and the description of the issue
• Try to reproduce it using the apitrace (and find out if it also happens on different drivers)
• Make an educated guess like broken cubemap rendering or broken shadow maps or what ever
• Use Apitrace, Renderdoc and printf debugging to try and confirm that guess
• Fix it

And this process obviously require a very good knowledge of the codebase, D3D11 and Vulkan. This is very vague because as I already said there is no magic formula to narrow down issues and it depends on the issue itself.

[ryao]

@K0bin printf debugging is slow. You should use dynamic tracing. It is much faster because it avoids having to edit files and recompile. There are some examples here showing it in both userspace and the kernel:

http://www.brendangregg.com/perf.html#DynamicTracingEg

You need to build with debug symbols for it to work though. You also need to use the winelib version of DXVK so that you can use Linux infrastructure to instrument it. Unfortunately, I have been unsuccessful in building/installing the winelib in a way that does not crash despite doitsujin saying that he uses it to debug. I really want to know the details of how he (or anyone else using it) does it. That would help me work out what is different between our environments.

Also, I already said “I understand that the examples that I am requesting will not be cookie cutter solutions that can be applied to arbitrary problems”. I am aware that there is no one method. I just wanted to know the methods in use so that I can study them, try to apply them and see if I learn anything useful before filing certain bug reports.

Do people have a way of using the Linux build of apitrace or is the Windows build a must? The same question applies for renderdoc.

By the way, the dynamic tracing remark demonstrates why sharing information on how to debug makes the community more efficient. I can use dynamic tracing to find an issue in the fraction of the time that someone else checking the same things with printfs would find it. :)

[pchome]

By the way, the dynamic tracing remark demonstrates ...

... how information remain in issue comments.

Probably one of participants will reference your comment for a next one requesting such information. Or, probably not.

nobody seems to want things summarized

It's always preferable for things to be summarized, but you asking for people to do it for you.

[ryao]

It's always preferable for things to be summarized, but you asking for people to do it for you.

That is what a request for documentation is.

There is a certain amount of work that is reasonable to expect new contributors to do, but there are also some things that really ought to be shared to reduce the barrier of entry. That is why I was asking for documentation. It could go up on the wiki so everyone would have access to it. The Gentoo project for example has its devmanual:

https://devmanual.gentoo.org/

It is a reference of basic information to make development easier for both current and new contributors. DXVK has nearly nothing to support new contributors. It is a situation where it would be reasonable for those knowledgeable to write something for those who are new. Documenting how examples of a few key classes of bugs were diagnosed would give them something.

[John-Gee]

Unfortunately, I have been unsuccessful in building/installing the winelib in a way that does not crash

I use this which works well for me:
https://aur.archlinux.org/packages/dxvk-wine64-git/

It's not gentoo, but it might show you how to get it running. I have not tried debugging it though so not sure.

As for apitrace, I imagine you cannot use the Linux build to debug DXVK as the calls it would see would be VK ones. Of course I'm happy to be wrong.

[ryao]

@John-Gee Thanks. I will look through that.

[doitsujin]

@John-Gee

As for apitrace, I imagine you cannot use the Linux build to debug DXVK as the calls it would see would be VK ones. Of course I'm happy to be wrong.

You're wrong in that Apitrace doesn't support Vulkan, so you'd see nothing at all.

@ryao K0bin is absolutely right with his answer. Either you have experience with debugging graphics-related issues, in which case you probably know what to look for anyway, or you don't, in which case I would kindly suggest to just stick to the usual screenshot + log + apitrace thing so that I can reproduce the problem locally. This applies to all bugs that cause crashes or graphical issues.

And please don't use #514 as an excuse to not follow the issue template. That issue discusses the way DXVK translates D3D shaders, not a particular issue with a particular game.