Walletwasabi: 1.1.9 "No such file or directory" warning around HWI

[hkjn]

General Description

~When start Wasabi 1.1.9, it fails to fully load:~

_Edit 2:_ The original title and description of this issue was wrong, the failure to learn was due to Tor issues. The remaining issue covers a warning around "No such file or directory" seen around HWI:

2019-09-17 10:47:38 INFO        Program (43)    Wasabi GUI started (10655ea5-a16e-40e5-9d52-a8f36bbd0bba).
2019-09-17 10:47:43 INFO        Global (142)    Config is successfully initialized.
2019-09-17 10:47:43 INFO        HwiProcessManager (240) Updating HWI...
2019-09-17 10:47:43 INFO        HwiProcessManager (270) Extracted /nix/store/03cdyx1srylqxx62pvf208fsyhcq71bx-wasabiwallet-1.1.9/opt/wasabiwallet/Hwi/Software/hwi-linux64.zip to `/home/user/.walletwasabi/client/hwi`.
2019-09-17 10:47:43 INFO        HwiProcessManager (282) Shell command executed: chmod -R 750 /home/user/.walletwasabi/client/hwi.
2019-09-17 10:47:43 INFO        TorProcessManager (253) Starting Tor monitor...
2019-09-17 10:47:43 INFO        Global (203)    TorProcessManager is initialized.
2019-09-17 10:47:43 INFO        TorProcessManager (66)  Tor is already running.
2019-09-17 10:47:44 INFO        Global (331)    Loaded AddressManager from `/home/user/.walletwasabi/client/AddressManager/AddressManagerMain.dat`.
2019-09-17 10:47:44 WARNING     WalletManagerViewModel (177)    System.ComponentModel.Win32Exception (2): No such file or directory
   at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setUser, UInt32 userId, UInt32 groupId, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at WalletWasabi.Hwi.HwiProcessManager.SendCommandAsync(String command, CancellationToken cancellationToken, Boolean isMutexPriority)
   at WalletWasabi.Hwi.HwiProcessManager.EnumerateAsync()
   at WalletWasabi.Gui.Tabs.WalletManager.WalletManagerViewModel.RefreshHardwareWalletListAsync()
2019-09-17 10:47:44 INFO        MainWindow.xaml (70)    UiConfig is successfully initialized.
2019-09-17 10:47:47 WARNING     WalletManagerViewModel (177)    System.ComponentModel.Win32Exception (2): No such file or directory
   at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setUser, UInt32 userId, UInt32 groupId, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at WalletWasabi.Hwi.HwiProcessManager.SendCommandAsync(String command, CancellationToken cancellationToken, Boolean isMutexPriority)
   at WalletWasabi.Hwi.HwiProcessManager.EnumerateAsync()
   at WalletWasabi.Gui.Tabs.WalletManager.WalletManagerViewModel.RefreshHardwareWalletListAsync()
2019-09-17 10:47:48 INFO        Global (280)    Start connecting to nodes...
2019-09-17 10:47:48 INFO        Global (301)    Start synchronizing filters...
2019-09-17 10:47:50 WARNING     WalletManagerViewModel (177)    System.ComponentModel.Win32Exception (2): No such file or directory
   at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setUser, UInt32 userId, UInt32 groupId, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
2019-09-17 10:47:54 INFO        Global (505)    Wallet loaded: [redacted].
2019-09-17 10:47:54 INFO        Global (435)    Start Chaumian CoinJoin service...
2019-09-17 10:47:54 INFO        Global (437)    Starting WalletService...
2019-09-17 10:47:54 INFO        CcjClient (137) CcjClient is successfully initialized.
2019-09-17 10:47:56 WARNING     WalletManagerViewModel (177)
[... "no such file or directory" stack trace repeats every 3 sec or so indefinitely ...]

~The stack trace appears as soon as Wasabi is started. In the GUI, selecting Load Wallet seems to have no effect in this state, with the "Loading.." state and spinning cog showing indefinitely, although it is accompanied by the Wallet loaded, Starting WalletService.. messages etc in the logs above.~

~The wallet files seem to exist on the filesystem, under ~.walletwasabi/client/Wallets/. With 1.1.8, the wallets seem to have loaded fine, and I recall starting Wasabi 1.1.9 successfully yesterday as well. After I restarted the app today, it seems to be in this state indefinitely. I'm not sure what changed.~

_Edit:_ Provided more info from logs and more details.
_Edit 2:_ The speculation above was incorrect. See comments for full details.

How To Reproduce?

1. Start wasabi 1.1.9 _on NixOS_
2. See repeated error in logs

Operating System

NixOS 19.03

Logs

See above.

Wasabi Version

1.1.9

[nopara73]

@lontivero can you take a look? If you fail, I'll take it.

[hkjn]

Actually, the wallet loaded eventually if I leave Wasabi running, after ~30 minutes of No such file or directory warnings logged:

2019-09-17 17:02:45 WARNING     WalletManagerViewModel (177)    System.ComponentModel.Win32Exception (2): No such file or directory
   at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setUser, UInt32 userId, UInt32 groupId, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at WalletWasabi.Hwi.HwiProcessManager.SendCommandAsync(String command, CancellationToken cancellationToken, Boolean isMutexPriority)
   at WalletWasabi.Hwi.HwiProcessManager.EnumerateAsync()
   at WalletWasabi.Gui.Tabs.WalletManager.WalletManagerViewModel.RefreshHardwareWalletListAsync()
WalletWasabi.Gui.Tabs.WalletManager.WalletManagerViewModel.RefreshHardwareWalletListAsync()
2019-09-17 17:02:51 WARNING     WalletService (410)     WalletWasabi.Exceptions.TorSocks5FailureResponseException: Tor SOCKS5 proxy responded with TtlExpired.
   at WalletWasabi.TorSocks5.TorSocks5Client.ConnectToDestinationAsync(String host, Int32 port, Boolean isRecursiveCall)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancel)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpMethod method, String relativeUri, HttpContent content, CancellationToken cancel)
   at TorHttpClientExtensions.SendAndRetryAsync(TorHttpClient client, HttpMethod method, HttpStatusCode expectedCode, String relativeUri, Int32 retry, HttpContent content, CancellationToken cancel)
   at WalletWasabi.WebClients.Wasabi.WasabiClient.GetMempoolHashesAsync(Int32 compactness, CancellationToken cancel)
   at WalletWasabi.Services.WalletService.LoadDummyMempoolAsync(SmartTransaction[] unconfirmedTransactions)
2019-09-17 17:02:51 INFO        Global (439)    WalletService started.

However, Wasabi still seemed to be in a strange state once the wallet does load. When I tried to register coins for CJ, they show up as queued, but the information about the time remaining to the next round is missing, and it permanently shows Number of registered peers: 0 / 100.

When I tried to send a coin in this state, the UI shows a red No coins selected where the amount field should be, even after I've selected all coins.

At this point, I started suspecting that the No such file or directory was a red herring, and then I noticed the TorSocks5FailureResponseException in the log output below.

I'm using my system service for Tor, and after restarting it, Wasabi was able to get a connection, download blocks, register coins for CJ, and send coins without any further action. After restarting Wasabi it loads the wallet quickly without issues, so it seems very likely that there was some issue with the system Tor service.

To summarize:

1. when I start Wasabi 1.1.9, it logs WARNING WalletManagerViewModel (177) System.ComponentModel.Win32Exception (2): No such file or directory every 2 sec until I load a wallet
2. there's some unknown state that Tor can end up in where it's really slow, making Wasabi take 30min+ to load a wallet

The warning log message might have been present earlier as well and might be innocuous, but is at any rate a red herring. I don't know how to repro the slow / broken Tor service, or whether there's anything Wasabi could be doing differently in that state (maybe give up on using it after some number of seconds and show an error instead of loading indefinitely?).

Feel free to close this issue or keep it open to track any suggestions as you see fit.

[lontivero]

Still researching. I am clueless because I have tried it all and I cannot make it fail.
Btw, there are two different errors in the shared log fragment. The hardest one is the first one related to hwi.

[lontivero]

@hkjn how are you running Wasabi? Did you build it from source code? Are you running it from command line? How exactly?

[hkjn]

@hkjn how are you running Wasabi? Did you build it from source code? Are you running it from command line? How exactly?

I'm running Wasabi from the CLI as:

$ wasabiwallet

I did not build it from source code, but installed the wasabiwallet package from the NixOS package manager. See https://github.com/zkSNACKs/WalletWasabi/issues/2130 for background.

@lontivero: You probably saw it already, but just in case I'll refer to my previous comment where I noticed that the issue preventing Wasabi from loading seems related to some Tor issue, and the warning logged around hwi + No such file or directory seems like it's unrelated to that Tor issue. With that said, I'd be happy to help look into what the hwi error is about; do we know which line of code is trying to do a file read just before the stack trace is thrown? Perhaps we could add the path that we're trying to read in the logging?

[lontivero]

@hkjn the problem is that we verify file exist and only in case the file is found in the correct folder we execute it. That's why this error is so weird, because the file is there but System.Diagnostics.Process.Start doesn't find it. It looks like a dotnet bug to me but that's is also hard to believe.

Below you can see where it fails. The line is number 146.

https://github.com/zkSNACKs/WalletWasabi/blob/8cc22a264ac8a4ab6af97459ad578b3d39313aa5/WalletWasabi/Hwi/HwiProcessManager.cs#L140-L155

[hkjn]

@hkjn the problem is that we verify file exist and only in case the file is found in the correct folder we execute it. That's why this error is so weird, because the file is there but System.Diagnostics.Process.Start doesn't find it. It looks like a dotnet bug to me but that's is also hard to believe.

Hmm, that is weird. Thanks for the info.

I've edited the issue's title and description to make it accurate, since the failure to load wallets was a red herring related to Tor issues on my system, as described in earlier comments.

[nopara73]

Let's ice this issue in favor of PR https://github.com/zkSNACKs/WalletWasabi/pull/1905, because I'm now fulltime on that PR and it completely refactors how we're doing these things.

[nopara73]

In fact, I just created a release to test the final packages: https://github.com/nopara73/WalletWasabi/releases/tag/hwitest2

Can you test it, too? (Namely if on the Hardware Wallets tab with the "Search for Hardware Wallets" throws an exception or not.)

[hkjn]

Thanks @nopara73, I'll try your release soon.

(I'm assuming https://github.com/nopara73/WalletWasabi/releases/tag/hwitest3 replaces https://github.com/nopara73/WalletWasabi/releases/tag/hwitest2.)

[nopara73]

Correct.

[hkjn]

@nopara73: When I click Hardware Wallet > Search for Hardware Wallets in the hwitest3 release, I get a similar stack trace as before:

2019-09-25 12:28:03 ERROR       LoadWalletViewModel (156)       System.ComponentModel.Win32Exception (2): No such file or directory
   at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setUser, UInt32 userId, UInt32 groupId, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at WalletWasabi.Hwi.ProcessBridge.HwiProcessBridge.SendCommandAsync(String arguments, Boolean openConsole, CancellationToken cancel)
   at WalletWasabi.Hwi.HwiClient.SendCommandAsync(IEnumerable`1 options, Nullable`1 command, String commandArguments, Boolean openConsole, CancellationToken cancel)
   at WalletWasabi.Hwi.HwiClient.EnumerateAsync(CancellationToken cancel)
   at WalletWasabi.Gui.Tabs.WalletManager.LoadWalletViewModel.EnumerateHardwareWalletsAsync()
   at WalletWasabi.Gui.Tabs.WalletManager.LoadWalletViewModel.<>c__DisplayClass30_0.<<-ctor>b__11>d.MoveNext()

To test the release, I first uninstalled the 1.1.9 release, then updated the package definition locally to point to your release, and rebuilt+installed the package. The version information agrees that I'm running your release:

$ wasabiwallet --version
Wasabi Client Version: 1.1.9.1
Compatible Coordinator Version: 3

[nopara73]

I'm installing a nix virtual machine.

[nopara73]

I'm not succeeding. Couldn't find vbox image published.

[nopara73]

The more I look at this issue the less it makes sense. It smells like a .NET Core bug. Which sucks, because it doesn't officially support nixos, so not sure what to do about it.

There are 3 ways to go about this issue:

• Update to .NET Core 3, that just came out today and hope for the best.
• Fixing it in a PR to .NET Core.
• Try to find a workaround.

[MaxHillebrand]

What other relevant features are in .NET Core 3? I assume we'll update to it eventually?

[nopara73]

It's a larger update than Avalonia, but it should be a more stable one.

[nopara73]

There was an off by one error in the code. @lontivero could this be related? https://github.com/zkSNACKs/WalletWasabi/pull/1905#discussion_r328737269

[nopara73]

Yes, as I see it, this has a high chance of being the issue. @hkjn are you able to try PR https://github.com/zkSNACKs/WalletWasabi/pull/1905 ? Or would you prefer me to release a new version?

[nopara73]

I just rolled out another test release, so no need to build: https://github.com/nopara73/WalletWasabi/releases

[hkjn]

I just rolled out another test release, so no need to build: https://github.com/nopara73/WalletWasabi/releases

Thanks @nopara73, but on hwitest7 release, I still see an exception in logs when I click Hardware Wallet > Search for Hardware Wallets:

2019-09-28 18:47:05 ERROR       LoadWalletViewModel (147)       System.ComponentModel.Win32Exception (2): No such file or directory
   at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setUser, UInt32 userId, UInt32 groupId, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at WalletWasabi.Hwi.ProcessBridge.HwiProcessBridge.SendCommandAsync(String arguments, Boolean openConsole, CancellationToken cancel)
   at WalletWasabi.Hwi.HwiClient.SendCommandAsync(IEnumerable`1 options, Nullable`1 command, String commandArguments, Boolean openConsole, CancellationToken cancel, Boolean isRecursion)
   at WalletWasabi.Hwi.HwiClient.EnumerateAsync(CancellationToken cancel, Boolean isRecursion)
   at WalletWasabi.Gui.Tabs.WalletManager.LoadWalletViewModel.EnumerateHardwareWalletsAsync()
   at WalletWasabi.Gui.Tabs.WalletManager.LoadWalletViewModel.<>c__DisplayClass30_0.<<-ctor>b__11>d.MoveNext()

The output of wasabiwallet --version:

Wasabi Client Version: 1.1.9.2
Compatible Coordinator Version: 3

[lontivero]

This is the same that we discussed here: https://github.com/zkSNACKs/WalletWasabi/issues/2856#issuecomment-575324567

[nopara73]

Actually not closing until fix is confirmed.

[molnard]

@jmacato Can you take a quick look at this on ur NixOs?

[jmacato]

@nopara73 @molnard seems like it's still there in the nixos package (v1.1.9.2). Will test master if it isnt there

[jmacato]

hmm.. i can't directly run wasabi master here on NixOS unstable, @hkjn do you have any guides on patching elf bins? seems like it can't find libX11 for some reason...

[lontivero]

@jmacato this issue is bacause some dependencies cannot be found in NixOS because the distro doesn't conform to FHS so, libraries are not where we assumed they are. Take a look at this comment https://github.com/zkSNACKs/WalletWasabi/issues/2856#issuecomment-575324567 and also to this question: https://unix.stackexchange.com/questions/91577/cannot-execute-binary-in-nixos-no-such-file-or-directory

[molnard]

@lontivero I read that in your comment that with env variable hack it is possible to run Wasabi but basically it is the lack of OS functionality.
Is there anything we can do with it?

1. Close this issue and add the env var tutorial to the docs.
2. Do something with Wasabi to make it work, however I don't know what.
3. Just close this issues.

My preference is 3 because we cannot fix the OS, neither we can change our executables and adding the tutorial to the docs which is extremely advanced can cause confusion.

[lontivero]

This problem exists only in NixOS because the OS works in a different way than the rest of the Linux distros. This is why it says "No such file or directory" even when the file is there. I think we are not going to fix it (in fact the problem is in HWI dependencies and not in Wasabi) so, we should close the issue.