Walletwasabi: Wasabi client did not connect back after backend restart

[molnard]

When I restarted the backend my client was running and it successfully detected that the backend is not connected. But afterward, it never connected back, even after an hour.

2020-08-05 11:47:33 ERROR   WasabiSynchronizer (337)    WalletWasabi.Exceptions.TorSocks5FailureResponseException: Tor SOCKS5 proxy responded with GeneralSocksServerFailure.
   at WalletWasabi.TorSocks5.TorSocks5Client.ConnectToDestinationAsync(String host, Int32 port, Boolean isRecursiveCall)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancel)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpMethod method, String relativeUri, HttpContent content, CancellationToken cancel)
   at TorHttpClientExtensions.SendAndRetryAsync(ITorHttpClient client, HttpMethod method, HttpStatusCode expectedCode, String relativeUri, Int32 retry, HttpContent content, CancellationToken cancel)
   at WalletWasabi.WebClients.Wasabi.WasabiClient.GetSynchronizeAsync(uint256 bestKnownBlockHash, Int32 count, Nullable`1 estimateMode, CancellationToken cancel)
   at System.Threading.Tasks.TaskExtensions.WithAwaitCancellationAsync[T](Task`1 me, CancellationToken cancel, Int32 waitForGracefulTerminationMilliseconds)
   at WalletWasabi.Services.WasabiSynchronizer.<>c__DisplayClass60_0.<<Start>b__0>d.MoveNext()
2020-08-05 11:47:36 ERROR   PeriodicRunner (73) WalletWasabi.Exceptions.TorSocks5FailureResponseException: Tor SOCKS5 proxy responded with GeneralSocksServerFailure.
   at WalletWasabi.TorSocks5.TorSocks5Client.ConnectToDestinationAsync(String host, Int32 port, Boolean isRecursiveCall)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancel)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpMethod method, String relativeUri, HttpContent content, CancellationToken cancel)
   at TorHttpClientExtensions.SendAndRetryAsync(ITorHttpClient client, HttpMethod method, HttpStatusCode expectedCode, String relativeUri, Int32 retry, HttpContent content, CancellationToken cancel)
   at WalletWasabi.WebClients.Wasabi.WasabiClient.GetVersionsAsync(CancellationToken cancel)
   at WalletWasabi.WebClients.Wasabi.WasabiClient.CheckUpdatesAsync(CancellationToken cancel)
   at WalletWasabi.Services.UpdateChecker.ActionAsync(CancellationToken cancel)
   at WalletWasabi.Bases.PeriodicRunner.ExecuteAsync(CancellationToken stoppingToken)
2020-08-05 11:47:58 ERROR   WasabiSynchronizer (337)    WalletWasabi.Exceptions.TorSocks5FailureResponseException: Tor SOCKS5 proxy responded with GeneralSocksServerFailure.
   at WalletWasabi.TorSocks5.TorSocks5Client.ConnectToDestinationAsync(String host, Int32 port, Boolean isRecursiveCall)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancel)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpMethod method, String relativeUri, HttpContent content, CancellationToken cancel)
   at TorHttpClientExtensions.SendAndRetryAsync(ITorHttpClient client, HttpMethod method, HttpStatusCode expectedCode, String relativeUri, Int32 retry, HttpContent content, CancellationToken cancel)
   at WalletWasabi.WebClients.Wasabi.WasabiClient.GetSynchronizeAsync(uint256 bestKnownBlockHash, Int32 count, Nullable`1 estimateMode, CancellationToken cancel)
   at System.Threading.Tasks.TaskExtensions.WithAwaitCancellationAsync[T](Task`1 me, CancellationToken cancel, Int32 waitForGracefulTerminationMilliseconds)
   at WalletWasabi.Services.WasabiSynchronizer.<>c__DisplayClass60_0.<<Start>b__0>d.MoveNext()
2020-08-05 11:48:24 ERROR   WasabiSynchronizer (337)    WalletWasabi.Exceptions.TorSocks5FailureResponseException: Tor SOCKS5 proxy responded with GeneralSocksServerFailure.
   at WalletWasabi.TorSocks5.TorSocks5Client.ConnectToDestinationAsync(String host, Int32 port, Boolean isRecursiveCall)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancel)
   at WalletWasabi.TorSocks5.TorHttpClient.SendAsync(HttpMethod method, String relativeUri, HttpContent content, CancellationToken cancel)
   at TorHttpClientExtensions.SendAndRetryAsync(ITorHttpClient client, HttpMethod method, HttpStatusCode expectedCode, String relativeUri, Int32 retry, HttpContent content, CancellationToken cancel)
   at WalletWasabi.WebClients.Wasabi.WasabiClient.GetSynchronizeAsync(uint256 bestKnownBlockHash, Int32 count, Nullable`1 estimateMode, CancellationToken cancel)
   at System.Threading.Tasks.TaskExtensions.WithAwaitCancellationAsync[T](Task`1 me, CancellationToken cancel, Int32 waitForGracefulTerminationMilliseconds)
   at WalletWasabi.Services.WasabiSynchronizer.<>c__DisplayClass60_0.<<Start>b__0>d.MoveNext()
2020-08-05 13:01:06 INFO    MainWindow.xaml (109)   UiConfig is saved.
2020-08-05 13:01:06 WARNING Global (647)    Process is exiting.
2020-08-05 13:01:06 INFO    WalletManager (356) KeyManager backup saved to 
2020-08-05 13:01:06 INFO    ClientState (60)    Coin removed from the waiting list: 
2020-08-05 13:01:06 INFO    CoinJoinClient (923)    Coin dequeued:  Reason: ApplicationExit.
2020-08-05 13:01:06 INFO    Wallet (276)    ChaumianClient is stopped.
2020-08-05 13:01:06 INFO    WalletManager (358) Wallet is stopped.
2020-08-05 13:01:06 INFO    WalletManager (356) KeyManager backup saved to `C:\Users\molnard\AppData\Roaming\WalletWasabi\Client\WalletBackups\LizardMain2 - Copy.json`.
2020-08-05 13:01:06 INFO    Wallet (276)    ChaumianClient is stopped.
2020-08-05 13:01:06 INFO    WalletManager (358) Wallet is stopped.
2020-08-05 13:01:06 INFO    Global (692)    RpcServer is stopped.
2020-08-05 13:01:06 INFO    Global (699)    Disposed FeeProviders.
2020-08-05 13:01:06 INFO    Global (706)    CoinJoinProcessor is disposed.
2020-08-05 13:01:06 INFO    Global (713)    Synchronizer is stopped.
2020-08-05 13:01:06 INFO    HostedServices (80) Stopped RPC Monitor.
2020-08-05 13:01:06 INFO    HostedServices (80) Stopped Block Notifier.
2020-08-05 13:01:06 INFO    HostedServices (80) Stopped Software Update Checker.
2020-08-05 13:01:06 INFO    HostedServices (80) Stopped RPC Fee Provider.
2020-08-05 13:01:07 INFO    HostedServices (130)    Disposed Software Update Checker.
2020-08-05 13:01:07 INFO    HostedServices (130)    Disposed Block Notifier.
2020-08-05 13:01:07 INFO    HostedServices (130)    Disposed RPC Monitor.
2020-08-05 13:01:07 INFO    HostedServices (130)    Disposed RPC Fee Provider.
2020-08-05 13:01:07 INFO    Global (722)    Stopped background services.
2020-08-05 13:01:07 INFO    Global (733)    AddressManager is saved to `C:\Users\molnard\AppData\Roaming\WalletWasabi\Client\AddressManager\AddressManagerMain.dat`.
2020-08-05 13:01:07 INFO    Global (746)    Nodes are disposed.
2020-08-05 13:01:07 INFO    P2pNode (188)   P2p Bitcoin node is disconnected.
2020-08-05 13:01:08 INFO    TorProcessManager (278) Tor did not work properly for 4415 seconds. Maybe it crashed. Attempting to start it...
2020-08-05 13:01:08 INFO    CoreNode (260)  Stopped.
2020-08-05 13:01:08 INFO    TorProcessManager (74)  Tor is already running.
2020-08-05 13:01:08 INFO    Global (771)    TorManager is stopped.
2020-08-05 13:01:08 INFO    Global (794)    AsyncMutex(es) are stopped.
2020-08-05 13:01:08 INFO    Global (810)    Wasabi stopped gracefully (7792ed0f-cb12-4bc2-82db-d871e6919960).
2020-08-05 13:01:09 INFO    Program (151)   Wasabi GUI stopped gracefully (7792ed0f-cb12-4bc2-82db-d871e6919960).

On another machine, instead of restarting Wasabi I just terminated the Tor process manually. It solved the problem.

[molnard]

Might relate: https://github.com/zkSNACKs/WalletWasabi/issues/4045

[molnard]

Test 1

sudo systemctl stop walletwasabi.service
sudo systemctl start walletwasabi.service

Wasabi reconnected every time.

Test 2

sudo apt-get update && cd ~/WalletWasabi && git pull && cd ~/WalletWasabi/WalletWasabi.Backend && dotnet restore && cd ~
sudo systemctl stop walletwasabi.service
dotnet publish ~/WalletWasabi/WalletWasabi.Backend --configuration Release --self-contained false
sudo systemctl start walletwasabi.service

Reconnect every time, except one time. So I went to the backend and sudo killall tor I saw that on client-side there was an exception and then Wasabi was able to reconnect.

Theory: quick walk around - restart Tor on backend every time after the service was restarted.

[lontivero]

Side note: Knots starts its own Tor daemon. When we killall tor we are killing it too. Probably is is not important but I think it is good to know.

[kiminuo]

@molnard The problem here is that you guess the real issue. If you are ok with logging (like #4131), then I would suggest to add way more logging statements for this logic to actually know what the problem is.

Networking issues are hard, it's hard to simulate, hard to guess problems. Logging is common approach here.

[molnard]

ACK

[molnard]

I tested this again and cannot reproduce this with the latest release. It connects back automatically every time, no matter what I do even if I do not restart Tor when deploying on Backend.