Substrate: Windows client fail to connect to network

Created on 13 Feb 2019  Â·  22Comments  Â·  Source: paritytech/substrate

latest master branch(c5077d1e4526e8290090cad54d8c18eb819a8cfe) fails connect to charred-cherry network.

screen shot 2019-02-14 at 9 47 16 am

Z1-question
Source
picture kenhuang

All 22 comments

Master won't connect any more as there have been breaking changes. v0.9 branch should still connect though.

gavofyork picture gavofyork on 15 Feb 2019

When running from mac, it is able to sync block without issue. On windows is not able to start the sync process.

➜  substrate git:(c5077d1e) ✗ ./target/release/substrate --chain ~/Downloads/charred-cherry.json
2019-02-20 11:35:37 Substrate Node
2019-02-20 11:35:37   version 0.10.0-c5077d1e-x86_64-macos
2019-02-20 11:35:37   by Parity Technologies, 2017-2019
2019-02-20 11:35:37 Chain specification: Charred Cherry
2019-02-20 11:35:37 Node name: tangible-sidewalk-9781
2019-02-20 11:35:37 Roles: FULL
2019-02-20 11:35:37 Best block: #1559
2019-02-20 11:35:37 Local node address is: /ip4/0.0.0.0/tcp/30333/p2p/Qmc5Yj42JWYG3e5W5JLF6HMmWMQU3pmsmaYWbpP2JDxzh8
2019-02-20 11:35:37 Kademlia random query has yielded empty results
2019-02-20 11:35:37 Listening for new connections on 127.0.0.1:9944.
2019-02-20 11:35:37 Idle (0 peers), best: #1559 (0xe30d…3722), finalized #0 (0x7e22…f3c3), ⬇ 0 ⬆ 0
2019-02-20 11:35:38 Kademlia random query has yielded empty results
2019-02-20 11:35:42 Syncing 46.8 bps, target=#447398 (2 peers), best: #1793 (0x0402…a557), finalized #0 (0x7e22…f3c3), ⬇ 23.1kiB/s ⬆ 0.8kiB/s
2019-02-20 11:35:43 Network misbehaviour from PeerId(QmYHdf5E3Deq64KsPFWPfaX6cR37CgGDatFjFDsW6bhxDb) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-20 11:35:44 Network misbehaviour from PeerId(QmYWrEtg4iQYwV9PG37PhfLHLATQJUTYiZRyoUvSYny9ba) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-20 11:35:47 Network misbehaviour from PeerId(QmYaCKQgKJ68YDv6BraM4vTh1rUz6v2D3m72hYuT2ra57H) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-20 11:35:47 Syncing 172.6 bps, target=#447398 (2 peers), best: #2656 (0x8a44…a2c0), finalized #0 (0x7e22…f3c3), ⬇ 64.6kiB/s ⬆ 1.1kiB/s
2019-02-20 11:35:52 Syncing 174.6 bps, target=#447398 (2 peers), best: #3529 (0x241f…774a), finalized #0 (0x7e22…f3c3), ⬇ 52.2kiB/s ⬆ 0.1kiB/s
2019-02-20 11:35:57 Syncing 168.8 bps, target=#447398 (2 peers), best: #4373 (0xca26…8174), finalized #0 (0x7e22…f3c3), ⬇ 33.1kiB/s ⬆ 0.3kiB/s
2019-02-20 11:36:02 Syncing 162.6 bps, target=#447398 (2 peers), best: #5186 (0xedd2…dd2b), finalized #0 (0x7e22…f3c3), ⬇ 22.0kiB/s ⬆ 90 B/s
2019-02-20 11:36:07 Syncing 171.0 bps, target=#447399 (2 peers), best: #6041 (0x12bf…6a4c), finalized #0 (0x7e22…f3c3), ⬇ 38.2kiB/s ⬆ 75 B/s
2019-02-20 11:36:12 Syncing 159.0 bps, target=#447399 (2 peers), best: #6836 (0x283b…e6b7), finalized #0 (0x7e22…f3c3), ⬇ 33.1kiB/s ⬆ 0.2kiB/s
kenhuang picture kenhuang on 19 Feb 2019

We can not reproduce this on windows on a local machine. Maybe you could try latest master and just run the node. (it will pick automatically the latest test net)

bkchr picture bkchr on 25 Feb 2019

I've compiled Substrate on a Windows machine and ran it, and it seems to sync just fine.
Do you still see the issue @kenhuang ?

tomaka picture tomaka on 25 Feb 2019

@xlc are you still facing the issue?

shawntabrizi picture shawntabrizi on 25 Feb 2019

Still having network issue on the latest master branch.

$ ./target/release/substrate.exe
2019-02-27 22:59:18 Substrate Node
2019-02-27 22:59:18   version 0.10.0-cf46d62d-x86_64-windows-msvc
2019-02-27 22:59:18   by Parity Technologies, 2017-2019
2019-02-27 22:59:18 Chain specification: Dried Danta
2019-02-27 22:59:18 Node name: cagey-tank-7866
2019-02-27 22:59:18 Roles: FULL
2019-02-27 22:59:18 Generated a new keypair: 046617c2a396d7675287c9b804ffa0d542e73f867a7a60afa9fc70e21855f75f (5CAUNo8n...)
2019-02-27 22:59:19 Initialising Genesis block/state (state: 0x29bb…8986, header-hash: 0xfc0f…8777)
2019-02-27 22:59:19 Loaded block-time = 6 seconds from genesis on first-launch
2019-02-27 22:59:19 Loading GRANDPA authorities from genesis on what appears to be first startup.
2019-02-27 22:59:19 Best block: #0
2019-02-27 22:59:19 Local node address is: /ip4/0.0.0.0/tcp/30333/p2p/QmWiXq6JTi79NHs5CeHv8svdryj2QqnPTf4hz1n1RawhJP
2019-02-27 22:59:19 Multi-threaded server is not available on Windows. Falling back to single thread.
2019-02-27 22:59:19 Listening for new connections on 127.0.0.1:9944.
2019-02-27 22:59:20 Kademlia random query has yielded empty results
2019-02-27 22:59:31 Network misbehaviour from PeerId(QmYHdf5E3Deq64KsPFWPfaX6cR37CgGDatFjFDsW6bhxDb) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:31 Network misbehaviour from PeerId(QmZuDX6HSeXY4FVHwrEQ5SzKQGaGm443ZtYpk1rxkvpHP6) with protocol [115, 117, 112]: Upgrade(Select(MultistreamSelectError(IoError(Custom { kind: WriteZero, error: StringError("connection is closed") }))))
2019-02-27 22:59:32 Network misbehaviour from PeerId(QmYT3p4qGj1jwb7hDx1A6cDzAPtaHp3VR34vmw5BsXXB8D) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:37 Network misbehaviour from PeerId(QmV87bak5PD98CBJ1LxqwtyzuAL6X1gpQPtyWWgn7u4gPj) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:40 Network misbehaviour from PeerId(QmePFHyN1kNNdKCQ77Yxdf2bB4dnt2eummzAowoPuVc1y8) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:40 Network misbehaviour from PeerId(QmPEzArHGw29rhbhKPoK2Lc2XVxG1hGwdtp8rsAV1XhvPi) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:52 Network misbehaviour from PeerId(QmUMDwv88QrtFBkYzjLSgrBxxzz2wCG58byJCA1za7rkbb) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:52 Network misbehaviour from PeerId(QmRX3imcmrwHSNzAr92VMozzBRVpE6t4vFbEEPUw4awxZB) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:53 Network misbehaviour from PeerId(QmPkRr3X5QuMNoyC9j4UWDvxeVJb51oW87neMZcYJGNwwK) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-27 22:59:55 connection error: decode error: i/o error: I/O error: An established connection was aborted by the software in your host machine. (os error 10053)
kenhuang picture kenhuang on 27 Feb 2019

I could indeed reproduce, but only from time to time.

tomaka picture tomaka on 27 Feb 2019

The node looks stuck, but I think it's more that it's extremely slow.
TheImportQueueWorker thread runs at full speed, but the other threads are asleep most of the time.

I don't really understand where this line comes from:

Multi-threaded server is not available on Windows. Falling back to single thread.

tomaka picture tomaka on 27 Feb 2019

Switch to a more powerful machine (iMac 27" with i7 CPU) same issue, CPU for the substrate process is around 15%.

Logs from the new machine:

$ ~/Downloads/substrate.exe
2019-02-28 00:30:22 Substrate Node
2019-02-28 00:30:22   version 0.10.0-cf46d62d-x86_64-windows-msvc
2019-02-28 00:30:22   by Parity Technologies, 2017-2019
2019-02-28 00:30:22 Chain specification: Dried Danta
2019-02-28 00:30:22 Node name: overconfident-test-4394
2019-02-28 00:30:22 Roles: FULL
2019-02-28 00:30:29 Best block: #0
2019-02-28 00:30:29 Local node address is: /ip4/0.0.0.0/tcp/30333/p2p/QmSesHsxv1rREqsxjDh31USuv4REvhstQ1C11pFeLWpzcm
2019-02-28 00:30:29 Multi-threaded server is not available on Windows. Falling back to single thread.
2019-02-28 00:30:29 Listening for new connections on 127.0.0.1:9944.
2019-02-28 00:30:29 Kademlia random query has yielded empty results
2019-02-28 00:30:32 Network misbehaviour from PeerId(QmPkRr3X5QuMNoyC9j4UWDvxeVJb51oW87neMZcYJGNwwK) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:32 Network misbehaviour from PeerId(QmYT3p4qGj1jwb7hDx1A6cDzAPtaHp3VR34vmw5BsXXB8D) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:32 Network misbehaviour from PeerId(QmPEzArHGw29rhbhKPoK2Lc2XVxG1hGwdtp8rsAV1XhvPi) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:33 Network misbehaviour from PeerId(QmYHdf5E3Deq64KsPFWPfaX6cR37CgGDatFjFDsW6bhxDb) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:33 Network misbehaviour from PeerId(QmV87bak5PD98CBJ1LxqwtyzuAL6X1gpQPtyWWgn7u4gPj) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:34 Network misbehaviour from PeerId(QmePFHyN1kNNdKCQ77Yxdf2bB4dnt2eummzAowoPuVc1y8) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:35 Network misbehaviour from PeerId(QmRX3imcmrwHSNzAr92VMozzBRVpE6t4vFbEEPUw4awxZB) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-02-28 00:30:38 Network misbehaviour from PeerId(QmZuDX6HSeXY4FVHwrEQ5SzKQGaGm443ZtYpk1rxkvpHP6) with protocol [115, 117, 112]: Upgrade(Select(MultistreamSelectError(IoError(Custom { kind: WriteZero, error: StringError("connection is closed") }))))
kenhuang picture kenhuang on 27 Feb 2019

Nevermind, the slowness also happens on Linux and is caused by the latest master.
I don't think I am able to reproduce.

tomaka picture tomaka on 27 Feb 2019

@kenhuang could you please run ~/Downloads/substrate.exe -ldebug? Maybe that gives use some better insights.

bkchr picture bkchr on 27 Feb 2019

Here is the debug logs:
debug.log

kenhuang picture kenhuang on 27 Feb 2019

@kenhuang ty, lets wait for @tomaka as he probably is the best for interpreting this libp2p logs :)

bkchr picture bkchr on 27 Feb 2019

Hmm... That's a bit random but maybe the Windows firewall is blocking Substrate?

tomaka picture tomaka on 28 Feb 2019

Turn offed windows firewall seems still have the same issue.

screen shot 2019-03-01 at 10 31 58 am

kenhuang picture kenhuang on 28 Feb 2019

Do you see the node using a lot of CPU while it's stuck?

tomaka picture tomaka on 1 Mar 2019

Using around 15% of the CPU.

kenhuang picture kenhuang on 1 Mar 2019

Are these error messages expected when running a client on windows?

shawn@DESKTOP-9EGNR6V MINGW64 ~/Documents/GitHub/substrate (master)
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 2.15s
     Running `target\debug\substrate.exe`
2019-03-14 00:19:49 Substrate Node
2019-03-14 00:19:49   version 0.10.0-8930f297-x86_64-windows-msvc
2019-03-14 00:19:49   by Parity Technologies, 2017-2019
2019-03-14 00:19:49 Chain specification: Dried Danta
2019-03-14 00:19:49 Node name: stereotyped-icicle-4914
2019-03-14 00:19:49 Roles: FULL
2019-03-14 00:19:50 Generated a new keypair: bb7b9dc71bf5aec2172badf2b06cc708fbcf65743d325e3b39e32cd5f68bce89 (5GJXXtLG...)
2019-03-14 00:19:51 Initialising Genesis block/state (state: 0x29bb…8986, header-hash: 0xfc0f…8777)
2019-03-14 00:19:51 Loaded block-time = 6 seconds from genesis on first-launch
2019-03-14 00:19:51 Loading GRANDPA authority set from genesis on what appears to be first startup.
2019-03-14 00:19:51 Best block: #0
2019-03-14 00:19:51 Local node address is: /ip4/0.0.0.0/tcp/30333/p2p/QmQwSq5HJTdcywpL2hcgk7wVsVTr2AekgmoqV8no3BVX3k
2019-03-14 00:19:51 Multi-threaded server is not available on Windows. Falling back to single thread.
2019-03-14 00:19:51 Listening for new connections on 127.0.0.1:9944.
2019-03-14 00:19:52 Kademlia random query has yielded empty results
2019-03-14 00:19:54 Network misbehaviour from PeerId(QmRn8yqn6LPTcM7fc2Wo76u9wgqw2WKU9Ysmegm9NCSrwU) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:19:54 Network misbehaviour from PeerId(QmUk6BtZNLC5TdZH31ZKvSpbvQaUomd2ZG7wjYHgGWrLuB) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:19:58 Handler initialization process is too long
2019-03-14 00:20:08 Network misbehaviour from PeerId(QmPiGU1jwL9UDw2FMyMQFr9FdpF9hURKxkfy6PWw6aLsur) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:20:08 Syncing, target=#300553 (5 peers), best: #128 (0x317a…b98a), finalized #0 (0xfc0f…8777), ⬇ 0.9kiB/s ⬆ 0.2kiB/s
2019-03-14 00:20:23 Syncing  8.6 bps, target=#300554 (5 peers), best: #256 (0xed97…dded), finalized #0 (0xfc0f…8777), ⬇ 0.9kiB/s ⬆ 0
2019-03-14 00:20:23 Network misbehaviour from PeerId(QmQmfm7XXXf5qLd8qANhMJ34rR1XyRYDndLnMpe8RK2LHF) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:20:23 Handler initialization process is too long
2019-03-14 00:20:23 Handler initialization process is too long
2019-03-14 00:20:23 Handler initialization process is too long
2019-03-14 00:20:24 Network misbehaviour from PeerId(QmVGwAmgrS4HTi8Qka6aVAdG3uS7u7nEzcdE1qdbgSttiw) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:20:24 Network misbehaviour from PeerId(QmUghPWmHR8pQbZyBMeYzvPcH7VRcTiBibcyBG7wMKHaSZ) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))

It does seem that it is syncing between these error messages:

2019-03-14 00:21:23 Network misbehaviour from PeerId(QmYHdf5E3Deq64KsPFWPfaX6cR37CgGDatFjFDsW6bhxDb) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:23 Network misbehaviour from PeerId(QmYWrEtg4iQYwV9PG37PhfLHLATQJUTYiZRyoUvSYny9ba) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:23 Network misbehaviour from PeerId(Qmdu5PV9ofUYKCwZiLouviYhW4DnPK9LeaGS8RfSeVNUfW) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:24 Network misbehaviour from PeerId(QmV87bak5PD98CBJ1LxqwtyzuAL6X1gpQPtyWWgn7u4gPj) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:24 Handler initialization process is too long
2019-03-14 00:21:24 Handler initialization process is too long
2019-03-14 00:21:24 Handler initialization process is too long
2019-03-14 00:21:24 Handler initialization process is too long
2019-03-14 00:21:25 Network misbehaviour from PeerId(QmRX3imcmrwHSNzAr92VMozzBRVpE6t4vFbEEPUw4awxZB) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:25 Network misbehaviour from PeerId(QmTasL84sqnp1dHAKtWkeoYRPNZF6aKidDby7rMuX6ZW9G) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:25 Network misbehaviour from PeerId(QmPkRr3X5QuMNoyC9j4UWDvxeVJb51oW87neMZcYJGNwwK) with protocol [115, 117, 112]: Upgrade(Select(NoProtocolFound))
2019-03-14 00:21:28 Syncing  8.4 bps, target=#300569 (5 peers), best: #810 (0x3828…46f6), finalized #0 (0xfc0f…8777), ⬇ 18.0kiB/s ⬆ 7.5kiB/s
2019-03-14 00:21:33 Syncing  8.3 bps, target=#300570 (5 peers), best: #852 (0x0b52…7fc4), finalized #0 (0xfc0f…8777), ⬇ 96.6kiB/s ⬆ 1.0kiB/s
2019-03-14 00:21:38 Syncing  8.8 bps, target=#300571 (5 peers), best: #896 (0xde89…664b), finalized #0 (0xfc0f…8777), ⬇ 2.0kiB/s ⬆ 0.8kiB/s
2019-03-14 00:21:43 Syncing  8.7 bps, target=#300572 (5 peers), best: #940 (0xe6e1…0df7), finalized #0 (0xfc0f…8777), ⬇ 7.3kiB/s ⬆ 2.6kiB/s
2019-03-14 00:21:44 Handler initialization process is too long
2019-03-14 00:21:44 Handler initialization process is too long
2019-03-14 00:21:44 Handler initialization process is too long
2019-03-14 00:21:44 Handler initialization process is too long
2019-03-14 00:21:48 Syncing  8.5 bps, target=#300572 (5 peers), best: #983 (0x4dc2…c844), finalized #0 (0xfc0f…8777), ⬇ 3.7kiB/s ⬆ 1.8kiB/s
2019-03-14 00:21:53 Syncing  8.3 bps, target=#300573 (5 peers), best: #1025 (0x3ec6…efd3), finalized #0 (0xfc0f…8777), ⬇ 2.1kiB/s ⬆ 0.7kiB/s
2019-03-14 00:21:58 Syncing  8.3 bps, target=#300574 (5 peers), best: #1067 (0x2a3f…b7c3), finalized #0 (0xfc0f…8777), ⬇ 1.9kiB/s ⬆ 1.6kiB/s
2019-03-14 00:22:03 Syncing  8.8 bps, target=#300575 (5 peers), best: #1111 (0x568e…e375), finalized #0 (0xfc0f…8777), ⬇ 3.2kiB/s ⬆ 1.5kiB/s
2019-03-14 00:22:08 Syncing  8.3 bps, target=#300576 (5 peers), best: #1153 (0x4fad…95c7), finalized #0 (0xfc0f…8777), ⬇ 2.1kiB/s ⬆ 0.9kiB/s
2019-03-14 00:22:13 Syncing  8.3 bps, target=#300577 (5 peers), best: #1195 (0x162e…7478), finalized #0 (0xfc0f…8777), ⬇ 1.4kiB/s ⬆ 1.8kiB/s
2019-03-14 00:22:18 Syncing  8.5 bps, target=#300577 (5 peers), best: #1238 (0xdb23…8f3c), finalized #0 (0xfc0f…8777), ⬇ 3.1kiB/s ⬆ 1.6kiB/s
2019-03-14 00:22:23 Syncing  8.5 bps, target=#300578 (5 peers), best: #1281 (0xff9c…52ea), finalized #0 (0xfc0f…8777), ⬇ 1.0kiB/s ⬆ 0.7kiB/s

These were the steps (in order) to install substrate dependencies:

Install Substrate on Windows
===

1. https://aka.ms/buildtools
    vs_buildtools.exe
    > Please ensure the Windows 10 SDK component is included when installing the Visual C++ Build Tools.

    ![](https://i.imgur.com/zayVLmu.png)

    Requires Restart
2. rustup-init.exe
    Did not ask me to install vs_buildtools
    Default installation
    > To get started you need Cargo's bin directory (%USERPROFILE%\.cargo\bin) in your PATH environment variable. Future applications will automatically have the correct environment, but you may need to restart your current shell.

3. Commands in CMD
    ```
    rustup update nightly
    rustup target add wasm32-unknown-unknown --toolchain nightly
    rustup update stable

    ```

4. Install wasm-gc
    ```
    cargo install --git https://github.com/alexcrichton/wasm-gc
    ```

5. Install LLVM
    https://releases.llvm.org/download.html
    Did not install into System PATH

6. Install OpenSSL (through vcpkg)
    ```
    mkdir \Tools
    cd \Tools
    git clone https://github.com/Microsoft/vcpkg.git
    cd vcpkg
    .\bootstrap-vcpkg.bat
    .\vcpkg.exe install openssl:x64-windows-static
    ```

7. Add OpenSSL to System Variables

    ```
    $env:OPENSSL_DIR = 'C:\Tools\vcpkg\installed\x64-windows-static'
    $env:OPENSSL_STATIC = 'Yes'
    [System.Environment]::SetEnvironmentVariable('OPENSSL_DIR', $env:OPENSSL_DIR, [System.EnvironmentVariableTarget]::User)
    [System.Environment]::SetEnvironmentVariable('OPENSSL_STATIC', $env:OPENSSL_STATIC, [System.EnvironmentVariableTarget]::User)
    ```

8. Install cmake
    https://cmake.org/download/
shawntabrizi picture shawntabrizi on 14 Mar 2019

@shawntabrizi Yes

tomaka picture tomaka on 14 Mar 2019
👍1

I recently had similar issues with windows node not connecting successfully on our own chain, while the linux and mac nodes seemed to be working fine.

All platforms were built with stable rustc v1.34.0.

I reverted windows build to compile with v1.33.0 and the node now successfully connects and syncs with its peers.

mnaamani picture mnaamani on 16 Apr 2019

Triage.
This issue has been opened a long time ago, and a lot of code has changed in the networking since then.
Would need testing again.

tomaka picture tomaka on 24 Jul 2019

I think we can close this for now. No reaction in the last months, so it is probably fixed.

bkchr picture bkchr on 12 Aug 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Mischi picture Mischi  Â·  5Comments
JoshOrndorff picture JoshOrndorff  Â·  4Comments
gavofyork picture gavofyork  Â·  4Comments
drozdziak1 picture drozdziak1  Â·  5Comments
pepyakin picture pepyakin  Â·  5Comments