Pub: IO HTTP requests randomly hang under WSL Ubuntu
Running pub under Ubuntu 18 under WSL, version Dart pub 2.0.0-dev.53.0, the command pub install and other commands randomly hang during IO HTTP requests.
Verbose log usually gets stuck as follows
...
| content-type: application/json
| x-frame-options: SAMEORIGIN
| x-xss-protection: 1; mode=block
| x-content-type-options: nosniff
| server: dart:io with Shelf
IO : Get versions from https://pub.dartlang.org/api/packages/dart_style.
IO : HTTP GET https://pub.dartlang.org/api/packages/dart_style
| Accept: application/vnd.pub.v2+json
| user-agent: Dart pub 2.0.0-dev.53.0
Keeping it running for a few minutes will yield either Connection closed while receiving data or Connection closed before full header was received.
kaetemi
All 20 comments
ERR : Connection closed while receiving data
FINE: Exception type: ClientException
FINE: package:pub/src/source/hosted.dart 344 BoundHostedSource._throwFriendlyError
| package:pub/src/source/hosted.dart 144 BoundHostedSource.doGetVersions
| ===== asynchronous gap ===========================
| dart:async _Completer.completeError
| package:pub/src/source/hosted.dart BoundHostedSource.doGetVersions
| ===== asynchronous gap ===========================
| dart:async _asyncErrorWrapperHelper
| package:pub/src/source/hosted.dart 133 BoundHostedSource.doGetVersions
| package:pub/src/source.dart 168 BoundSource.getVersions
| package:pub/src/solver/package_lister.dart 76 PackageLister._versions.<fn>.<fn>
| package:pub/src/http.dart 273 withDependencyType
| package:pub/src/solver/package_lister.dart 75 PackageLister._versions.<fn>
| ===== asynchronous gap ===========================
| dart:async new Future.microtask
| package:pub/src/solver/package_lister.dart 74 PackageLister._versions.<fn>
| dart:async new Future.sync
| package:async/src/async_memoizer.dart 43 AsyncMemoizer.runOnce
| package:pub/src/solver/package_lister.dart 74 PackageLister._versions
| package:pub/src/solver/package_lister.dart 111 PackageLister.countVersions
| ===== asynchronous gap ===========================
| dart:async new Future.microtask
| package:pub/src/solver/package_lister.dart 108 PackageLister.countVersions
| package:pub/src/solver/version_solver.dart 354 VersionSolver._choosePackageVersion.<fn>
| ===== asynchronous gap ===========================
| dart:async new Future.microtask
| package:pub/src/solver/version_solver.dart 350 VersionSolver._choosePackageVersion.<fn>
| package:pub/src/utils.dart 302 minByAsync
| ===== asynchronous gap ===========================
| dart:async _asyncThenWrapperHelper
| package:pub/src/utils.dart 298 minByAsync
| package:pub/src/solver/version_solver.dart 350 VersionSolver._choosePackageVersion
| ===== asynchronous gap ===========================
| dart:async new Future.microtask
| package:pub/src/solver/version_solver.dart 316 VersionSolver._choosePackageVersion
| package:pub/src/solver/version_solver.dart 97 VersionSolver.solve
| ===== asynchronous gap ===========================
| dart:async _asyncThenWrapperHelper
| package:pub/src/solver/version_solver.dart 86 VersionSolver.solve
| package:pub/src/solver.dart 35 resolveVersions.<fn>
| package:pub/src/log.dart 378 progress
| package:pub/src/solver.dart 32 resolveVersions
| package:pub/src/entrypoint.dart 193 Entrypoint.acquireDependencies
| ===== asynchronous gap ===========================
| dart:async new Future.microtask
| package:pub/src/entrypoint.dart 192 Entrypoint.acquireDependencies
| package:pub/src/command/get.dart 38 GetCommand.run
| package:args/command_runner.dart 194 CommandRunner.runCommand
| ===== asynchronous gap ===========================
| dart:async new Future.microtask
| package:args/command_runner.dart 142 CommandRunner.runCommand
| package:pub/src/command_runner.dart 168 PubCommandRunner.runCommand.<fn>
| dart:async new Future.sync
| package:pub/src/utils.dart 108 captureErrors.<fn>
| package:stack_trace Chain.capture
| package:pub/src/utils.dart 123 captureErrors
| package:pub/src/command_runner.dart 168 PubCommandRunner.runCommand
This sounds like an issue with dart:io's HTTP implementation. Can you try writing a simple script that uses dart:io's HTTP APIs in a loop to see if they also hang for you?
nex3
on 19 May 2018
Running: Pub 2.0.0-dev.63.0
This is difficult even getting started using "pub get" for a newly created project. It eventually quits with the message:
IO : Spawning "tar --warning=no-unknown-keyword --extract --gunzip --no-same-owner --no-same-permissions --directory
ERR : Connection closed while receiving data
FINE: Exception type: ClientException
FINE: package:http/src/io_client.dart 52 IOClient.send.
| ===== asynchronous gap ===========================
| dart:async StreamView.listen
| package:pub/src/io.dart 627 _store
| package:pub/src/io.dart 873 extractTarGz
| ===== asynchronous gap ===========================
| dart:async _asyncThenWrapperHelper
| package:pub/src/io.dart 843 extractTarGz
| package:pub/src/source/hosted.dart 305 BoundHostedSource._download
| ===== asynchronous gap ===========================
| dart:async _asyncThenWrapperHelper
| package:pub/src/source/hosted.dart 297 BoundHostedSource._download
| package:pub/src/source/hosted.dart 199 BoundHostedSource.downloadToSystemCache
| ===== asynchronous gap ===========================
| dart:async _asyncThenWrapperHelper
| package:pub/src/entrypoint.dart 209 Entrypoint.acquireDependencies
| package:pub/src/command/get.dart 38 GetCommand.run
| package:args/command_runner.dart 194 CommandRunner.runCommand
mrjrieke
on 20 Jun 2018
This problem pretty much makes using Dart under WSL impossible. Is no-one doing that?
rtm
on 16 Aug 2018
kaetemi
on 29 Dec 2018
I'm having this issue too. It's pretty annoying but my current workaround is to use docker. I've added the following line to my ~/.aliases file and this works fine for project level dependencies.
alias pub="docker run --rm -it -v '$PWD:/app' -w /app google/dart pub $@"
Hope this helps some other dart devs on WSL until a better solution is found.
ezynda3
on 23 Jan 2019
Similar problem. I was trying to build flutter from source in Ubuntu in WSL. I was wondering why it was taking so long to download the android sdk. Dart appears to be the reason.
it just hangs after a few seconds of downloading, and stays stuck like this indefinitely. Working fine in an Ubuntu VM
[________ running 'src/third_party/dart/tools/sdks/dart-sdk/bin/dart --enable-asserts src/tools/android/android_sdk_downloader/lib/main.dart -y --out=src/third_party/android_tools --platform=28 --platform-revision=6 --build-tools-version=28.0.3 --platform-tools-version=28.0.1 --tools-version=26.1.1 --ndk-version=19.0.5232133' in '/mnt/c/Projects/flutterengine/engine'
src/tools/android/android_sdk_downloader/lib/main.dart:1: Warning: Interpreting this as package URI, 'package:android_sdk_downloader/main.dart'.
Downloading Android SDK and NDK artifacts...
SDK Platform 28: 1% SDK Build-Tools 28.0.3: 0% SDK Platform-Tools: 14% SDK Tools: 0% NDK: 0%]
rmawatson
on 8 Feb 2019
@rmawatson, if someone would take care to find out if it's the dart:io package that has a problem or just pub... generally, I would think a bug in dart:io under WSL is a bug in WSL :)
I noticed that there is some platform detection in the tarballing handling... so if downloading using dart:io under WSL works, then it's probably this platform detection that's doing something wrong.
Do you have tar in Ubuntu in WSL? Could you try to install tar and see if this fixes the issue... Normally, it'll be available on any distro?
How did you install dart? with apt-get?
jonasfj
on 11 Feb 2019
tar is installed by default. I didn't install dart. When building the flutter engine from source, gclient sync downloads everything it needs by the looks of it, including its own dart runtime, which is used for the android sdk download.
~/flutter-engine/engine/src/third_party/dart/tools/sdks/dart-sdk/bin$ ./dart --version
Dart VM version: 2.1.0-dev.9.4 (Thu Nov 8 23:00:07 2018 +0100) on "linux_x64"
I would think a bug in 'dart:io' under WSL is a bug in WSL :)
I would agree, but I've not seen similar behavior in anything else.
rmawatson
on 11 Feb 2019
So the best way to debug this would be to install dart, clone https://github.com/dart-lang/pub.git and do pub get && pub run test tests.. But if pub doesn't work, this is unlikely to get past the pub get stage... hmm...
jonasfj
on 11 Feb 2019
as you say, it hangs on the pub get stage.
Resolving dependencies... (15.0s)
+ analyzer 0.35.0
+ args 1.5.1
...
...
+ pub_semver 1.4.2
+ pubspec_parse 0.1.4
+ quiver 2.0.1
...
...
+ web_socket_channel 1.0.9
+ yaml 2.1.15
Downloading analyzer 0.35.0...
Downloading pub_semver 1.4.2...
...
...
Downloading dart_style 1.2.3...
Downloading quiver 2.0.1... <hangs>
I guess, we could test it by running pub get in a docker container, then copy out the ~/.pub-cache folder to the host and try to do pub run test... that might just work if you also copy out the .packages file create by pub get.. (maybe contents of .packages need fixing if files have been moved between the two environments).
jonasfj
on 11 Feb 2019
The problem showing with pub is surely just a symptom of the underlying issue that lies with dart:io or some other part of dart in WSL? My original paste was not using pub, but it what appears to be a dart script that is downloaded as part of gclient sync, when building flutter, (src/tools/android/android_sdk_downloader/lib/main.dart), and then executed to perform the android sdk download. Not really sure what pub has to do with that, but maybe I've missed something.
rmawatson
on 11 Feb 2019
In that case, maybe we should build a minimal example as suggested in: https://github.com/dart-lang/sdk/issues/27554#issuecomment-254803033
Maybe try to the following: https://gist.github.com/jonasfj/df09d7855294a1cf71a8de55d688b573
jonasfj
on 11 Feb 2019
I suffer from this problem as well.
I have just installed a fresh copy of WSL Ubuntu 18.04. Did a sudo apt update && sudo apt upgrade, installed Dart per the dartlang.org instructions and did run the programs suggested by @jonasfj , here are the results:
marcelo@MAQ04:~/d1$ time dart download.dart
status: 200
real 0m16.924s
user 0m1.484s
sys 0m1.156s
marcelo@MAQ04:~/d1$ time dart extract.dart
exitcode: 0
real 0m1.846s
user 0m1.281s
sys 0m1.156s
As you can see, the sample programs worked as expected, however pub fails:
marcelo@MAQ04:~/d1$ time pub global activate stagehand
Resolving dependencies... (4:01.2s)
Connection closed while receiving data
real 4m1.621s
user 0m0.484s
sys 0m0.781s
One thing that I have noticed in the past is that pub works if my computer is using almost all available internet bandwidth, like if I'm doing a big download with Firefox or if Windows is downloading updates, then pub on the WSL works.
Info about my environment:
marcelo@MAQ04:~/d1$ screenfetch
awk: fatal: cannot open file `/proc/fb' for reading (No such file or directory)
./+o+- marcelo@MAQ04
yyyyy- -yyyyyy+ OS: Ubuntu 18.04 bionic [Ubuntu on Windows 10]
://+//////-yyyyyyo Kernel: x86_64 Linux 4.4.0-17134-Microsoft
.++ .:/++++++/-.+sss/` Uptime: 31m
.:++o: /++++++++/:--:/- Packages: 498
o:+o+:++.`..```.-/oo+++++/ Shell: bash 4.4.19
.:+o:+o/. `+sssoo+/ CPU: Intel Core i5-4210U @ 4x 1.701GHz
.++/+:+oo+o:` /sssooo. GPU:
/+++//+:`oo+o /::--:. RAM: 3737MiB / 8095MiB
\+/+o+++`o++o ++////.
.++.o+++oo+:` /dddhhh.
.+.o+oo:. `oddhhhh+
\+.++o+o``-````.:ohdhhhhh+
`:o+++ `ohhhhhhhhyo++os:
.o:`.syhhhhhhh/.oo++o`
/osyyyyyyo++ooo+++/
````` +oo+++o\:
`oo++.
startdust10191
on 25 Feb 2019
Hmm, then it's not the way we use "tar" as a sub-process...
Maybe someone could look into what Platform.isWindows does... Maybe try to clone this repo and run the tests to debug it...
Like I mentioned before this would involve copying the pub-cache from a docker container after installing there... And probably the .packages too, and fixing paths in .packages, but it might reveal the issue..
jonasfj
on 25 Feb 2019
@jonasfj , I did as you suggested and was able to run pub tests in WSL, 95 tests failed.
Here is the complete log:
pubtest.zip
To explore this further, I have modified your download program to make it download 10 times in a row. My dart is a bit rusty, so maybe I could have introduced some bug in the code.
import 'dart:io';
final metaPackageUrl =
'https://storage.googleapis.com/pub-packages/packages/meta-1.1.7.tar.gz';
void main() async {
for(var i=0; i < 10; i++) {
var client = HttpClient();
var req = await client.getUrl(Uri.parse(metaPackageUrl));
var res = await req.close();
print((i+1).toString() + ' : status: ${res.statusCode}');
var data = <int>[];
await for (var chunk in res) {
data.addAll(chunk);
}
var name = 'output-'+(i+1).toString()+'.tar.gz';
await File(name).writeAsBytes(data);
}
}
This time I got an error:
marcelo@MAQ04:~/d1$ dart download-many.dart
1 : status: 200
2 : status: 200
3 : status: 200
4 : status: 200
5 : status: 200
6 : status: 200
7 : status: 200
8 : status: 200
Unhandled exception:
HttpException: Connection closed while receiving data, uri = https://storage.googleapis.com/pub-packages/packages/meta-1.1.7.tar.gz
#0 _HttpIncoming.listen.<anonymous closure> (dart:_http/http_impl.dart:161:7)
#1 _invokeErrorHandler (dart:async/async_error.dart:17:29)
#2 _HandleErrorStream._handleError (dart:async/stream_pipe.dart:286:9)
#3 _ForwardingStreamSubscription._handleError (dart:async/stream_pipe.dart:168:13)
#4 _RootZone.runBinaryGuarded (dart:async/zone.dart:1326:10)
#5 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:355:15)
#6 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:373:16)
#7 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:272:7)
#8 _SyncStreamController._sendError (dart:async/stream_controller.dart:768:19)
#9 _StreamController._addError (dart:async/stream_controller.dart:648:7)
#10 _StreamController.addError (dart:async/stream_controller.dart:600:5)
#11 _HttpParser._onDone (dart:_http/http_parser.dart:822:25)
#12 _RootZone.runGuarded (dart:async/zone.dart:1302:10)
#13 _BufferingStreamSubscription._sendDone.sendDone (dart:async/stream_impl.dart:389:13)
#14 _BufferingStreamSubscription._sendDone (dart:async/stream_impl.dart:399:15)
#15 _BufferingStreamSubscription._close (dart:async/stream_impl.dart:283:7)
#16 _SyncStreamController._sendDone (dart:async/stream_controller.dart:772:19)
#17 _StreamController._closeUnchecked (dart:async/stream_controller.dart:629:7)
#18 _StreamController.close (dart:async/stream_controller.dart:622:5)
#19 _Socket._onData (dart:io/runtime/binsocket_patch.dart:1781:21)
#20 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
#21 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#22 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7)
#23 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19)
#24 _StreamController._add (dart:async/stream_controller.dart:640:7)
#25 _StreamController.add (dart:async/stream_controller.dart:586:5)
#26 _RawSecureSocket._closeHandler (dart:io/secure_socket.dart:797:21)
#27 _RawSecureSocket._eventDispatcher (dart:io/secure_socket.dart:754:9)
#28 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
#29 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#30 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7)
#31 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19)
#32 _StreamController._add (dart:async/stream_controller.dart:640:7)
#33 _StreamController.add (dart:async/stream_controller.dart:586:5)
#34 new _RawSocket.<anonymous closure> (dart:io/runtime/binsocket_patch.dart:1330:35)
#35 _NativeSocket.issueReadEvent.issue (dart:io/runtime/binsocket_patch.dart:837:18)
#36 _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#37 _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#38 _runPendingImmediateCallback (dart:isolate/runtime/libisolate_patch.dart:115:13)
#39 _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:172:5)
I also decided to run the tests on Windows (not WSL), and 110 tests failed. However I never experienced problems with pub in Windows. Maybe I should open a new issue?
startdust10191
on 28 Feb 2019
@startdust10191, are you suggesting it might be an intermittent bug?
jonasfj
on 28 Feb 2019
@jonasfj , Like I said in my first message:
One thing that I have noticed in the past is that
pubworks if my computer is using almost all available internet bandwidth, like if I'm doing a big download with Firefox or if Windows is downloading updates, thenpubon the WSL works.
But the last time this I experienced this behavior was back on Dart 1.x line, on my home computer where I have only a 2Mb/s internet connection.
The fact that pub tests failed on Windows (non-WSL) was a surprise to me, since I have never experienced pub problems on Windows before.
startdust10191
on 1 Mar 2019
This sounds like a low-level bug in WSL or how it interacts with Dart-VM, it might be worth search github.com/dart-lang/sdk for a similar bug, or file on with a minimal test case that doesn't rely on network I/O if possible (or network over localhost, if possible).
jonasfj
on 4 Mar 2019
Related issues
DartBot
路
27Comments
sanjidtt
路
36Comments
samueladekunle
路
25Comments
Scorpiion
路
24Comments
jayoung-lee
路
38Comments
Most helpful comment
This problem pretty much makes using Dart under WSL impossible. Is no-one doing that?