Fsharp: build.cmd hangs forever on restoring packages behind a proxy

[vasily-kirichenko]

Restoring BuildTools version 1.0.25-prerelease-00404-01...
Initializing BuildTools ...

^C^ 

restore "E:\github\visualfsharp\packages\Microsoft.DotNet.BuildTools\1.0.25-prerelease-00404-01\lib\\tool-runtime\project.json" --source https://dotnet.myget.oCrg/F/dotnet-core/api/v3/index.json --source https://dotnet.myget.org/F/dotnet-buildtools/api/v3/index.json --source https://api.nuget.org/v3/index.json was unexpected at this time.

[forki]

not sure what you mean with forever, but even if you run that step without proxy issues it takes ages. much longer than compiling proto

[vasily-kirichenko]

I waited for about 10 minutes. No any progress. Finally Ctrl+C it.

[dsyme]

We really need a developer path that doesn't require CoreCLR, especially for those working on the Visual F# IDE Tools (which don't use CoreCLR). Or we need to split the Visual F# IDE tools into their own repository so the dependencies are clear.

[vasily-kirichenko]

Trying to run build.cmd at home, it downloads like crazy with no visible feedback. I'm for a separate path, which does not need to download a half of the Internet.

[forki]

Remember my blog posts? And there are 2 or 3 new episodes of game of
thrones available by now.
On May 18, 2016 7:17 PM, "Vasily Kirichenko" [email protected]
wrote:

Trying to run build.cmd at home, it downloads like crazy with no visible
feedback. I'm for a separate path, which does not need to download a half
of the Internet.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/visualfsharp/issues/1202#issuecomment-220096823

[smoothdeveloper]

@dsyme can someone at MS also push up the concern to dotnet/cli that "dotnet restore" downloading the basic tools is obviously not doing things in an effective way, printing so much useless output, and I don't know how many thousands of concurrent web requests (which almost all fail on a slow connection).

For those wanting a separate path, to build proto / dotnet40, my FAKE script can do that job, I'll push this effort forward to be able to run the tests as well, this seems to be most reasonable direction for multiplatform (and F#) build script.

[dsyme]

@smoothdeveloper Either add a dotnet/cli GitHub issue about that - or point us to an existing issue we can comment on.

[smoothdeveloper]

@dsyme I created this issue which I hope is comprehensive:

https://github.com/dotnet/cli/issues/3123

In this case, dotnet restore just stalled for no apparent reason.

As Microsoft is refocusing on the core value of dotnet core (languages, standard libraries and runtime foremost, engaging and driving with community for the rest), it would be great to piggy back on "level A" community projects to extend them for the purpose of our future platform.

What I'm seeing is that nuget client does a poor job at restoring stuff for this repository, while paket does the job without any issue, speaking from end user perspective, I'm sure if business people at Microsoft were presented with output and outcome of both commands as I experienced it (and I'm not alone), they'd favor investing in paket current rather than nuget client vnext.

[KevinRansom]

I never see this behavior you guys seem to see. I delete packages, tools and proto all of the time and build never takes ages. I do this at home a lot and I have a bog standard American cable connection. I am probably going to be making things worse for you over the next few weeks.

dotnet/cli uses a super chatty protocol dependency protocol unfortunately, and we do download publish dependencies for all platforms which is kind of unfortunate.

Also we don't update our dependencies that frequently, and so this should be a one time infrequent cost, even when you do see it.

What do I need to do to be able to reproduce the symptoms you guys see?

Kevin

[forki]

Maybe you need to leave Redmond network locality to reproduce ;-)
On May 20, 2016 8:10 PM, "Kevin Ransom (msft)" [email protected]
wrote:

I never see this behavior you guys seem to see. I delete packages, tools
and proto all of the time and build never takes ages. I do this at home a
lot and I have a bog standard American cable connection. I am probably
going to be making things worse for you over the next few weeks.

dotnet/cli uses a super chatty protocol dependency protocol unfortunately,
and we do download publish dependencies for all platforms which is kind of
unfortunate.

Also we don't update our dependencies that frequently, and so this should
be a one time infrequent cost, even when you do see it.

What do I need to do to be able to reproduce the symptoms you guys see?

Kevin

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/visualfsharp/issues/1202#issuecomment-220678852

[smoothdeveloper]

@KevinRansom, thanks for sharing insight on your experience and what's coming.

In my case, I'm on close to slowest ADSL possible (fiber will come in 1 year) at farthest distance to DSLAM to actually have ADSL.

Eventually, running the command several times, I'll get the tool downloaded, I can also remote to another more fortunate network location, run it there, zip a folder and download that in a single HTTP session which works very well.

The overal concerns (from my perspective, with varying degree of importance):

• the standard output of init-tools.cmd seems to have passed 0 level of QA in terms of end user experience, this has confused not only me on slow connection, but all others I've seen opening tickets and digging into their first experience of .net core (we know it's a big deal to bootstrap all of that stuff since anouncement of .NET == Open Source :smile:)
• for the purpose of repeated workflow impacting all new contributors to visualfsharp, streamlining the experience is a huge deal, being it by making better tools upstream (dotnet cli / nuget client?), and thinking a bit out of the box (why everyone has to run thousands of HTTP requests to download 40mb of packages) rather than just taking "init-tools" as it is, MS can probably setup a "dev-tools.zip" release in this repository (paket has something similar with paket.boostrapper.exe, with outstanding experience compared to "init-tools"), from outside, it seems that MS just carry on business (in Redmond) as usual but it is primary concerns to streamline as much as possible experience of external contributors when you go open source
• for the purpose of making visualfsharp repo "xplat ready" we should think outside the box too and not bet too much on "outstanding V1.0 experience" on the dotnet core tooling, especially in the context of F# community; we can rely on mono as a dependency for building this repository on mac/linux, and doing that start using tools which makes the community and most MS effective (for me that mostly means no more packages.config in this repository => paket, and removing shell scripts => fake) and setup our cross platform targetting on those proven tools until dotnet core stabilizes and provides experience on par with what we know (we don't have to dogfood everything MS is coming up with, we just want to make F# the greatest/greater), all F# users are fluent with .fsx, FAKE and paket, more than dotnet core.

What do I need to do to be able to reproduce the symptoms you guys see?

Apparently, from dotnet/cli#3123:

• have VMWare or Virtual PC installed with "virtual" network adapters
• run windows 7 64
• slow connection (but @vasily-kirichenko created this ticket on fast connection)
• tend to be scared with output of dotnet restore (sorry I'm used to paket output and seeing it work well to restore large graph of dependencies, even if sometimes there might be a small bug, this will generally get addressed and answered quickly) and dismiss hope that it will eventually restore

Actually after a long while (I left the process going) it eventually completed not sure after how long, but in the meantime I tried and ran paket succesfully and made the bug report.

Also we don't update our dependencies that frequently

a single "dev-tools.zip" hosted in the release section of this repository and a simple .fsx script to retrieve it / extract it will save kilo-hours, even if we update those only once a month (but I think it will be more frequent until release of VS15).

Sorry if all of this is a bit verbose, I hope this helps more than anything else :smile:

[smoothdeveloper]

@forki

Remember my blog posts? And there are 2 or 3 new episodes of game of
thrones available by now.

I think you are "misguided" :smile: , it's just the matrix effect of your console, printing the internet, not the actual download.

For users accustomed to "semantic" console output by default in other languages ecosystems, that comes with some amount of surprise and unexpected reactions.

[latkin]

To repro one could try Clumsy, which can simulate network latency/dropped packets/etc. Very simple UI and in my experience just works.

[smoothdeveloper]

I don't need it but thanks for very great reference, useful to all others!

[isaacabraham]

@smoothdeveloper we've discussed Fake several times over the past year or two - I believe someone even started the work on that - but my understanding is that there's no desire to move to Fake internally, and so this is at a dead end (or perhaps roadworks having closed the road is a better analogy as they can be cleared up eventually!).

[KevinRansom]

Right now my focus is to ensure that the F# compiler works correctly on coreclr and cross platform. A consequence of that is we pay a price when we run on an OS with all of the right sdks already installed. However I can add a windows only build switch for developers who are interested in only using windows and the desktop CLR. Since it won't be for CI we can even eliminate the proto build.

[vasily-kirichenko]

I'm not interested in building the compiler anymore, closing the issue.

[isaacabraham]

Unfortunately I'm in the same boat as @vasily-kirichenko in that I've tried on three separate occasions since F#4 was coming out, and every time it was a painful experience compared to what I've been used to with other F# OSS projects, whether it was having to fight with Perl scripts, batch files or dependency issues.

Clearly some people are getting it to work outside of MS - @forki being a great example - and we shouldn't forget that, but it's a pity that the compiler itself which IMHO should be a flagship project for F# development practices and tools etc. is still causing problems to developers that _want_ to get involved.

[dsyme]

@vasily-kirichenko I'll reopen since this remains a real issue, if that's ok, thanks.

[enricosada]

i'll send the pr #1221 to https://github.com/dotnet/corefx so we can import later the improvement.

The pr check if .net core sdk of required version is already installed in PATH, if installed ok, otherwise download
We should pin to use .NET Core SDK Preview 1 installed like the other deps ( one time install for dev from http://dot.net, on appveyor is already installed in the build worker image ).

About chatty restore, #1222 should remove a bit of restore (the v2 nuget is duplicated and not needed because there is already the faster and less chatty v3)

after both pr are merged, in my machine (italy) build it's ok, because after the first restore all deps are cached in ~/.nuget/packages and restore it's just ask nuget server for version number, packages are not downloaded.

we can also improve using the same version of .net deps, to avoid multiple packages downloads. the released .net core rc2 works ok xplat (osx/win/ubuntu/docker) but that's another issue.

another are of improvement is Initializing BuildTools .. (local publish take time, it's not internet in my case) but that's also in the https://github.com/dotnet/corefx repository. The proxy issue too.
The bad thing is there is no output or progress, the output is redirected to a log file, so seem stuck ( i ctrl-C the first times too)

[ghost]

Its more fun when we do dotnet restore --no-cache.

[ghost]

Here we go https://github.com/dotnet/cli/pull/3229 by @billwert. And that's merged!

[billwert]

Hopefully it helps! Let us know.

[dsyme]

@smoothdeveloper Thanks for taking the initiative on addressing this at the root cause of the problem.