Paket pack is unbearable slow when there are about hundred of projects to pack.
Paket version 4.8.7
We have a solution with a little more than 100 C# projects in it. Some projects are class libraries and a few are "leaf" projects - EXE and WebSites, which use those class libraries.
I want to use "paket pack" to package build output to many NuGet packages with inter-dependencies. Class libraries are usually consumed by multiple "leaf" projects, so I want NuGet packages for those "leaf" projects reference class libraries packages, instead of duplicating binaries in the build drop. For that purpose I placed tiny autogenerated paket.template file to each project folder.
When I run paket.exe pack output nugetFolder include-referenced-projects lock-dependencies it scans the folder tree in seconds and I see messages "No description was provided for package" for each project. But then it become absolutely quiet (even in verbose mode) and it takes at least half hour (actually I never waited to the end of it). Using Process Monitor I can see that Paket continuously read the same project and template files over and over again. One CPU core is 100% loaded. Nothing is written to the output folder.
What exactly it is doing for so long? Is there anything I can do to make it faster? This sort of kills the whole idea.
Thanks beforehand!
Konstantin
konste
All 20 comments
@mavnn I think it is your implementation. Any idea why so slow?
konste
on 9 Jun 2017
Probably only a profiler can tell
forki
on 9 Jun 2017
@konste Submit a massive dummy project to act as a perf test? Then people can work on it.
dsyme
on 9 Jun 2017
Hi @konst! I haven't worked on this code in literally years so I can't
really comment. I'm pretty certain that the code has been used on fairly
large projects in between so I would guess that there is something specific
about your use case which is triggering this bug. As @dsyme mentioned, a
reproduction might help someone track down the root cause.
On Fri, 9 Jun 2017, 13:51 Don Syme, notifications@github.com wrote:
@konste https://github.com/konste Submit a massive dummy project to act
as a perf test? Then people can work on it.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2403#issuecomment-307380012,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAS4QHUQxzZ5K8dqR4bbnw6cm6eWodLRks5sCT-2gaJpZM4N01wY
.
mavnn
on 9 Jun 2017
I got in a sticky situation here, because I badly need "paket pack" to work, but I don't know F# at all and cannot publish original big project as the repro. So what I could do is to profile it myself and when I got some idea I debugged the code and kind of understood what is going on, but I still need somebody with F# background to fix it.
Here is what I think happening:
Imagine solution with 100 projects. First project is independent, second depends on first, third depends on the first two, etc. Each next project depends on all previous.
Problematic code is in
Paket.Core\PaketConfigFiles\ProjectFile.fs, member this.GetAllReferencedProjects()
When it gets invoked for the project N, it reads in all N-1 dependencies and then recursively calls itself on each dependency. Then the process repeats itself for N-2, etc.
This means processing time is proportional to factorial of the number of projects! For 100 projects it is 1.0e158! No surprise I cannot get it to completion.
Instead each project scanned should cache the results of the scan in the hashtable with the project path as a key. Then each new project would only 1 (one) project scan as all previous projects are already scanned and results available. So when it is done right the algorithm should be linear, instead of N!
Who can make it happen, please?
Besides I also noticed that GetAllReferencedProjects is always invoked TWICE for each project, don't know why. That's on top of N! Would be nice to remove that duplication too.
Thank you for your attention!
konste
on 10 Jun 2017
Something failed in GetPackageDetails, trying again with force: Couldn't get package details for package FAKE 4.61.3 on https://www.nuget.org/api/v2.
Performance:
- Disk IO: 1 second
- Average Request Time: 1 second
- Number of Requests: 48
- Runtime: 1 minute, 10 seconds
Paket failed with:
Could not download FAKE 4.61.3.
Couldn't get package details for package FAKE 4.61.3 on https://www.nuget.org/api/v2.
vasily-kirichenko
on 10 Jun 2017
Also I cannot load most of the projects in Rider, see https://github.com/fsprojects/Paket/issues/2406
vasily-kirichenko
on 10 Jun 2017
will this suffice?
f#
member this.GetAllReferencedProjects() =
let dependencyHash = HashSet<string>()
let rec getProjects (project:ProjectFile) =
seq {
let projects = seq {
for proj in project.GetInterProjectDependencies() do
if not dependencyHash.Contains proj.Path then
let projFile = (ProjectFile.tryLoad(proj.Path).Value)
dependencyHash.Add proj.Path
yield projFile }
yield! projects
for proj in projects do
yield! (getProjects proj)
}
seq {
yield this
yield! getProjects this
}
Or should the progFile results be cached in a dictionary to still return the result without running the load?
Regarding GetAllReferencedProjects being called twice, PackageMetaData value allReferences calls GetAllReferencedProjects line 282, as well as it being called in 'ProjectFile.GetCompileItems' (GetProjects) so does appear to be duplicating work.
gerardtoconnor
on 10 Jun 2017
Could you please send a pull request for discussion? Thanks!
Am 10.06.2017 1:56 nachm. schrieb "gerardtoconnor" <[email protected]
:
will this suffice?
member this.GetAllReferencedProjects() =
let dependencyHash = HashSet()
let rec getProjects (project:ProjectFile) =
seq {
let projects = seq {
for proj in project.GetInterProjectDependencies() do
if not dependencyHash.Contains proj.Path then
let projFile = (ProjectFile.tryLoad(proj.Path).Value)
dependencyHash.Add proj.Path
yield projFile }
yield! projects
for proj in projects do
yield! (getProjects proj)
}
seq {
yield this
yield! getProjects this
}Gre
'member this.GetCompileItems' contains two calls to GetAllReferencedProjects
```f#
this.GetProjects includeReferencedProjects // calls GetAllReferencedProjects|> this.ProjectsWithoutTemplates // calls GetAllReferencedProjects—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2403#issuecomment-307560841,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNN-03t9ZtM8JNVgblbUFWXU8aW1Rks5sCoR8gaJpZM4N01wY
.
forki
on 10 Jun 2017
PR added #2409
This does not fix the issue of GetAllReferencedProjects being called twice, PackageMetaData & ProjectFile.GetComplileItems. I do not know how/why the PageageMetaData is being called but it might be worth caching result of GetAllReferencedProjects if both continue to need seq independently.
gerardtoconnor
on 10 Jun 2017
@konste I have updated the GetAllReferences function in my fork with caching and no repeated calls, if you would like to clone, build and test, let me know if any speed up.
gerardtoconnor
on 10 Jun 2017
@gerardtoconnor Thank you! I will sure try it later today, when I get to the right computer.
konste
on 10 Jun 2017
@gerardtoconnor, when I try to compile your branch I get one error:
1>C:\GitHub\Paket-master\src\Paket.Core\PaketConfigFiles\ProjectFile.fs(1722,63): error FS0039: The field, constructor or member 'Contains' is not defined. Maybe you want one of the following: ContainsKey ContainsValue
and about ten warnings like warning FS0044: This construct is deprecated.
Does it compile for you? I use VS 2017.
konste
on 11 Jun 2017
@konste Apologies, I pushed a "tweak with error", but pushed fix few mins ago, if you pull & build again should be fine
gerardtoconnor
on 11 Jun 2017
Congratulations @gerardtoconnor! With the new implementation execution time reduced from eternity to 11 seconds, which is remarkable and definitely more usable! I'm looking forward seeing it released.
One other thing about project scanning process is still not clear. It seems Paket scans folder tree for all files named like ".proj" and tries to parse them as projects. In most cases they are indeed projects, but some of them may be very much irrelevant to the currently packed solution and not really referenced from anywhere. In one case it picked up .sqlproj file and stopped because corresponding DLL was not found. But that project does not produce DLL (it produces .dacpac). I wonder how hard would it be to instill some more smartness to Paket and make it not read projects which do not belong to closure of references.
konste
on 11 Jun 2017
Cool good work!
I think we already have a whitelist somewhere. I think Vb, f#, c#, c++ was
whitelisted
Am 11.06.2017 6:10 vorm. schrieb "Konstantin Erman" <
[email protected]>:
Congratulations @gerardtoconnor https://github.com/gerardtoconnor! With
the new implementation execution time reduced from eternity to 11 seconds,
which is remarkable and definitely more usable! I'm looking forward seeing
it released.One other thing about project scanning process is still not clear. It
seems Paket scans folder tree for all files named like ".proj" and tries
to parse them as projects. In most cases they are indeed projects, but some
of them may be very much irrelevant to the currently packed solution and
not really referenced from anywhere. In one case it picked up .sqlproj file
and stopped because corresponding DLL was not found. But that project does
not produce DLL (it produces .dacpac). I wonder how hard would it be to
instill some more smartness to Paket and make it not read projects which do
not belong to closure of references.—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2403#issuecomment-307605092,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNNeEgs8hnFado1r4Pl-cg9vQMRWHks5sC2iegaJpZM4N01wY
.
forki
on 11 Jun 2017
There should be Nemerle as well.
vasily-kirichenko
on 11 Jun 2017
I think we already have that in that filter. But don't know where that one is right now
forki
on 11 Jun 2017
vasily-kirichenko
on 11 Jun 2017
fixed in rc005
forki
on 11 Jun 2017
Related issues
ModernRonin
·
7Comments
Zaid-Ajaj
·
3Comments
NatElkins
·
6Comments
0x53A
·
8Comments
pshrosbree
·
3Comments
Most helpful comment
I got in a sticky situation here, because I badly need "paket pack" to work, but I don't know F# at all and cannot publish original big project as the repro. So what I could do is to profile it myself and when I got some idea I debugged the code and kind of understood what is going on, but I still need somebody with F# background to fix it.
Here is what I think happening:
Imagine solution with 100 projects. First project is independent, second depends on first, third depends on the first two, etc. Each next project depends on all previous.
Problematic code is in
Paket.Core\PaketConfigFiles\ProjectFile.fs, member this.GetAllReferencedProjects()When it gets invoked for the project N, it reads in all N-1 dependencies and then recursively calls itself on each dependency. Then the process repeats itself for N-2, etc.
This means processing time is proportional to factorial of the number of projects! For 100 projects it is 1.0e158! No surprise I cannot get it to completion.
Instead each project scanned should cache the results of the scan in the hashtable with the project path as a key. Then each new project would only 1 (one) project scan as all previous projects are already scanned and results available. So when it is done right the algorithm should be linear, instead of N!
Who can make it happen, please?
Besides I also noticed that GetAllReferencedProjects is always invoked TWICE for each project, don't know why. That's on top of N! Would be nice to remove that duplication too.
Thank you for your attention!