Python-language-server: Analysis of peewee with the new LS never finishes

[MikhailArkhipov]

https://github.com/Microsoft/vscode-python/issues/2682

_"Check out peewee. When the new LS tries to analyse the main file, peewee.py, it uses >100% of CPU and the RAM usage increases continuously (over 40GB at some point). This doesn't seem to complete, even after leaving it open and running for an hour. Things like the Go to symbol list never get rendered."_

[MikhailArkhipov]

Looks like #173

Assertion Failed
Hash code changed when merging AnalysisValues

   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet.AddOne(BucketSet& buckets, AnalysisValue key, Int32 hc, IEqualityComparer`1 comparer) in F:\PLS\src\Analysis\Engine\Impl\AnalysisHashSet.cs:line 425
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet.AddOne(BucketSet& buckets, AnalysisValue key, IEqualityComparer`1 comparer) in F:\PLS\src\Analysis\Engine\Impl\AnalysisHashSet.cs:line 329
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet.AddFromEnumerator(BucketSet& buckets, IEnumerator`1 items, IEqualityComparer`1 comparer) in F:\PLS\src\Analysis\Engine\Impl\AnalysisHashSet.cs:line 240
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet.AddFromEnumerator(IEnumerator`1 items) in F:\PLS\src\Analysis\Engine\Impl\AnalysisHashSet.cs:line 232
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet..ctor(IEnumerable`1 enumerable, IEqualityComparer`1 comparer) in F:\PLS\src\Analysis\Engine\Impl\AnalysisHashSet.cs:line 85
   at Microsoft.PythonTools.Analysis.AnalysisSet.AsUnion(IAnalysisSet set, UnionComparer comparer, Boolean& wasChanged) in F:\PLS\src\Analysis\Engine\Impl\AnalysisSet.cs:line 288
   at Microsoft.PythonTools.Analysis.AnalysisSet.AsUnion(IAnalysisSet set, Int32 strength, Boolean& wasChanged) in F:\PLS\src\Analysis\Engine\Impl\AnalysisSet.cs:line 247
   at Microsoft.PythonTools.Analysis.Analyzer.TypedDependencyInfo.MakeUnion(Int32 strength) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\DependencyInfo.cs:line 130
   at Microsoft.PythonTools.Analysis.TypedDef`1.MakeUnion(Int32 strength) in F:\PLS\src\Analysis\Engine\Impl\VariableDef.cs:line 415
   at Microsoft.PythonTools.Analysis.TypedDef`1.MakeUnionStronger() in F:\PLS\src\Analysis\Engine\Impl\VariableDef.cs:line 406
   at Microsoft.PythonTools.Analysis.TypedDef`1.MakeUnionStrongerIfMoreThan(Int32 typeCount, IAnalysisSet extraTypes) in F:\PLS\src\Analysis\Engine\Impl\VariableDef.cs:line 398
   at Microsoft.PythonTools.Analysis.Analyzer.FunctionScope.UpdateParameters(FunctionAnalysisUnit unit, ArgumentSet others, Boolean enqueue, FunctionScope scopeWithDefaultParameters, Boolean usePlaceholders) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\FunctionScope.cs:line 198
   at Microsoft.PythonTools.Analysis.Analyzer.FunctionAnalysisUnit.UpdateParameters(ArgumentSet callArgs, Boolean enqueue) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\FunctionAnalysisUnit.cs:line 64
   at Microsoft.PythonTools.Analysis.Values.FunctionInfo.DoCall(Node node, AnalysisUnit callingUnit, FunctionAnalysisUnit calledUnit, ArgumentSet callArgs) in F:\PLS\src\Analysis\Engine\Impl\Values\FunctionInfo.cs:line 140
   at Microsoft.PythonTools.Analysis.Values.FunctionInfo.Call(Node node, AnalysisUnit unit, IAnalysisSet[] args, NameExpression[] keywordArgNames) in F:\PLS\src\Analysis\Engine\Impl\Values\FunctionInfo.cs:line 96
   at Microsoft.PythonTools.Analysis.Values.ClassInfo.AddCall(Node node, NameExpression[] keywordArgNames, AnalysisUnit unit, IAnalysisSet[] args) in F:\PLS\src\Analysis\Engine\Impl\Values\ClassInfo.cs:line 66
   at Microsoft.PythonTools.Analysis.Values.ClassInfo.Call(Node node, AnalysisUnit unit, IAnalysisSet[] args, NameExpression[] keywordArgNames) in F:\PLS\src\Analysis\Engine\Impl\Values\ClassInfo.cs:line 56
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateCall(ExpressionEvaluator ee, Node node) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\ExpressionEvaluator.cs:line 382
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateWorker(Node node) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\ExpressionEvaluator.cs:line 214
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.Evaluate(Expression node) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\ExpressionEvaluator.cs:line 54
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(ReturnStatement node) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\DDG.cs:line 578
   at Microsoft.PythonTools.Parsing.Ast.ReturnStatement.Walk(PythonWalker walker) in F:\PLS\src\Analysis\Engine\Impl\Parsing\Ast\ReturnStatement.cs:line 32
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(SuiteStatement node) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\DDG.cs:line 662
   at Microsoft.PythonTools.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker) in F:\PLS\src\Analysis\Engine\Impl\Parsing\Ast\SuiteStatement.cs:line 34
   at Microsoft.PythonTools.Analysis.Analyzer.FunctionAnalysisUnit.AnalyzeWorker(DDG ddg, CancellationToken cancel) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\FunctionAnalysisUnit.cs:line 89
   at Microsoft.PythonTools.Analysis.AnalysisUnit.Analyze(DDG ddg, CancellationToken cancel) in F:\PLS\src\Analysis\Engine\Impl\AnalysisUnit.cs:line 139
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Analyze(Deque`1 queue, CancellationToken cancel, Action`1 reportQueueSize, Int32 reportQueueInterval) in F:\PLS\src\Analysis\Engine\Impl\Analyzer\DDG.cs:line 75
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.AnalyzeQueuedEntries(CancellationToken cancel) in F:\PLS\src\Analysis\Engine\Impl\PythonAnalyzer.cs:line 925
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.GroupAnalysis.Analyze(CancellationToken cancel) in F:\PLS\src\Analysis\Engine\Impl\Intellisense\AnalysisQueue.cs:line 224
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.HandleAnalyzable(IAnalyzable item, AnalysisPriority priority, CancellationToken cancellationToken) in F:\PLS\src\Analysis\Engine\Impl\Intellisense\AnalysisQueue.cs:line 148
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.HandleAnalyzable(IAnalyzable item, AnalysisPriority priority, CancellationToken cancellationToken)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.<>c__DisplayClass28_0.<Enqueue>b__0(CancellationToken ct) in F:\PLS\src\Analysis\Engine\Impl\Intellisense\AnalysisQueue.cs:line 132
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ConsumerLoop() in F:\PLS\src\Analysis\Engine\Impl\Intellisense\AnalysisQueue.cs:line 60
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ConsumeAsync() in F:\PLS\src\Analysis\Engine\Impl\Intellisense\AnalysisQueue.cs:line 85
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

[MikhailArkhipov]

https://github.com/Microsoft/python-language-server/pull/201

[stessaris]

I still have the RAM problem on OSX with Version 2018.9.0

[MikhailArkhipov]

@stessaris - how big is the RAM consumption? Runaway or large but stable? What is the project size? Is it better with Jedi?

[stessaris]

@MikhailArkhipov - over 5 GB (enough to bring my 8GB iMac to its knees with the rest of applications running), I got the feeling that is not growing a lot after that but I'll do some tests. The project is 20 files for a total of 6500 lines of code. No issues with Jedi

[MikhailArkhipov]

@stessaris - This bug is specifically about peewee, could you open another specific to your project please.

[GinRyan]

It's still happening in language-server.0.1.60 (Platform: on Windows 10 1803, VSCode 1.28.2, Python 3.6.4)

There are just 5 python scripts in my project.

After I killed this process, it launched again and auto increment to 4GB even 10GB .Auto increment step by 10MB per sec.

I just import numpy and matplotlib in this project.

[jakebailey]

@GinRyan As previously mentioned, this issue is specifically about peewee. If this is occurring for another project, I'd appreciate it if you opened a new issue (especially if you can share the project that causes the high memory usage so we can profile it).

[MikhailArkhipov]

0.2.16 is now available on beta channel.

"python.analysis.downloadChannel": "beta"

[NiNJAD3vel0per]

@MikhailArkhipov
What is this ? ""python.analysis.downloadChannel": "beta""
Why we should put this odds line in setting.json without explanation ?
My Mac is always +300% whenever turning of vscode with Python. This is not good !