Dart-code: Searching/listing all symbols is very slow

[xster]

Maybe add some form of caching to the analyzer query (maybe trigger on file saves or something) to speed up the symbol lookups.

[DanTup]

Are you able to give more details/explicit examples of what seems to be slow? How big is the project? By "symbol lookups" do you mean "Go to definition"? Are you invoking this on from within files in your own project or outside of the opened workspace folder (eg. SDK files)? Which classes are you invoking it for (again, classes in your own project or in the SDK?).

[xster]

Hi,

I'm using it on the Flutter project https://github.com/flutter/flutter
I'm using workbench.action.showAllSymbols with the repo open as the workspace. May be worth excluding gitignored files in the workspace.

Though workbench.action.gotoSymbol takes a while on small projects as well.

[DanTup]

Thanks for the info!

@bwilkerson Currently we're sending searches for .*.* and it's taking a long time (40 secs) for the results:

[10:55:16]: ==> {"id":"3","method":"search.findTopLevelDeclarations","params":{"pattern":".*.*"}}
[10:55:16]: ==> {"id":"4","method":"search.findMemberDeclarations","params":{"name":".*.*"}}
[10:55:22]: <== {"id":"3","result":{"id":"0"}}
[10:55:22]: <== {"id":"4","result":{"id":"1"}}
[10:56:05]: <== {"event":"search.results", /* SNIP */
[10:56:07]: <== {"event":"search.results","params":{"id":"1","results":[],"isLast":true}}

I guess I might be using findMemberDeclarations wrong since it returned no results, but am I using findTopLevelDeclerations wrong or is it expected that it could be this slow?

I could add some caching into Dart Code but I'd have to invalidate it so frequently I'm not sure if it'd be that useful. Any suggestions?

There was a discussion about this functionality some time ago in analyzer-discuss:

https://groups.google.com/a/dartlang.org/forum/#!searchin/analyzer-discuss/findTopLevelDeclarations/analyzer-discuss/conkKX7XiKI/c_2b1ggyCAAJ

I just noticed you said:

Calling both and merging the results seems like a reasonable short-term work-around, but it seems like we ought to provide a more convenient and performant API.

So maybe it's known this wouldn't be so fast! I guess there isn't yet a better way?

[bwilkerson]

No, we haven't added any new search APIs yet. Having a single request could potentially be faster because we'd only be searching each compilation unit one time rather than twice, but we'd have to do some measurements to know for sure.

We should look into the performance of the existing APIs, but adding a new API probably won't happen for a while. How big is the code base being searched? (One measure might be the number of results returned.)

[DanTup]

How big is the code base being searched?

Er, looks like there's about 75MB of JSON being returned from findTopLevelDeclarations... (it's the flutter repo btw).

I wonder if something is going wrong, let me dig a little more and get some better info!

[DanTup]

Ok, I tried to run this on a tiny project and it was still very slow and produced a 1.5MB response. I looked at the results, and it includes things from the SDK:

    {
        "location": {
            "file": "M:\\Apps\\Dart-1.23.0-dev.11.11\\lib\\internal\\internal.dart",
            "offset": 2249,
            "length": 13,
            "startLine": 77,
            "startColumn": 5
        },
        "kind": "DECLARATION",
        "isPotential": false,
        "path": [ /*SNIP*/]
    }

I wouldn't have expected this (also, the name hasn't come through?), I only expected things from the users project.

@bwilkerson Do you know exactly where findTopLevelDeclarations should be searching? Should it include SDK files? How about referenced packages? And should any results not have names?

[DanTup]

@bwilkerson I just tested with an older SDK, and it returns only a tiny number of items. I think the behaviour has changed (since 1.19), I'm not sure if it was intended. Shall I raise an issue at https://github.com/dart-lang/sdk ?

[bwilkerson]

Yes, please. It might be related to the change to the new analysis driver. And thanks for digging into this!

[DanTup]

@bwilkerson Done at https://github.com/dart-lang/sdk/issues/29510. Let me know if you need any more info!

[DanTup]

This is progressing. Once the CL from https://github.com/dart-lang/sdk/issues/29510#issuecomment-367191212 lands I'll do some testing of it. If it works well we can bump the server version and start using that instead of the old mechanism for SDKs that support it.

[DanTup]

You'll be happy to know that @scheglov has implemented a new search API that is incredibly fast! The next version of Dart Code will switch to it if it detects that the analysis server supports it. There will no doubt publish a beta release (or 7) before it goes out so if you're interested in helping test it keep an eye on Gitter/Twitter in a few weeks time.