Atom: Shebang/modeline recognition

[ccorcos]

Hey, some scripts of mine, I like to leave off the extension so I can run them more naturally as executables. So it would be really cool if you could recognize the shebang's for the filetype!

[DanBrooker]

I don't think this would be very practical, loading and checking files would be slow.

[ccorcos]

even just the first line?

[Goodwine]

I can see this being problematic when you have tons of files, and atom is already slow on those cases, perhaps having it as an opt-in option that would check the first line of a file without extension with a total size Also being able to have a cache for that would be helpful, assuming the first line would never change, so IDK :v

[michael-yx-wu]

Atom must store the language syntax highlighting somewhere. Is it possible to fetch that metadata about files without extensions? If so, you might be able to save yourself the trouble of needing to open the file itself.

[Alhadis]

I've been researching this already. Atom doesn't attempt to parse a file's grammar until it's opened, which makes sense from a performance point-of-view. As Goodwine said, processing a project with tens of thousands of files would drain system resources, and would eat up a potentially large amount of disk space if such data were to be cached (recall that Atom would need to map each examined grammar to a file based on its SHA hash, etc).

Theoretically, it _might_ be possible for the package to asynchronously parse the headers of each file as a background process, and only for files that're visible in the tree view (with the results being cached, obviously). The easiest solution would be to identify a file's language when it's opened (as the hashbang also identifies the file's grammar to Atom).

[ccorcos]

Is there really no way to efficiently parse only the first line or 100 characters form a file?

[Alhadis]

@ccorcos We can easily scan the first line of the _currently-opened_ file, that's no problem.

Scanning the lines of _every single file_ in an opened project is definitely a no-go, because we can't possibly predict how many files will need to be processed. Even for systems with heavy-duty CPUs, there'd still be a latency issue - default icons would appear for a split second between opening the window and finishing the filesystem scan.

If we limit scanning to the currently-opened file, it means the correct icon won't be displayed until the user's opened the file. I don't know how disconcerting this would be for users, though. We can always cache the result so the icons appear the next time the project's opened... but that's only solving half the problem...

[Alhadis]

We should probably implement this with support for Vim/Emacs modelines too:

Lines like these are the "traditional" method of identifying a file's language to an editor: it makes sense to leverage them. Question is, do we avoid checking files that already have an icon (which is faster) or check every one (which is slower)? Bear in mind we'd only be scanning files in directories as they're being opened, and caching the data somewhere in Atom's user-cache.

I'm just trying to look at the most performant way possible of implementing this, as while it's clearly a helpful/important addition, it has the potential to inflict a lot of lag on unsuspecting users with densely-populated projects.

[Alhadis]

@DanBrooker I'm going to try and implement this for v2.0.0, but I'll drop it if it proves to add even a split second of delay with large projects (yes I'm an overly-cautious prick, sue me).

[ccorcos]

what if you only scanned files that are in the tree view?

[Alhadis]

Well, that's what I'm suggesting.

[michael-yx-wu]

There might still be a large number of files though. For example, some projects check in their node modules.

Michael

On May 4, 2016, at 6:40 PM, John Gardner [email protected] wrote:

Well, that's what I'm suggesting.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

[Alhadis]

Yes, but we're still limiting the search only to those files that're visible. For instance, if the workspace was opened with this in the tree-view, only five files would be checked:

So the closed folders wouldn't be searched, and scanning of huge folders wouldn't start until the user scrolled.

[Alhadis]

Guys, I'm closing this in favour of a dedicated issue to track all V2-related progress. I recommend you subscribe there for updates.