Julia: Code loading might be better just fully parsing TOML files

[KristofferC]

Right now, code loading uses some regexes that can parse the Project files in the format that Pkg writes them.
This has a few negative side effects:

• We do not actually support TOML file, only the subset that the specialized regexes manage to parse.
• There is no way to cache any results, instead, for everything getting loaded we start parsing from scratch.
• Giving good error messages seems hard. After we found that for some reason we cannot load a package we cannot look up things that we have already parsed, since we have thrown away all the parsed information.
• The code is kinda hard to read when parsing of TOML files is blended together with the code loading logic. There are currently ~6 functions that parse TOML files in different ways depending on what step in code loading we are doing.

I would argue that just parsing the full TOML files, cache the results and use that in code loading would lead to faster and more simple code. The drawback is that we need a full TOML parser ni Base, but since we have decided to use TOML for code loading that seems inevitable, should have called it some other file extension if we didn't intend to support TOML fully.

[StefanKarpinski]

They are valid TOML files so using the .toml extension is not wrong, we just only support a certain subset of the format in code loading. I agree that this needs to be more general, however.

[KristofferC]

Running using Plots makes the Project.toml file be opened and parsed (although not fully parsed) 535 times and the Manifest.toml file 81 times which seems kinda silly.

[JeffBezanson]

:astonished:

[vtjnash]

You might be interested in https://github.com/JuliaLang/julia/compare/jn/loading-plus, which reimplements the code to process and use these files into two separate steps. Needs some thought on how to integrate it—right now it's just all of the low level support and approximating the current API—and also since parsing them 535 times is currently defined to be a feature (because the Project can be edited half way through loading something).

[KristofferC]

(because the Project can be edited half way through loading something).

Yeah but I don't think we need to support that.

[StefanKarpinski]

Running using Plots makes the Project.toml file be opened and parsed (although not fully parsed) 535 times and the Manifest.toml file 81 times which seems kinda silly.

Wow, that is nuts. It seems like it would be much better to parse once and cache the result.