Julia: C code needs proper documentation

[plokmijnuhby]

It would appear that the functions in C code marked with JL_DLLEXPORT are intended to be considered a public interface, but the overwhelming majority of them do not appear to have any clear documentation. Could this be changed?

[jpsamaroo]

I think so! Please feel free to open PRs adding documentation for these functions either inline or (probably better) in the devdocs section of the manual.

[StefanKarpinski]

To be clear, while the C API is exported thus technically a public interface, it is neither public nor stable in the same way that the language's API itself is. Unlike CPython, it's unnecessary to use Julia's C API to accomplish any programming tasks; the only case where using Julia's C API is really necessary is if you want to embed Julia. So documenting this is a low priority since we don't want people to use this API if they can avoid it. What is it that you'd like to accomplish?

[JeffBezanson]

Many of those functions are marked JL_DLLEXPORT not because they are public but just to make them accessible from other internal code written in julia (which looks up C functions via dynamic symbols).

[plokmijnuhby]

The thing _I_ wanted to accomplish was that I was trying to write the equivalent of fuckit.js, and as far as I can tell, you can't dynamically parse code in Julia without writing it to a file (which is a bit slow) or doing a ccall to jl_parse_all. (Meta.parse only does single expressions - turns out, that's not good enough.) I realise this is a slightly ...niche... usage of Julia though.
The one thing that did make me think these were public is that jl_parse_input_line only exists "for backwards compat", which suggests some effort is being made towards backwards compatibility in some cases. I think it is important to have some documentation indicating, at the very least, which functions are considered a stable interface and which aren't, otherwise embedding Julia in a forward compatible manner is basically impossible.

[JeffBezanson]

You're in luck, parseall is on the way (#34715).

Meta.parse only does single expressions - turns out, that's not good enough.

Call it repeatedly?

which functions are considered a stable interface and which aren't

Everything in jlapi.c is stable and public.

We will often leave things in for compatibility since people do call non-public functions. If it's easy to avoid breaking their code I'll do it.

[plokmijnuhby]

You're in luck, parseall is on the way (#34715).

Alright, so that was a bad example. But there are many things that can't be done in pure Julia. Want to set breakpoints and debug calls, for instance? You can't do that with pure julia.

Call it repeatedly?

No. Even if you use raise = false, if there's any syntax error of any kind (not just if there's more than one expression, if there's any kind of error at all) it only tells you what the error was. It doesn't tell you what the expression before the error was, and it doesn't tell you what line the error occurred. "There's an error somewhere" is not helpful enough. And greedy = false is definitely wrong.

Everything in jlapi.c is stable and public.

We will often leave things in for compatibility since people do call non-public functions. If it's easy to avoid breaking their code I'll do it.

Then that information needs to be in the docs. Even if we don't add anything else as a result of this issue, those two lines (or words to that effect) need to be added in somewhere.

[Keno]

Everything in jlapi.c is stable and public.

To be clear, it's stable in practice, but I think we exempted it from our stability guidelines wrt semantic versioning. Us not breaking it is more of a courtesy than a guarantee. Just by the nature of things people will depend on internal APIs. We spot that kind of thing using PkgEval and will either leave it in for compatibility or update the packages.

[JeffBezanson]

But there are many things that can't be done in pure Julia. Want to set breakpoints and debug calls, for instance? You can't do that with pure julia.

Debugger.jl? But I'm not sure how that is related. I don't think it's accurate to say that we have loads of useful functionality hidden in undocumented C functions. What gives you that impression? What are the other examples?