Bazel: Spin Starlark into a stand-alone library

[johnynek]

there are many use case for the skylark and the build language.

It would be great to have the BUILD language for many configuration problems on the JVM. It would be cool for this code to be available as a maven jar, or if that is not possible as a bazel repository we can cleanly depend on.

I imagine the user plugs in the standard library of functions that can be called. So, bazel proper would give access to all the built-ins of skylark's current implementation.

Basically, at this point skylark looks like a total, dynamically typed programming language (one could even imagine a project to add type inference to skylark, which I think may be possible pretty easily since it lacks recursion).

/cc @jart @laurentlb

[ittaiz]

@mikelalcon is this something that is interesting for copybara as well?

[jart]

Congratulations @laurentlb it looks like you're going to be the next Guido van Rossum.

[mikelalcon]

We already use Skylark as a standalone library :). Well not really standalone, but almost :) https://github.com/google/copybara/blob/master/WORKSPACE#L90

[laurentlb]

We're working on it. :)

We're doing some experimentation (e.g. with copybara), but it's not ready yet for prime-time.

There's a standalone binary here if you want to play with it: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/skylark/Skylark.java
Please don't depend on it or on the libraries, we're likely to change the interface at some point.

Thank you for the request/feedback, this is very useful for us to prioritize the work!

[laszlocsomor]

@laurentlb : Could you prioritize this issue please?

[ttsugriy]

I also like the idea of making skylark a standalone library and I'm very much interested in using it. I used a hacky way to get a skylark standalone jar:

• apply patch https://gist.github.com/ttsugriy/cf1d8eccd7c45e0b8298cc205796bf75 to turn skylark-lang into a java_binary
• bazel build //src/main/java/com/google/devtools/build/lib:skylark-lang_deploy.jar
• use jarjar tool to shade or third-party jars like guava

[laurentlb]

For the record, there is now a separate implementation of the interpreter in Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

Edit: i.e. we still want to provide a standalone Skylark library, to be used by Bazel and other projects.

[ittaiz]

So you're deprecating the java interpreter?
Copybara is jvm based and we were also interested to depend on it inside
the jvm
On Wed, 4 Oct 2017 at 10:59 Laurent Le Brun [e-mail]
wrote:

For the record, there is a separate implementation of the interpreter in
Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334078465,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIFzxpOz4QnmyDALCkrgm2C5svetNgks5sozrggaJpZM4LmE4b
.

[dslomov]

We are not deprecating the Java implementation. Skylark-in-Go is a parallel
effort implementing the same language.

On Oct 4, 2017 01:08, "Ittai Zeidman" [e-mail] wrote:

So you're deprecating the java interpreter?
Copybara is jvm based and we were also interested to depend on it inside
the jvm
On Wed, 4 Oct 2017 at 10:59 Laurent Le Brun [e-mail]
wrote:

For the record, there is a separate implementation of the interpreter in
Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334078465
,
or mute the thread
ABUIFzxpOz4QnmyDALCkrgm2C5svetNgks5sozrggaJpZM4LmE4b>
.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334080693,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AExrX7jxVjywy6bNxjGv4xmevIwmALVyks5soz0XgaJpZM4LmE4b
.

[ittaiz]

Great.
Is splitting the java interpreter out to a standalone library still a goal?
On Wed, 4 Oct 2017 at 12:50 Dmitry Lomov [e-mail] wrote:

We are not deprecating the Java implementation. Skylark-in-Go is a parallel
effort implementing the same language.

On Oct 4, 2017 01:08, "Ittai Zeidman" [e-mail] wrote:

So you're deprecating the java interpreter?
Copybara is jvm based and we were also interested to depend on it inside
the jvm
On Wed, 4 Oct 2017 at 10:59 Laurent Le Brun [e-mail]
wrote:

For the record, there is a separate implementation of the interpreter
in
Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334078465
,
or mute the thread
ABUIFzxpOz4QnmyDALCkrgm2C5svetNgks5sozrggaJpZM4LmE4b>
.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334080693
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AExrX7jxVjywy6bNxjGv4xmevIwmALVyks5soz0XgaJpZM4LmE4b

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334105668,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF5S0cGjHeWmYhd-RPIzrdQkkZ_u8ks5so1UBgaJpZM4LmE4b
.

[laurentlb]

Yes, that's what I meant (sorry if I was unclear). We're still cleaning and improving the Java code.

[johnynek]

I wonder if at some point the speed of interpreting skylark is an issue. I
wonder if it ever makes sense to use a JIT like V8 or luajit to improve
skylark speed.

On Wed, Oct 4, 2017 at 05:33 Laurent Le Brun [e-mail]
wrote:

Yes, that's what I meant (sorry if I was unclear). We're still cleaning
and improving the Java code.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334195544,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEJdslC2yAL050syBATijtES29C11Ajks5so6UsgaJpZM4LmE4b
.

>

P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

[ttsugriy]

@johnynek, JVM already has a JIT, so hot Skylark code should be optimized as long as it's executed enough times, which depends on the complexity of your project. In practice, most likely most of the Java functions will be compiled, leaving only Skylark functions not compiled, since it's unlikely that they will reach JVM's second tier threshold.

[johnynek]

@ttsugriy I'm not sure that's really true. The interpreter code is jitted, but the skylark code itself won't be very effectively optimized by the JIT. If it were, then interpreter patterns seen in FP (see free monad, etc...) would not suffer a slow-down on the jvm.

I hear you that there should be a small amount of code in skylark and that a high fraction of build time should be in actions that are not skylark related. I have seen posts where people had O(n^2) work in skylark and things took a long time. Certainly O(n^2) work is no good, but the geek in me wonders if any off-the-shelf JITs (pypy, v8) could be leveraged to make skylark actually fast.

[laurentlb]

There are many ways to improve the performance. If we want JIT, we can compile the Skylark code to JVM bytecode and run it. Using LLVM or V8 doesn't make sense for Bazel, since the data structure should be shared with the rest of Bazel.

To be honest, we haven't optimized the interpreter yet. There are many things we could do (do faster variable lookups, optimize method resolution, etc.), but that's a separate discussion. Feel free to file a separate issue for that.

I keep this issue open for providing a nice, self-contained library. This will be useful for Buck, among others.

[ittaiz]

Just as another data point- we built a small WORKSPACE parser and BUILD
parser we'd love to replace with the above
On Tue, 10 Oct 2017 at 14:29 Laurent Le Brun [e-mail]
wrote:

There are many ways to improve the performance. If we want JIT, we can
compile the Skylark code to JVM bytecode and run it. Using LLVM or V8
doesn't make sense for Bazel, since the data structure should be shared
with the rest of Bazel.

To be honest, we haven't optimized the interpreter yet. There are many
things we could do (do faster variable lookups, optimize method resolution,
etc.), but that's a separate discussion. Feel free to file a separate issue
for that.

I keep this issue open for providing a nice, self-contained library. This
will be useful for Buck, among others.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-335443274,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF52ZN74tt1-JN_Dj1NlYJSnAzK1nks5sq1UqgaJpZM4LmE4b
.

[ittaiz]

cc @or-shachar (as our maintainer of a very poor WORKSPACE + BUILD file parsers)

[dtretyakov]

Any progress on providing Skylark Java language library? It would be very useful for parsing BUILD/WORKSPACE files.

[ittaiz]

+1
On Wed, 19 Sep 2018 at 22:13 Dmitry Tretyakov [e-mail]
wrote:

Any progress on providing Skylark Java language library? It would be very
useful for parsing BUILD/WORKSPACE files.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2367#issuecomment-422924499,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF9KhhLcI8UsghqTQeC7yrJKgPElOks5ucpc8gaJpZM4LmE4b
.

[laurentlb]

@alandonovan has recently made a lot of cleanup and some progress in this area.
Do you have any update to share, Alan? Maybe you could associate your future commits with this issue.

[alandonovan]

I've been working towards splitting the interpreter out for over 6 months, primarily for the health of the Blaze and Copybara code bases, not to support other clients. There remains much to do in terms of breaking dependencies and cleaning up API mistakes---another quarter I suspect.

Once that's done, we can discuss the bigger question of whether to publish the package, move it into a separate repository, and permanently freeze its API. I don't think any decision to do that has yet been made.