Scala-dev: Investigate compiler performance problems related to dynamic classloading of compiler plugins

Created on 10 Dec 2017  ·  14Comments  ·  Source: scala/scala-dev

https://github.com/scalameta/scalameta/pull/1181#issuecomment-350521760

performance
Source
picture retronym

All 14 comments

The classloading profiler in JMH (-prof hs_cl) shows that ~0 classes are loaded in each steady state iteration in the baseline, but ~900 classes are loaded when -Xplugin is provided.

⚡ (for EXTRA in "" -Xplugin:/Users/jz/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar; do sbt 'set scalaVersion in compilation := "2.12.4"' "hot -psource=scalap -f1 -wi 5 -i 4 -pextraArgs=$EXTRA -prof hs_cl" 2>&1; done) | egrep "(HotScalacBenchmark.compile|java.cls.loadedClasses) "
[info] HotScalacBenchmark.compile                                                     a8c43dc                    false    scalap  sample   50     831.059 ±      16.278  ms/op
[info] HotScalacBenchmark.compile:·java.cls.loadedClasses                             a8c43dc                    false    scalap  sample    4       1.500 ±      15.383      ?
[info] HotScalacBenchmark.compile                                                     a8c43dc  -Xplugin:/Users/jz/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample   46        913.811 ±        22.850  ms/op
[info] HotScalacBenchmark.compile:·java.cls.loadedClasses                             a8c43dc  -Xplugin:/Users/jz/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample    4        921.750 ±       311.804      ?

If I let them both warm up for 10x10s, I see that the enabling the macro-paradise plugin slows the result from 785ms to 897 (1.14x).

Next experiment is to add macro-paradise to the classpath of the forked JVM directly, to find out how much of that 14% is due to inherent cost of the plugin vs how much is due to the unintended costs of dynamic classlosding.

retronym picture retronym on 10 Dec 2017

I've just tested what happens when the plugin is added to the classpath containing scala-compiler.jar. The plugin classes are indeed loaded from this classloader (the compiler still creates a new classloader for the plugin in each instance of Global, but due to parent classloader delegation it doesn't actually reload the classes.

% (for EXTRA in -Xplugin:/Users/jz/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar; do sbt 'set unmanagedJars in compilation in Compile += file("/Users/jz/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar")' 'set scalaVersion in compilation := "2.12.4"' "hot -psource=scalap -f1 -wi 10 -i 8 -pextraArgs=$EXTRA -vEXTRA" 2>&1; done)

| Scenario | Score | Factor vs baseline |
|--|--|--|
| No Plugin (baseline) | 777.467 ± 2.331ms | |
| Plugin statically loaded | 818.063 ± 4.678 | 1.05x |
| Plugin dynamically loaded | 886.732 ± 8.195 | 1.14x |

So the presence of macro paradise slows down this benchmark by 14%, but only 5% of that seems to be the direct fault of the plugin itself.

(Updated numbers with 3 forks in the benchmark)

retronym picture retronym on 13 Dec 2017

Macro implementations are also called with a dynamic classloader. It would be interesting to see if the same trick of putting the macro on the compiler classpath (or, just caching the dynamically created classloader) affects the performance of compilation of code that uses macros.

retronym picture retronym on 13 Dec 2017

I have a prototype at https://github.com/scalacenter/scala/tree/ticket/sd-458 for both macros and plugins. I'm in the process of benchmarking the impact in small and big OSS projects. I will post benchmark results as soon as I have them, and then open a pull request.

My approach caches dynamically created classloaders only if all the URLs are either jar or zip files (as suggested by @retronym). The invalidation of these files assumes that the last modified time of the zip file changes if one of the zip archive entries changes. This may not be true if a postprocessing step is applied to compilation products, like reproducible-maven-build-plugin, which sets all zip and jar timestamps to zero to achieve build reproducibility.

(However, as I understand it, this breaks classpath caching for the compiler too; so I'm not sure it's an immediate problem we need to solve now. If we aim for a solid solution, we should hash the contents of every classpath entry, but this takes time for huge classpaths.)

If a file that is not a zip or jar is found, the caching doesn't happen. This occurs, by default, when a classpath entry ends with the following classpath entries in sbt (show classpathTypes: Set(eclipse-plugin, bundle, scala-jar, hk2-jar, orbit)). As a note, all other classpath entries whose extension does not appear in the classpathTypes key will be filtered out before compiling.

jvican picture jvican on 21 Jan 2018 
jvican in /data/rw/code/experiments/compiler-benchmark                                  [19:28:12] 
> $ for SV in "2.12.5--bin-dbd90495c2-SNAPSHOT"; do /usr/bin/sbt "set scalaVersion in ThisBuild := \"$SV\"" "hot -psource=scalap -f1 -wi 10 -i 8 -vEXTRA" 2>&1; done

[info] Benchmark                                   (corpusVersion)  (extraArgs)  (resident)  (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest                    false    scalap  sample   87   975.417 ± 12.725  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest                    false    scalap  sample        918.553           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest                    false    scalap  sample        967.836           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest                    false    scalap  sample       1024.249           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest                    false    scalap  sample       1046.479           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest                    false    scalap  sample       1084.228           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest                    false    scalap  sample       1084.228           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest                    false    scalap  sample       1084.228           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest                    false    scalap  sample       1084.228           ms/op

jvican in /data/rw/code/experiments/compiler-benchmark                                                                                                                                            [19:19:47] 
> $ for SV in "2.12.5--bin-dbd90495c2-SNAPSHOT" "2.12.5-bin-f18e3c59fd-SNAPSHOT"; do /usr/bin/sbt "set scalaVersion in ThisBuild := \"$SV\"" "hot -psource=scalap -f1 -wi 10 -i 8 -pextraArgs=-Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar -vEXTRA" 2>&1; done

[info] Benchmark                                   (corpusVersion)                                                                                       (extraArgs)  (resident)  (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample   73  1137.892 ± 17.863  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1060.110           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1130.365           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1200.829           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1221.172           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op

[info] Benchmark                                   (corpusVersion)                                                                                       (extraArgs)  (resident)  (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample   86   985.844 ± 10.220  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample        937.427           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample        981.467           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1027.919           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1033.162           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op

Summary

| Scenario | Scala version | Compilation time (ms) | Factor vs baseline |
| -------------------- | -------------------------------- | --------------------- | ------------------ |
| no plugin (baseline) | 2.12.5-bin-dbd90495c2-SNAPSHOT | 975 | |
| dynamically loaded | 2.12.5-bin-dbd90495c2-SNAPSHOT | 1137 | 1.17 |
| dynamically loaded | 2.12.5-bin-f18e3c59fd-SNAPSHOT | 985 | 1.01 |

Where f18e3c59fd is the commit hash of
https://github.com/scalacenter/scala/tree/ticket/sd-458, and dbd90495c2 the
latest commit in the 2.12.5 branch to this date.

These results confirm that caching dynamically loaded compiler plugins works,
where the caching improvements behave as well as the baseline. Before the patch,
dynamically loading a compiler plugin slowed down a warmed up compiler by 17%.

Machine

cat /proc/cpuinfo:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 78
model name  : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping    : 3
microcode   : 0x88
cpu MHz     : 3213.470
cache size  : 4096 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs        : cpu_meltdown
bogomips    : 5618.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 78
model name  : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping    : 3
microcode   : 0x88
cpu MHz     : 3225.341
cache size  : 4096 KB
physical id : 0
siblings    : 2
core id     : 1
cpu cores   : 2
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs        : cpu_meltdown
bogomips    : 5618.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

uname -a:

Linux tribox 4.14.13-1-ARCH #1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018 x86_64 GNU/Linux

Processor details

Hyper threading is disabled, NUMA is disabled, Turbo boost is disabled, cpu
frequency is set to 3.2GHz. intel_pstate is used.

Implications

Dynamically loaded compiler plugins are the norm, so more or less all builds
that use them will benefit from these speedups.

Next

I'm working on collecting benchmarking results for macro libraries. I will
report them soon.

jvican picture jvican on 21 Jan 2018
👍1

To measure the impact of this patch against code using macros, I have tried first with case-app, a CLI library built on top of shapeless.

show testsJVM/compile:scalacOptions:

[info] * -feature
[info] * -deprecation
[info] * -Xplugin:/home/jvican/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scalamacros/paradise_2.12.4/2.1.0/paradise_2.12.4-2.1.0.jar

show testsJVM/dependencyClasspath:

[info] * Attributed(/data/rw/code/scala/case-app/core/jvm/target/scala-2.12/classes)
[info] * Attributed(/data/rw/code/scala/case-app/annotations/jvm/target/scala-2.12/classes)
[info] * Attributed(/data/rw/code/scala/case-app/util/jvm/target/scala-2.12/classes)
[info] * Attributed(/data/rw/code/scala/scala-profiling/scalac/build/pack/lib/scala-library.jar)
[info] * Attributed(/home/jvican/.sbt/boot/scala-2.12.4/lib/scala-library.jar)
[info] * Attributed(/home/jvican/.coursier/cache/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_2.12/1.1.1/macro-compat_2.12-1.1.1.jar)
[info] * Attributed(/home/jvican/.coursier/cache/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.12/2.3.3/shapeless_2.12-2.3.3.jar)

For the benchmark I've used scalac-profiling's sbt plugin, which has a task that allows sbt to warm up a compiler for a certain duration. Here are the logs of the session. I'd like to try with HotSbtBenchmarking to get better statistics, average and percentiles; but, so far, the results seem to point out to a ~20% speedup (5 seconds with classloaders caches vs ~6 seconds).

Tomorrow I'll gather more fine-grained results for case-app and will try to test bigger projects like circe and scalatest.

jvican picture jvican on 22 Jan 2018
👍1

Reminder: you can easily remove SBT from the benchmarking equation with the instructions in https://github.com/scala/scala-dev/issues/392.

retronym picture retronym on 22 Jan 2018
👍1

which sets all zip and jar timestamps to zero to achieve build reproducibility.

Are you sure it zero-s the timestamp of the JAR file itself? I thought it was just of the ZIP metadata _inside_ the JAR.

We could/should add file size into the cache key as another way to help out.

retronym picture retronym on 22 Jan 2018

(I'll double check tomorrow, but that was my understanding from reading the source. IIRC Bazel also does zero all timestamps, including the ones of the file; I'm confirming tomorrow.)

jvican picture jvican on 22 Jan 2018

Are you sure it zero-s the timestamp of the JAR file itself? I thought it was just of the ZIP metadata inside the JAR.

Apparently the maven plugin only does it for the zip entries, but build tools like Bazel zero out all timestamps (including the file system's timestamp of a jar), so the current invalidation algorithm would fail. I agree adding the file size to the cache key is a good idea, I'll do that.

We may want to experiment with a more robust solution that reliably caches the classpath. We may be able to accommodate this use case in two ways:

  1. Adding a new compiler flag that allows build tools to pass in hashes for every classpath entry.
  2. Let the compiler hash every classpath entry on compilation (expensive, that's why I propose option 1).
jvican picture jvican on 23 Jan 2018

Performance

Here is my continuation about the impact of the paches in this Scala Center branch. This time, I have benchmarked two important open source projects: Circe and upickle.

Tested Scala versions

| Scala Version | Changes |
| ------------------------------ | ----------------------------------------- |
| 2.12.5-bin-dbd90495c2-SNAPSHOT | Last stable 2.12.5 change |
| 2.12.5-bin-f18e3c59fd-SNAPSHOT | Cache classloaders for plugins |
| 2.12.5-bin-0417fcf133-SNAPSHOT | Cache classloaders for plugins and macros |

Circe

This is a benchmark of Circe's test suite.

Command to generate circe benchmarks

for args_file in \
  /home/benchmarks/experiments/circe/modules/tests/js/target/test.args \
  /home/benchmarks/experiments/circe/modules/tests/js/target/test-with-classes-dir.args \
  /home/benchmarks/experiments/circe/modules/tests/jvm/target/test.args \
  /home/benchmarks/experiments/circe/modules/tests/jvm/target/test-with-classes-dir.args; do \
  for v in 2.12.5-bin-dbd90495c2-SNAPSHOT 2.12.5-bin-f18e3c59fd-SNAPSHOT 2.12.5-bin-0417fcf133-SNAPSHOT; do \
    /usr/bin/sbt "set scalaVersion in compilation := \"$v\"" "hot -f1 -wi 10 -i 8 -p source=@$args_file" || break; \
  done

Test argument files for Circe

The test.args files have been generated from ArgsFile.scala with exportJars := true enabled (meaning that internal Circe dependencies are present in jars and not in classes directories). The args files named test-with-classes-dir have the stock sbt options.


/home/benchmarks/experiments/circe/modules/tests/js/target/test.args
-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-Xplugin:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-compiler_2.12.4/0.6.22/scalajs-compiler_2.12.4-0.6.22.jar

-classpath

/home/benchmarks/experiments/circe/modules/tests/js/target/scala-2.12/circe-tests_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/core/js/target/scala-2.12/circe-core_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/numbers/js/target/scala-2.12/circe-numbers_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/parser/js/target/scala-2.12/circe-parser_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/scalajs/target/scala-2.12/circe-scalajs_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/testing/js/target/scala-2.12/circe-testing_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/numbers-testing/.js/target/scala-2.12/circe-numbers-testing_sjs0.6_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-reflect.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-platform_sjs0.6_2.12/0.0.5/catalysts-platform_sjs0.6_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/machinist_sjs0.6_2.12/0.6.2/machinist_sjs0.6_2.12-0.6.2.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalactic/scalactic_sjs0.6_2.12/3.0.4/scalactic_sjs0.6_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-macros_sjs0.6_2.12/1.0.1/cats-macros_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_sjs0.6_2.12/2.3.3/shapeless_sjs0.6_2.12-2.3.3.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_sjs0.6_2.12/1.1.1/macro-compat_sjs0.6_2.12-1.1.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-test-interface_2.12/0.6.22/scalajs-test-interface_2.12-0.6.22.jar:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-library.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-macros_sjs0.6_2.12/0.0.5/catalysts-macros_sjs0.6_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalacheck/scalacheck_sjs0.6_2.12/1.13.5/scalacheck_sjs0.6_2.12-1.13.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel-laws_sjs0.6_2.12/1.0.1/cats-kernel-laws_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_2.12/1.1.1/macro-compat_2.12-1.1.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-library_2.12/0.6.22/scalajs-library_2.12-0.6.22.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalatest/scalatest_sjs0.6_2.12/3.0.4/scalatest_sjs0.6_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel_sjs0.6_2.12/1.0.1/cats-kernel_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-laws_sjs0.6_2.12/1.0.1/cats-laws_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-core_sjs0.6_2.12/1.0.1/cats-core_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/discipline_sjs0.6_2.12/0.8/discipline_sjs0.6_2.12-0.8.jar

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/FloatJsonTests.scala

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/Spaces2PrinterExample.scala

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/LargeNumberDecoderTests.scala

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/scalajs/ScalaJsSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/DecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/EncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ObjectEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ACursorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonObjectSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/AccumulatingDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/SerializableSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/CodecSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ShowErrorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonNumberSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ArrayEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/PrinterSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/parser/ParserSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/syntax/SyntaxSuite.scala

/home/benchmarks/experiments/circe/modules/tests/js/target/scala-2.12/src_managed/test/io/circe/TupleCodecSuite.scala

/home/benchmarks/experiments/circe/modules/tests/js/target/scala-2.12/src_managed/test/io/circe/ProductCodecSuite.scala


/home/benchmarks/experiments/circe/modules/tests/js/target/test-with-classes-dir.args
-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds
<>
-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-Xplugin:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-compiler_2.12.4/0.6.22/scalajs-compiler_2.12.4-0.6.22.jar

-classpath

/home/benchmarks/experiments/circe/modules/tests/js/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/core/js/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/numbers/js/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/parser/js/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/scalajs/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/testing/js/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/numbers-testing/.js/target/scala-2.12/classes:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-reflect.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-platform_sjs0.6_2.12/0.0.5/catalysts-platform_sjs0.6_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/machinist_sjs0.6_2.12/0.6.2/machinist_sjs0.6_2.12-0.6.2.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalactic/scalactic_sjs0.6_2.12/3.0.4/scalactic_sjs0.6_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-macros_sjs0.6_2.12/1.0.1/cats-macros_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_sjs0.6_2.12/2.3.3/shapeless_sjs0.6_2.12-2.3.3.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_sjs0.6_2.12/1.1.1/macro-compat_sjs0.6_2.12-1.1.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-test-interface_2.12/0.6.22/scalajs-test-interface_2.12-0.6.22.jar:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-library.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-macros_sjs0.6_2.12/0.0.5/catalysts-macros_sjs0.6_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalacheck/scalacheck_sjs0.6_2.12/1.13.5/scalacheck_sjs0.6_2.12-1.13.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel-laws_sjs0.6_2.12/1.0.1/cats-kernel-laws_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_2.12/1.1.1/macro-compat_2.12-1.1.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-library_2.12/0.6.22/scalajs-library_2.12-0.6.22.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalatest/scalatest_sjs0.6_2.12/3.0.4/scalatest_sjs0.6_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel_sjs0.6_2.12/1.0.1/cats-kernel_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-laws_sjs0.6_2.12/1.0.1/cats-laws_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-core_sjs0.6_2.12/1.0.1/cats-core_sjs0.6_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/discipline_sjs0.6_2.12/0.8/discipline_sjs0.6_2.12-0.8.jar

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/FloatJsonTests.scala

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/Spaces2PrinterExample.scala

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/LargeNumberDecoderTests.scala

/home/benchmarks/experiments/circe/modules/tests/js/src/test/scala/io/circe/scalajs/ScalaJsSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/DecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/EncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ObjectEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ACursorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonObjectSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/AccumulatingDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/SerializableSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/CodecSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ShowErrorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonNumberSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ArrayEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/PrinterSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/parser/ParserSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/syntax/SyntaxSuite.scala

/home/benchmarks/experiments/circe/modules/tests/js/target/scala-2.12/src_managed/test/io/circe/TupleCodecSuite.scala

/home/benchmarks/experiments/circe/modules/tests/js/target/scala-2.12/src_managed/test/io/circe/ProductCodecSuite.scala


/home/benchmarks/experiments/circe/modules/tests/jvm/target/test.args
-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-classpath

/home/benchmarks/experiments/circe/modules/tests/jvm/target/scala-2.12/circe-tests_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/core/jvm/target/scala-2.12/circe-core_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/numbers/jvm/target/scala-2.12/circe-numbers_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/parser/jvm/target/scala-2.12/circe-parser_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/jawn/target/scala-2.12/circe-jawn_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/testing/jvm/target/scala-2.12/circe-testing_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/experiments/circe/modules/numbers-testing/.jvm/target/scala-2.12/circe-numbers-testing_2.12-0.9.2-SNAPSHOT.jar:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-reflect.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-laws_2.12/1.0.1/cats-laws_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.0.4/scala-parser-combinators_2.12-1.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-platform_2.12/0.0.5/catalysts-platform_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel_2.12/1.0.1/cats-kernel_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalacheck/scalacheck_2.12/1.13.5/scalacheck_2.12-1.13.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.12/1.0.5/scala-xml_2.12-1.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/machinist_2.12/0.6.2/machinist_2.12-0.6.2.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-library.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-core_2.12/1.0.1/cats-core_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.4/scalatest_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-macros_2.12/1.0.1/cats-macros_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.12/2.3.3/shapeless_2.12-2.3.3.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-macros_2.12/0.0.5/catalysts-macros_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel-laws_2.12/1.0.1/cats-kernel-laws_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/spire-math/jawn-parser_2.12/0.11.0/jawn-parser_2.12-0.11.0.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_2.12/1.1.1/macro-compat_2.12-1.1.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalactic/scalactic_2.12/3.0.4/scalactic_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/discipline_2.12/0.8/discipline_2.12-0.8.jar

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/FloatJsonTests.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/PrinterWriterReuseSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/Spaces2PrinterExample.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/MemoizedPiecesSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/LargeNumberDecoderTests.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/jawn/JawnParserSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/DecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/EncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ObjectEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ACursorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonObjectSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/AccumulatingDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/SerializableSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/CodecSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ShowErrorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonNumberSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ArrayEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/PrinterSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/parser/ParserSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/syntax/SyntaxSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/target/scala-2.12/src_managed/test/io/circe/TupleCodecSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/target/scala-2.12/src_managed/test/io/circe/ProductCodecSuite.scala


/home/benchmarks/experiments/circe/modules/tests/jvm/target/test-with-classes-dir.args
-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-deprecation

-encoding

UTF-8

-feature

-language:existentials

-language:higherKinds

-unchecked

-Yno-adapted-args

-Ywarn-dead-code

-Ywarn-numeric-widen

-Xfuture

-Ywarn-unused-import

-classpath

/home/benchmarks/experiments/circe/modules/tests/jvm/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/core/jvm/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/numbers/jvm/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/parser/jvm/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/jawn/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/testing/jvm/target/scala-2.12/classes:/home/benchmarks/experiments/circe/modules/numbers-testing/.jvm/target/scala-2.12/classes:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-reflect.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-laws_2.12/1.0.1/cats-laws_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.0.4/scala-parser-combinators_2.12-1.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-platform_2.12/0.0.5/catalysts-platform_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel_2.12/1.0.1/cats-kernel_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalacheck/scalacheck_2.12/1.13.5/scalacheck_2.12-1.13.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.12/1.0.5/scala-xml_2.12-1.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/machinist_2.12/0.6.2/machinist_2.12-0.6.2.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar:/home/benchmarks/.sbt/boot/scala-2.12.4/lib/scala-library.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-core_2.12/1.0.1/cats-core_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.4/scalatest_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-macros_2.12/1.0.1/cats-macros_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.12/2.3.3/shapeless_2.12-2.3.3.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/catalysts-macros_2.12/0.0.5/catalysts-macros_2.12-0.0.5.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel-laws_2.12/1.0.1/cats-kernel-laws_2.12-1.0.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/spire-math/jawn-parser_2.12/0.11.0/jawn-parser_2.12-0.11.0.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/macro-compat_2.12/1.1.1/macro-compat_2.12-1.1.1.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalactic/scalactic_2.12/3.0.4/scalactic_2.12-3.0.4.jar:/home/benchmarks/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/discipline_2.12/0.8/discipline_2.12-0.8.jar

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/FloatJsonTests.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/PrinterWriterReuseSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/Spaces2PrinterExample.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/MemoizedPiecesSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/LargeNumberDecoderTests.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/src/test/scala/io/circe/jawn/JawnParserSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/DecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/EncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ObjectEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ACursorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonObjectSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/AccumulatingDecoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/SerializableSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/CodecSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ShowErrorSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/JsonNumberSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/ArrayEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/PrinterSuites.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/KeyEncoderSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/parser/ParserSuite.scala

/home/benchmarks/experiments/circe/modules/tests/shared/src/test/scala/io/circe/syntax/SyntaxSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/target/scala-2.12/src_managed/test/io/circe/TupleCodecSuite.scala

/home/benchmarks/experiments/circe/modules/tests/jvm/target/scala-2.12/src_managed/test/io/circe/ProductCodecSuite.scala

Circe results under different scenarios

| Scenario | Scala version | Compilation time (ms) | Factor vs baseline |
| -------------------------- | -------------------------------- | --------------------- | ------------------ |
| Scala.js with jar deps | 2.12.5-bin-dbd90495c2-SNAPSHOT | 31985.762 ± 1755.206 | |
| Scala.js with jar deps | 2.12.5-bin-f18e3c59fd-SNAPSHOT | 23664.263 ± 1119.504 | 1.35x |
| Scala.js with jar deps | 2.12.5-bin-0417fcf133-SNAPSHOT | 23769.121 ± 1137.835 | 1.35x |
| Scala.js with classes dirs | 2.12.5-bin-dbd90495c2-SNAPSHOT | 32438.747 ± 1011.336 | |
| Scala.js with classes dirs | 2.12.5-bin-f18e3c59fd-SNAPSHOT | 23882.367 ± 924.465 | 1.36x |
| Scala.js with classes dirs | 2.12.5-bin-0417fcf133-SNAPSHOT | 23781.704 ± 1392.437 | 1.36x |
| JVM with jar deps | 2.12.5-bin-dbd90495c2-SNAPSHOT | 21222.515 ± 1291.782 | |
| JVM with jar deps | 2.12.5-bin-f18e3c59fd-SNAPSHOT | 21583.888 ± 1028.630 | 0.99x |
| JVM with jar deps | 2.12.5-bin-0417fcf133-SNAPSHOT | 21831.352 ± 1169.938 | 0.98x |
| JVM with classes dirs | 2.12.5-bin-dbd90495c2-SNAPSHOT | 21445.476 ± 1429.397 | |
| JVM with classes dirs | 2.12.5-bin-f18e3c59fd-SNAPSHOT | 21562.917 ± 1072.940 | 0.99x |
| JVM with classes dirs | 2.12.5-bin-0417fcf133-SNAPSHOT | 21055.406 ± 11114.899 | 1.02x |

Complete logs are available in this link.

Interpretation of Circe benchmarks

These results strongly suggest that:

  1. Code depending on compiler plugins is significantly faster after we cache their classloaders.

    • For a non trivial use case like Circe, compiling code with Scalajs is 35% faster.

  2. Caching classloaders for macro libraries has no effect.
  3. There is no improvement between having some dependencies be jars or classes directories.
  4. On a meta level, Scala.js compilation is (after this patch) only 9% slower than JVM compilation.

Upickle

This is a benchmark of Upickle's Scala.js test suite.

Command to generate upickle benchmarks

for i in 2.12.5-bin-dbd90495c2-SNAPSHOT 2.12.5-bin-f18e3c59fd-SNAPSHOT; do \
  sbt "set scalaVersion in compilation := \"$i\"" 'hot -f1 -wi 10 -i 8 -p source=@/data/rw/code/scala/upickle/upickle/js/target/test.args' || break; \
done

Upickle results under different scenarios

| Scenario | Scala version | Compilation time (ms) | Factor vs baseline |
| -------- | -------------------------------- | --------------------- | ------------------ |
| Scala.js | 2.12.5-bin-dbd90495c2-SNAPSHOT | 12381.585 ± 380.321 | |
| Scala.js | 2.12.5-bin-f18e3c59fd-SNAPSHOT | 10714.350 ± 355.112 | 1.16x |

Complete logs are available in this link.

Interpretation of Upickle benchmarks

These results strongly suggest that caching plugin classloaders for small-sized libraries also improves compile times. In this case, it's 16%, which seems to be the lowest improvement we've seen in all the studied scenarios.

General interpretation of results

We have seen several results of this patch in different open source projects. All of my results strongly suggest that caching compiler plugins' classloaders (dynamically loaded) is worth it, with compile-time improvements ranging from 16%, 20% to 35% in projects of different size.

My results also suggest that there is no compile-time improvement to scratch on macro-land, so the commit hash 0417fcf133 will be dropped from the upcoming PR to scala/scala, only f18e3c59fd will be included.

With the increasing use of compiler plugins in our Community for things like tooling (Scalameta, Scala-sculpt, Splain, scalac-profilng) and the compilation to other backends (Scala.js and Scala Native), this is good news because the compilation with these projects will significantly decrease.

/cc @densh @sjrd

jvican picture jvican on 7 Feb 2018

Thanks for the thorough analysis. I'm open to keeping macro classloaders caching even if it performance neutral, unless we can see some risks to the approach.

retronym picture retronym on 7 Feb 2018
👍1

Both those projects compile at such a glacial pace relative to "regular" Scala code.

For example, upickle test:

% cloc /Users/jz/code/upickle/upickle/shared/src/test
 ... 1745

% sbt

> test:cleanClasses
[success] Total time: 0 s, completed 08/02/2018 9:29:50 AM
> upickleJVM/test:compile
[info] Compiling 12 Scala sources to /Users/jz/code/upickle/upickle/jvm/target/scala-2.12/test-classes...
[warn] /Users/jz/code/upickle/upickle/shared/src/test/scala/upickle/StructTests.scala:60: object LinkedList in package mutable is deprecated (since 2.11.0): low-level linked lists are deprecated
[warn]         'LinkedList - rw(collection.mutable.LinkedList("omg", "i am", "cow"), """["omg","i am","cow"]""")
[warn]                                             ^
[warn] /Users/jz/code/upickle/upickle/shared/src/test/scala/upickle/StructTests.scala:60: object LinkedList in package mutable is deprecated (since 2.11.0): low-level linked lists are deprecated
[warn]         'LinkedList - rw(collection.mutable.LinkedList("omg", "i am", "cow"), """["omg","i am","cow"]""")
[warn]                         ^
[warn] two warnings found
[success] Total time: 10 s, completed 08/02/2018 9:30:08 AM

So about 0.17kloc/s. Compare with 2-3kloc/s for regular code.

So there must be something really inefficient in the way implicits are organized or typeclasses are being materialized, or maybe the code generated is really massive.

To really get a sense of the _why_ of the performance improvements, we'd need to understand what the bottlenecks are and compare profiles before and after. Not something that we need to do to justify the improvement, but interesting nonetheless.

retronym picture retronym on 8 Feb 2018

FWIW it's not surprising at all uPickle's test suite compiles glacially slow. uPickle's code-generating macros would normally be only a tiny part of any other program (in 0.5.x, with automatic/deep derivation gone) but in the uPickle test suite those code-generating macros are 100% of the compilation run.

a 10-15x ratio sounds about right; it wouldn't surprise me at all if on average each macro callsite expanded to 20-30 lines of "boring" code, especially in the test suite which tries to exercise derivation for big/complex datatypes, divided by two since only half of the lines in the test suite being macro expansions

lihaoyi picture lihaoyi on 21 Mar 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

adriaanm picture adriaanm  ·  6Comments
adriaanm picture adriaanm  ·  7Comments
lrytz picture lrytz  ·  4Comments
retronym picture retronym  ·  5Comments
retronym picture retronym  ·  8Comments