Elixir: .beam files differ between builds

[bmwiedemann]

Environment

• Elixir & Erlang/OTP versions (elixir --version):
  elixir-1.7.4 erlang-21.1.4
• Operating system: openSUSE Tumbleweed 20190124

Current behavior

Somewhat similar to #4814 building the openSUSE elixir package produces various .beam files that differ for every build, e.g. /usr/lib/elixir/lib/mix/ebin/Elixir.Mix.Tasks.Deps.Unlock.beam

Some of the diff is even there when doing builds as similar as possible (e.g. disabling ASLR, 1-core VM without parallelism)

The erlang package itself does not have this kind of variations in its .beam files (but others from timestamps)

Note: the elixir-ex_doc package (in the best case) only has 1 beam file with 1 bit diff so might be easier to debug.

Expected behavior

Builds should produce deterministic results.
See https://reproducible-builds.org/ for why this matters.

[josevalim]

Thanks @bmwiedemann. Do you have steps to reproduce this? How are you going the builds? Which Erlang/OTP version are you using?

I am getting consistent results with Elixir master and Erlang/OTP 21. Note that earlier Erlang/OTP versions would include timestamps, which would make them not reproducible.

[josevalim]

I also could not reproduce this in ExDoc:

~/OSS/ex_doc[master]$ mix compile --force
Compiling 18 files (.ex)
Generated ex_doc app
~/OSS/ex_doc[master]$ md5 _build/dev/lib/ex_doc/ebin/*
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.CLI.beam) = 48e25a8df33d27d27f9f556db1b52a41
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Config.beam) = 28887a5beed69bcecfc76074dd7a9e3d
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Assets.beam) = cb365ebd1781bf25bd36d6e095b5eed9
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Templates.beam) = c7589f848178d992a962b2d488c86b0a
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.beam) = a27fa4cfe66a6c20f71e74fced081456
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Assets.beam) = 5a55eaea4239a96ed9a0b435c73c2285
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Autolink.beam) = 3ec5d6a224a44c7c63bf3666518e08f4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam) = ffaaa7c414f24b76a1459b5ac4ae7807
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.beam) = 99b9c8836e9c529349e1f14ca9b1b655
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.FunctionNode.beam) = c5f4042c02cdd47be20cf7a518367fce
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.GroupMatcher.beam) = fb0135e08cc44e59ef05400cfd6f8a06
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Highlighter.beam) = 20f15b6cedae7a7f4a1c8fe60a39b031
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Cmark.beam) = a0fbb555b7ce8dc580b5aa91786c4758
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Earmark.beam) = 94b664aecbb6e9143eeac3d35369d06f
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.beam) = 28811e650fd5778f4f2e7011b1442f32
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.ModuleNode.beam) = 7422d37d42ecfddcb44b12cdcb65bc3b
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.Error.beam) = 012642520a72ed473a319f034c078e40
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.beam) = 9be29c27cc206b165680989f4d277343
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.TypeNode.beam) = 7fcfac1158bdea265885d8fbe153ca2c
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.beam) = a62fdf39d3f2c13444852e0ef8e77ea4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.Mix.Tasks.Docs.beam) = ac31564174cabc7b26faa2b592849f38
MD5 (_build/dev/lib/ex_doc/ebin/ex_doc.app) = 64735fd5d2ccd3ef261675b9c56fe3f0
~/OSS/ex_doc[master]$ mix compile --force
Compiling 18 files (.ex)
Generated ex_doc app
~/OSS/ex_doc[master]$ md5 _build/dev/lib/ex_doc/ebin/*
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.CLI.beam) = 48e25a8df33d27d27f9f556db1b52a41
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Config.beam) = 28887a5beed69bcecfc76074dd7a9e3d
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Assets.beam) = cb365ebd1781bf25bd36d6e095b5eed9
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Templates.beam) = c7589f848178d992a962b2d488c86b0a
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.beam) = a27fa4cfe66a6c20f71e74fced081456
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Assets.beam) = 5a55eaea4239a96ed9a0b435c73c2285
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Autolink.beam) = 3ec5d6a224a44c7c63bf3666518e08f4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam) = ffaaa7c414f24b76a1459b5ac4ae7807
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.beam) = 99b9c8836e9c529349e1f14ca9b1b655
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.FunctionNode.beam) = c5f4042c02cdd47be20cf7a518367fce
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.GroupMatcher.beam) = fb0135e08cc44e59ef05400cfd6f8a06
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Highlighter.beam) = 20f15b6cedae7a7f4a1c8fe60a39b031
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Cmark.beam) = a0fbb555b7ce8dc580b5aa91786c4758
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Earmark.beam) = 94b664aecbb6e9143eeac3d35369d06f
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.beam) = 28811e650fd5778f4f2e7011b1442f32
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.ModuleNode.beam) = 7422d37d42ecfddcb44b12cdcb65bc3b
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.Error.beam) = 012642520a72ed473a319f034c078e40
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.beam) = 9be29c27cc206b165680989f4d277343
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.TypeNode.beam) = 7fcfac1158bdea265885d8fbe153ca2c
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.beam) = a62fdf39d3f2c13444852e0ef8e77ea4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.Mix.Tasks.Docs.beam) = ac31564174cabc7b26faa2b592849f38
MD5 (_build/dev/lib/ex_doc/ebin/ex_doc.app) = 64735fd5d2ccd3ef261675b9c56fe3f0

[bmwiedemann]

erlang is 21.1.4

Here is a reproducer (also works in Debian, if you apt install osc obs-build)
needs a free openSUSE account, though.

osc checkout openSUSE:Factory/elixir-ex_doc ; cd $_
osc build --noservice
md5sum /var/tmp/build-root/standard-x86_64/home/abuild/rpmbuild/BUILD/ex_doc-v0.7.2/_build/prod/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam

produced here:
b02d8e1067bb0df91e164fac9dc46457 /var/tmp/build-root/standard-x86_64/home/abuild/rpmbuild/BUILD/ex_doc-v0.7.2/_build/prod/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam
460044305532c62996b564af01726f24 /var/tmp/build-root/standard-x86_64/home/abuild/rpmbuild/BUILD/ex_doc-v0.7.2/_build/prod/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam

Logs show, it just uses mix compile, but our build scripts scratch the whole BUILD dir to ensure a clean build.

[josevalim]

Oh, sorry, I see now that you included Erlang in your initial report, sorry.

Now I also see why I can't reproduce it, I am running on v1.8.0 which has this commit: d76f150274352dd207cb82abca523ae73bb21b40

The commit above made the compiler deterministic when meta-programming. Therefore this is fixed in v1.8+ and master. In any case, thank you for checking and for the report!

[bmwiedemann]

d76f150 cherry-picked cleanly to 1.7.4 and reduced the elixir diff to only have instances of

/usr/lib/elixir/lib/ex_unit/ebin/Elixir.ExUnit.Assertions.beam differs at offset '8' (Erlang BEAM file)
@@ -1,4 +1,4 @@
-00000000  46 4f 52 31 00 00 6f 74  42 45 41 4d 41 74 55 38  |FOR1..otBEAMAtU8|
+00000000  46 4f 52 31 00 00 6f 78  42 45 41 4d 41 74 55 38  |FOR1..oxBEAMAtU8|
 00000010  00 00 07 41 00 00 00 af  18 45 6c 69 78 69 72 2e  |...A.....Elixir.|

Is there another patch missing?

[josevalim]

This is a bit weird. Bytes 5 to 8 are the size of the BEAM chunk. So if the size is different, then something else should also show up in the diff?

[bmwiedemann]

indeed, our special-purpose diff tool was hiding the lower part diff.txt
This now looks pretty similar to what it was without the patch.
I'm wondering if I should spend effort on packaging 1.8 first.

[josevalim]

I see, so the difference is in the documentation chunk. I am not quite sure yet though. I can also reproduce it locally. I will take a look later after I pick the kids from school.

[josevalim]

Btw, this diff tool is neat, is it available externally? :D

[bmwiedemann]

The bad one is build-compare and the good one is https://github.com/bmwiedemann/reproducibleopensuse/blob/master/filterdiff
used as filterdiff xxd $file1 $file2
or filterdiff "hexdump -C" $file1 $file2

[bmwiedemann]

If you want to work with the openSUSE package, you can also try my rbk tool from the reproducibleopensuse repo.
I quickly upgraded to elixir-1.8.0 in https://build.opensuse.org/package/show/home:bmwiedemann:reproducible:test/elixir and it showed a similar diff to 1.7.4 .
Using 1.8 indeed fixed the indeterminism in the elixir-ex_doc build, so that was something else to what affects .beam files in elixir itself.

[josevalim]

Got it. So we literally had :rand.uniform being executed at the moment the docs were defined. Fixed now.

[bmwiedemann]

I applied this patch to my 1.8 package and am still getting diffs in at least 5 beam files. And github did not let me upload that 1MB file: https://rb.zq1.de/compare.factory-20190113/elixir-1.8-diff.txt
And this alternative view

Elixir.Stream.beam has the smallest diff of the 5.
Apart from above Code.Formatter, Enum, Stream, String, System,
extra candidates to watch are Kernel, Keyword, List .beam files.

Shall I open a new bug or do we keep tracking it here?

[eksperimental]

Could you please share which are these beam files? It will be easier to track down

[josevalim]

They are: Code.Formatter, Stream, Enum, String and System.

@bmwiedemann we can likely solve all of them but we cannot solve the System one as it does include build_info and that will naturally change every time it is built. Suggestions?

[josevalim]

So I cannot reproduce the one for Code.Formatter but I can reproduce it for Enum, Stream and String and they are related to the Dbgi chunk (which stores AST). I will post more updates soon.

[eksperimental]

@josevalim I would say it is
https://reproducible-builds.org/docs/source-date-epoch/
https://reproducible-builds.org/specs/source-date-epoch/

I haven't fully read it, but it all looks it is a common issue.

[josevalim]

I found the root cause for Enum, String, Stream, which were relying on map ordering when sorting specs, which is non deterministic. This would have happened when we have more than 32 specs in the same module. Please see the commit above.

I still cannot reproduce the formatter one properly. I will provide SOURCE_DATE_EPOCH support for the System one.

Btw, is anyone interested in contributing a shell script we can run in CI that checks our builds are deterministic? The following would be needed:

1. when we run make compile for the first time, we need to set SOURCE_DATE_EPOCH
2. after we run our whole test suite and it passes, we copy the contents of lib/elixir/ebin to another directory and touch lib/elixir/lib/kernel.ex
3. run make compile again with the same SOURCE_DATE_EPOCH
4. compare the contents of the copied ebin with lib/elixir/ebin once again and see if there are any differences

[eksperimental]

One option would be to deprecate build_date() and change it to something else based on the last commit date. That way given it will be deterministic.

[eksperimental]

I can volunteer for contributing such script

[josevalim]

@eksperimental the problem is that people may not always be assembling from the git repo, so we are back to square one. I think SOURCE_DATE_EPOCH is a good compromise.

I can volunteer for contributing such script

Beautiful! 🎉

[josevalim]

PR for SOURCE_DATE_EPOCH here: #8694.

@bmwiedemann all should be fixed except the Code.Formatter one, which I cannot reproduce locally and I cannot figure out the root cause based on the diff you sent. Can you reproduce it on Elixir master?

Thanks for the feedback and guidance here!

[bmwiedemann]

Tested master with #8694 applied and it gives perfectly reproducible results. Yay!
Though it fails 1 test after 2019-11-01 => #8702

[josevalim]

All issues have been addressed. Thanks @bmwiedemann!

[bmwiedemann]

I wanted to mention that working with you guys was among the most fun cooperations I did - that is after doing 200+ upstream patches for reproducible builds. Nice github reaction-smiley, fast responses, good patches.
Keep that up :+1:

And I still know not a bit of elixir syntax ;-)