Elixir: generate one new module and one atom per compilation during runtime

[redink]

https://github.com/elixir-lang/elixir/blob/d8a533d19f3a016512f605ac125e2709c6bd74ae/lib/elixir/src/elixir_code_server.erl#L132

Environment

$ iex
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 

Our use case

We using beam as a specific cache for database, we need compile some given modules to load the latest data from db to beam in interval time during runtime. Using:

Kernel.ParallelCompiler.files_to_path(source_files, beam_path)

Current behavior

create one new module and one atom per compilation, the atom just like:

elixir_compiler_8
elixir_compiler_7
elixir_compiler_6
elixir_compiler_5
elixir_compiler_4
elixir_compiler_3
elixir_compiler_2

and the module just like:

=mod:elixir_compiler_0
Current size: 0
Old size: 5193
Old attributes: 836C00000001680264000376736E6C000000016E1000E98FE57BEC2C0B4635218BB1ED55D6CF6A6A
Old compilation info: 836C0000000368026400076F7074696F6E736C0000000164000E6E6F7761726E5F6E6F6D617463686A680264000776657273696F6E6B0005372E312E326802640006736F757263656B005D2F6F70742F747562692F636D735F736572766963652F6C69622F706F6C6963795F656E67696E652D342E312E35302F7372632F6C69622F706F6C6963795F656E67696E652F67656E2F67656E5F7365726965735F706F6C6963792E65786A
=mod:elixir_compiler_12
Current size: 0
Old size: 9748
Old attributes: 836C00000001680264000376736E6C000000016E1000DAC1E389C40641D4E01D37111B6418976A6A
Old compilation info: 836C0000000368026400076F7074696F6E736C0000000164000E6E6F7761726E5F6E6F6D617463686A680264000776657273696F6E6B0005372E312E326802640006736F757263656B00602F6F70742F747562692F636D735F736572766963652F6C69622F706F6C6963795F656E67696E652D342E312E35302F7372632F6C69622F706F6C6963795F656E67696E652F67656E2F67656E5F636F6E7461696E65725F706F6C6963792E65786A
=mod:elixir_compiler_4
Current size: 0
Old size: 9027
Old attributes: 836C00000001680264000376736E6C000000016E10009D8CADBE367736B8F3FAFE93BDEF7E5C6A6A
Old compilation info: 836C0000000368026400076F7074696F6E736C0000000164000E6E6F7761726E5F6E6F6D617463686A680264000776657273696F6E6B0005372E312E326802640006736F757263656B00512F6F70742F747562692F636D735F736572766963652F6C69622F706F6C6963795F656E67696E652D342E312E35302F7372632F6C69622F706F6C6963795F656E67696E652F736C696365732F6E362E65786A

just because there is limitation for module amount in Erlang, the Elixir/Erlang node will be crash when the module amount reach the limitation, the crash dump slogan is:

no more index entries in module_code (max=65536)

the detailed information is:

=hash_table:module_code
size: 51437
used: 48890
objs: 65536
depth: 3
=index_table:module_code
size: 65536
limit: 65536
entries: 65536

Attempt

I tried to modify the Erlang vm args just as ERL_MAX_PORTS, BUT I failed.

Because the limitation for module amount can't modify in Erlang, it is hard-code in:

https://github.com/erlang/otp/blob/master/erts/emulator/beam/module.c#L38

Thanks.

[redink]

I noticed the logic for retrieve_compiler_module and return_compiler_module are complete. It will retrieve the compiler module when compile, and will return after compile.

dispatch(Module, Fun, Args, Purgeable, I, E) ->
  Res = Module:Fun(Args),
  code:delete(Module),
  if Purgeable ->
      code:purge(Module),
      return_compiler_module(I);
     true ->
       ok
  end,
  {Res, E}.

https://github.com/elixir-lang/elixir/blob/d8a533d19f3a016512f605ac125e2709c6bd74ae/lib/elixir/src/elixir_compiler.erl#L91

But, why not return when can't purge the module ?

[josevalim]

We can't return because we can't purge the module. If we return a non-purged module, it will be redefined later on, which then would cause the purging possibly breaking existing code. You rather need to figure out why your modules are not considered as purgeable. You can get the beam from the disk and call this:

beam_lib:chunks(Binary, [labeled_locals])

If you have labelled locals, i.e. local anonymous functions in the module body, then they can't be purged.

[redink]

Thanks for reply.

Just want to know, why anonymous functions will affect the purge ?

( I only know that anonymous functions maybe lead failed when hot upgrade, but I don't know why clearly.

[josevalim]

iex(1)> :beam_lib.chunks(Test, [:labeled_locals])

You need to run the command above for the :elixir_compiler_1 modules and so on. If the internal modules have functions, it means they can spawn processes and purging those modules mean they will crash.

If you can find an :elixir_compiler_X that has labelled locals, you can then call :elixir_compiler_X.__FILE__ or :elixir_compiler_X.__MODULE__ and it will tell you if a file or module are generating the unpurgeable modules. If you paste them here, I can help you figure out why they can't be purged.

[redink]

I can find an :elixir_compiler_X, but I can't call functions :elixir_compiler_X.__FILE__ and :elixir_compiler_X.__MODULE__, because :elixir_compiler_X is not available.

Just like:

iex(1)> ReconTrace.calls([{:beam_lib, :_, fn _ -> :return end}], 100)
26
iex(2)> Kernel.ParallelCompiler.files_to_path(["test.ex"], "./")

22:51:41.714382 <0.1445.0> :beam_lib.chunks(<<70, 79, 82, 49, 0, 0, 8, 60, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 1, 160, 0,
  0, 0, 32, 17, 101, 108, 105, 120, 105, 114, 95, 99, 111, 109, 112, 105, 108,
  101, 114, 95, 48, 10, 95, 95, 77, 79, 68, 85, 76, ...>>, [:labeled_locals])

22:51:41.726182 <0.1445.0> :beam_lib.chunks/2 --> {:ok,
 {:elixir_compiler_0, [labeled_locals: [{:"-__MODULE__/1-fun-0-", 1, 10}]]}}
[OtherModule]
iex(3)> 
nil
iex(4)> :elixir_compiler_0.module_info
** (UndefinedFunctionError) function :elixir_compiler_0.module_info/0 is undefined (module :elixir_compiler_0 is not available)
    :elixir_compiler_0.module_info()

I think this due to:

https://github.com/elixir-lang/elixir/blob/d8a533d19f3a016512f605ac125e2709c6bd74ae/lib/elixir/src/elixir_compiler.erl#L86

[josevalim]

@redink oh, you can try removing that line and running make erlang and then run against your local Elixir.

[josevalim]

@fishcakez suggested a less conservative approach for purging modules which will likely solve your problem. Please give master a try and let us know!

[redink]

during compilation, the state of elixir code server is:

iex(58)> :sys.get_state(:erlang.whereis(:elixir_code_server))
{:elixir_code_server, %{
   "/usr/local/bin/mix" => true
 }, {[7, 4], 8}, %{
   #Reference<0.3414257687.589824008.147734> => M1,
   #Reference<0.3414257687.589824008.147738> => M2,
   #Reference<0.3414257687.590086151.234246> => M3,
   #Reference<0.3414257687.590610436.53835> => M4,
   #Reference<0.3414257687.591396866.54944> => M5,
   #Reference<0.3414257687.591396866.54960> => M6,
   #Reference<0.3414257687.591396866.188095> => M7,
   #Reference<0.3414257687.591396869.7197> => M8
 }, %{#PID<0.586.0> => :ok}}

after compile, the state is:

iex(84)> :sys.get_state(:erlang.whereis(:elixir_code_server))
{:elixir_code_server, %{
   "/usr/local/bin/mix" => true
 }, {[1, 0, 2, 3, 6, 5, 4, 7], 8}, %{}, %{}}

after some times retry, the counter of mod_pool didn't increase.

so, I think it works.

Thanks all of you again.

[josevalim]

The approach we chose here is causing some false positives. We will have to revert this or come up with another strategy.

[josevalim]

To reproduce the issue, create a new project with those two files:

spawn(fn ->
  Enum.map(1..5, fn i ->
    Process.sleep(i * 1000)
  end)
end)

and

Process.sleep(10000)

and then compile it.

[josevalim]

This commit has been reverted in master and v1.6. /cc @Gazler

[michalmuskala]

Is this an issue with how this tried to use code:soft_purge or is that issue with code:soft_purge itself?

[josevalim]

soft_purge itself. From OTP 20 the module will be considered as purged even if it has anonymous functions lying around. When invoking those functions we get badfun/badarg. This is exactly what we were seeing.

[fishcakez]

I think we will need another solution to this issue.

[josevalim]

I am not sure we have a solution that allows us to properly garbage collect those modules/atoms. We could however allow explicit intervention:

1. Provide a Code.reclaim_compilation_modules/0 that purges all pending modules. Mix could automatically invoke this function after compiling each project.

2. Have a threshold that, once reached, logs a warning saying that N modules are laying around in memory. A default value could be 10000 entries.

[michalmuskala]

One way to detect if there are processes running code from the file/module compilation modules would be to hijack the group leader infrastructure in a similar way that application controller does.

For each compilation module we spin up a "guardian" process and then run the compilation function in a new process that has the guardian as group leader. If, after compilation is finished, there's any process with that group leader, we know we can't purge.

It's not a 100% foolproof method, but given it's good enough for OTP and application controller, it should be good enough here.

[redink]

Thanks, I will test it in our env asap.

[redink]

As far as I know, after upgraded 1.7.x the compiler works very well than before:

the speed for each compile improved significantly:

and, the atom amount not increase after compile:

Thanks again.