Elixir: Starting releases is slow

[tcoopman]

Environment

• elixir 1.9.1-otp-21
• OS: Linux (ubuntu and archlinux)

Current behavior

Starting an application build with the build in mix release from elixir 1.9.1 is way slower than starting the same application build with distillery.

Unfortunately, I don't have a public test case, but I did create some measurements. Every time I measured the time to entry in Application.start:

## Distillery (elixir 1.8.2-otp-21)
- date && _build/prod/rel/app/bin/app foreground: < 3 seconds
- date && _build/prod/rel/app/bin/app console: < 3 seconds

## Distillery (elixir 1.9.1-otp-21)
- date && _build/prod/rel/app/bin/app foreground: < 3 seconds
- date && _build/prod/rel/app/bin/app console: < 3 seconds

## Elixir releases (elixir 1.9.1-otp-21)
- date && _build/prod/rel/app/bin/app start: 10 seconds
- date && _build/prod/rel/app/bin/app start_iex: 10 seconds

## env MIX_ENV=prod mix phx.server --preload-modules (elixir 1.9.1-otp-21)
- date && env MIX_ENV=prod mix phx.server --preload-modules: < 3 seconds

## Experiment: Elixir releases without runtime configuration (elixir 1.9.1-otp-21)
- date && _build/prod/rel/app/bin/app start: < 3 seconds

As you can see, using regular releases with runtime configuration are way slower than distillery. I have the feeling that even without runtime configuration they are still slower, but that's hard to measure. There is a post on elixirforum where others are reporting the same issue (https://elixirforum.com/t/elixir-1-9-release-slow-boot-time/) and pointing out that not including runtime configuration did not fix this for them (although it improved a bit)

Having about 10 seconds for a fairly simple application to before it starts is a huge overhead. It would be nice if we had similar startup performance as with distillery.

[josevalim]

Can you please provide an app that reproduces the issue? You can try to create a minimal app or get an existing project like hex.pm. Otherwise, if we cannot reproduce it, there isn't much we can do to fix it. Thanks.

[josevalim]

Fwiw, I just tried it on hex.pm and it booted in less than 3 seconds.

[tcoopman]

I'll see if I can find the time to thinker with a public app to reproduce the problem.

[tcoopman]

Ok, I've taken the first open source app I could find and added distillery and elixir releases to it: https://github.com/tcoopman/elixir-phoenix-realworld-example-app

I didn't actually test the app itself, but timed until the application starts

Some observations:

• If the environment variables needed in releases.exs are not set, the startup fails very fast.
• If the environment variables are set, the startup is slow
• The distillery startup is fast

## mix releases
date +"%T.%N" && _build/prod/rel/real_world/bin/real_world start                                                                                                                                                                 
17:07:37.166014070
~U[2019-09-10 15:07:42.160030Z]

## distillery
date +"%T.%N" && _build/prod/rel/real_world/bin/real_world foreground                                                                                                                                                     
17:08:47.284464367
~U[2019-09-10 15:08:48.371708Z]

So for mix releases about 5 seconds and for distillery about 1 second.

[josevalim]

In this example app, it is much faster without the config/releases.exs file but you said removing the config file has no effect on the original report. Can you please confirm?

[tcoopman]

Sorry if I wasn’t clear.
For me it is also faster without releases.exs. But someone else reported that it was still slow without releases.exs. I’ll ask him to comment here with more details.

[josevalim]

I think I was the one who misread it then :) Thanks @tcoopman . I have identified the root cause, I know how to fix it. The issue is that preloading and unloading and then preloading all apps take quite some time. We should attempt to invoke the config provider sooner.

[josevalim]

Good news and bad news.

The bad news first: in order to fix this reliably, we will have to send a patch to Erlang/OTP.

The good news: you can get fast boot times today by doing RELEASE_MODE=interactive. This means module are not preloaded upfront though.

[josevalim]

PR to Erlang/OTP sent here: https://github.com/erlang/otp/pull/2461

Once the PR is merged, the plan is to start the release in interactive mode, until the config provider restarts the system in embedded mode. This way we get the fast boot times until configuration and the slow one when effectively starting the system.

[josevalim]

Closing in favor of #10091.