Elixir: Compilation of the dependencies is broken

[umatomba]

Precheck

Environment

• Elixir & Erlang versions (elixir -v): Elixir 1.4.0 & Erlang/OTP 19
• Operating system: Ubuntu 16.04.1 LTS

Current behavior

Mix is not able to compile some test application. Previously working on Elixir 1.3.4.

mix deps.get
MIX_DEBUG=1 mix compile

** Running mix loadconfig (inside Checkel.Mixfile)
** Running mix compile (inside Checkel.Mixfile)
** Running mix loadpaths (inside Checkel.Mixfile)
** Running mix archive.check (inside Checkel.Mixfile)
** Running mix deps.loadpaths (inside Checkel.Mixfile)
** Running mix deps.precompile (inside Checkel.Mixfile)
===> Compiling bear
===> Compiling src/bear.erl failed
src/bear.erl:none: undefined parse transform 'lager_transform'

** (Mix.Error) Could not compile dependency :bear, "/home/vtarasen/.mix/rebar3 bare compile --paths "/home/vtarasen/dev/CURRENT/checkel/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile bear", update it with "mix deps.update bear" or clean it with "mix deps.clean bear"
    (mix) lib/mix.ex:276: Mix.raise/1
    (elixir) lib/file.ex:1162: File.cd!/2
    (mix) lib/mix/tasks/deps.compile.ex:218: Mix.Tasks.Deps.Compile.do_command/5
    (mix) lib/mix/tasks/deps.compile.ex:80: anonymous fn/4 in Mix.Tasks.Deps.Compile.compile/2
    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
    (mix) lib/mix/tasks/deps.compile.ex:65: Mix.Tasks.Deps.Compile.compile/2
    (mix) lib/mix/tasks/deps.loadpaths.ex:83: Mix.Tasks.Deps.Loadpaths.deps_check/2
    (mix) lib/mix/tasks/deps.loadpaths.ex:27: Mix.Tasks.Deps.Loadpaths.run/1

One change i see after upgrade to Elixir 1.4 is that the dependencies previously compiled with rebar are compiled with rebar3 now. For example in bear the _build/dev/lib/bear/mix.rebar.config contains the couple of options not related to bear, for example erl_opts are from parent dependency. The same with deps (is it a plan?):

[eunit_opts: [:verbose], plugins: [],
 pre_hooks: [compile: './dicts_compiler.erl compile',
  clean: './dicts_compiler.erl clean'], cover_export_enabled: true,
 erl_first_files: ['src/exometer_igor.erl', 'src/exometer_util.erl',
  'src/exometer_entry.erl', 'src/exometer_probe.erl'],
 edoc_opts: [doclet: :edown_doclet,
  app_default: 'http://www.erlang.org/doc/man', doc_path: [],
  top_level_readme: {'./README.md',
   'https://github.com/Feuerlabs/exometer_core', 'master'}],
 xref_checks: [:undefined_function_calls, :undefined_functions,
  :locals_not_used, :deprecated_functions_calls, :deprecated_functions],
 cover_print_enabled: true, clean_files: ['test/app1/ebin/*.beam'],
 sub_dirs: ['deps', 'src'],
 deps: [{:lager, '.*', {:git, 'git://github.com/basho/lager', {:tag, '3.2.2'}}},
  {:meck, '.*',
   {:git, 'git://github.com/eproxus/meck.git', {:branch, 'master'}}},
  {:coveralls, '.*',
   {:git, 'git://github.com/markusn/coveralls-erl.git', {:branch, 'master'}}},
  {:exometer_core, '.*',
   {:git, 'git://github.com/Feuerlabs/exometer_core', {:branch, 'master'}}}],
 erl_opts: [:debug_info, {:parse_transform, :lager_transform},
  {:verbosity, :trace}], cover_enabled: true]

Application to reproduce the effect:
https://github.com/umatomba/checkel

Expected behavior

The expected behavior is a successful compilation.

[josevalim]

@umatomba I believe you can make it work by specifying manager: :rebar in the dependency that cannot be compiled properly on rebar 3. Ideally you would fix it in the child project that depends on the broken dependency but it can be done from your project by specifying the dependency with manager: :rebar and the overrides option.

This is very tricky because, if we don't default to rebar 3, recently published packages in Hex does not work by default. But if we default to rebar 3, old packages may not work.

[umatomba]

@josevalim sure, if I will specify the manager as :rebar it will work, but not because the failed deps don't support rebar3. bear and coverall in my example can be compiled with rebar3 directly:

[ bear]$ /home/user/.mix/rebar3 clean
===> Cleaning out bear...
[ bear]$ /home/user/.mix/rebar3 compile
===> Verifying dependencies...
===> Compiling bear

It looks like some problem in the mix support of the rebar3 logic.

[josevalim]

@umatomba I see, that's really helpful, as well as the reproducing example. Thank you!

[keyan]

Not sure if my issue is entirely related, but I am getting this failure during compilations (during Docker image builds) when upgrading to Elixir 1.4.0:

Could not find "rebar", which is needed to build dependency :poolboy

I can install a local copy which is just used by Mix

Shall I install rebar? (if running non-interactively, use: "mix local.rebar --force") [Yn] 
�[91m** (Mix) Could not find "rebar" to compile dependency :poolboy, please ensure "rebar" is available
�[0m
Removing intermediate container dd11b4454e73

The command '/bin/sh -c MIX_ENV=prod mix compile' returned a non-zero code: 1

I haven't spent long trying to triage, for now I have just gone back to 1.3.4.

[josevalim]

@keyan can you please explicitly run mix local.rebar inside the docker image verify that executables have been added to ~/.mix/rebar? It may be the case of wrong permissions or Elixir being unable to download rebar.

[keyan]

@josevalim Thanks for your response. Running mix local.rebar seems to have been enough to resolve the compilation failure. I don't quite understand why it needs to be included explicitly though.

For the record, here is my (slightly edited) Dockerfile:

FROM elixir

ENV APP_NAME app
EXPOSE 4000

# Setup local codebase
RUN mkdir -p /$APP_NAME
COPY . /$APP_NAME

WORKDIR /$APP_NAME
RUN mix local.hex --force
RUN mix local.rebar
RUN mix deps.get
RUN MIX_ENV=prod mix compile

# Start exworker
CMD MIX_ENV=$ENVIRONMENT elixir --sname $APP_NAME -S mix run --no-halt

And output from the added step:

Step 10 : RUN mix local.rebar
 ---> Running in aafc1aa4c62f
* creating /root/.mix/rebar
* creating /root/.mix/rebar3

Thanks again 😄

[ericmj]

I don't quite understand why it needs to be included explicitly though.

It could be that mix local.rebar installed a newer version of rebar3 that had some bug fix, but hard to say for sure. Glad it's resolved now.

[umatomba]

@ericmj, The problem is not solved for me. mix local.rebar contains the latest rebar version as before.
You can test it

docker run -it --rm elixir:1.4.0 bash

mix local.rebar

/root/.mix/rebar -V
    rebar 2.6.2 R15B03 20160628_123811 git 2.6.2

/root/.mix/rebar3 -v
    rebar 3.2.0 on Erlang/OTP 19 Erts 8.2

git clone https://github.com/umatomba/checkel

mix deps.get

mix deps.compile
===> Compiling src/bear.erl failed
src/bear.erl:none: undefined parse transform 'lager_transform'

** (Mix) Could not compile dependency :bear, "/root/.mix/rebar3 bare compile --paths "/home/checkel/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency wi
th "mix deps.compile bear", update it with "mix deps.update bear" or clean it with "mix deps.clean bear"

cat _build/dev/lib/bear/mix.rebar.config
{eunit_opts,[verbose]}.
{plugins,[]}.
{pre_hooks,[{compile,"./dicts_compiler.erl compile"},
           {clean,"./dicts_compiler.erl clean"}]}.
{cover_export_enabled,true}.
{erl_first_files,["src/exometer_igor.erl","src/exometer_util.erl",
                 "src/exometer_entry.erl","src/exometer_probe.erl"]}.
{edoc_opts,[{doclet,edown_doclet},
           {app_default,"http://www.erlang.org/doc/man"},
           {doc_path,[]},
           {top_level_readme,{"./README.md",
                              "https://github.com/Feuerlabs/exometer_core",
                              "master"}}]}.
{xref_checks,[undefined_function_calls,undefined_functions,locals_not_used,
             deprecated_functions_calls,deprecated_functions]}.
{cover_print_enabled,true}.
{clean_files,["test/app1/ebin/*.beam"]}.
{sub_dirs,["deps","src"]}.
{deps,[{lager,".*",{git,"git://github.com/basho/lager",{tag,"3.2.2"}}},
      {meck,".*",{git,"git://github.com/eproxus/meck.git",{branch,"master"}}},
      {coveralls,".*",
                 {git,"git://github.com/markusn/coveralls-erl.git",
                      {branch,"master"}}},
      {exometer_core,".*",
                     {git,"git://github.com/Feuerlabs/exometer_core",
                          {branch,"master"}}}]}.
{erl_opts,[debug_info,fail_on_warning,
          {parse_transform,lager_transform},
          {verbosity,trace}]}.
{cover_enabled,true}.

[ericmj]

This seems to be a bug in elixir. I will investigate.

[umatomba]

@ericmj, thank you

[tony612]

Got the same problem as the above issue "Error compiling hpack" shows.

[keyan]

I think my earlier issue was unrelated to this bug, as I am now seeing something more in line with OP:

** (Mix) Could not compile dependency :setup, "/root/.mix/rebar3 bare compile --paths "/api/_build/prod/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile setup", update it with "mix deps.update setup" or clean it with "mix deps.clean setup"

[xadhoom]

Seems I'm experiencing a very similar issue using :can package, as discussed here
https://elixirforum.com/t/undefined-parse-transform-lager-transform-with-elixir-1-4-0/

within my test repo https://github.com/xadhoom/can_test I have the very same error,
which is "fixed" if I force manager: :rebar

Same as @umatomba reports, I can compile :can using rebar3, but the mix.rebar.config of the first dep contains parameters not related to it, but to a child dep.

[josevalim]

I have investigate this a bit and it seems it is happening because we are accidentally passing the parent config to the child. Here is how the bear dependency looks like:

%Mix.Dep{app: :bear, deps: [],
 extra: [eunit_opts: [:verbose], plugins: [],
  pre_hooks: [compile: './dicts_compiler.erl compile',
   clean: './dicts_compiler.erl clean'], cover_export_enabled: true,
  erl_first_files: ['src/exometer_igor.erl', 'src/exometer_util.erl',
   'src/exometer_entry.erl', 'src/exometer_probe.erl'],
  edoc_opts: [doclet: :edown_doclet,
   app_default: 'http://www.erlang.org/doc/man', doc_path: [],
   top_level_readme: {'./README.md',
    'https://github.com/Feuerlabs/exometer_core', 'master'}],
  xref_checks: [:undefined_function_calls, :undefined_functions,
   :locals_not_used, :deprecated_functions_calls, :deprecated_functions],
  cover_print_enabled: true, clean_files: ['test/app1/ebin/*.beam'],
  sub_dirs: ['deps', 'src'],
  deps: [{:lager, '.*',
    {:git, 'git://github.com/basho/lager', {:tag, '3.2.2'}}},
   {:meck, '.*',
    {:git, 'git://github.com/eproxus/meck.git', {:branch, 'master'}}},
   {:coveralls, '.*',
    {:git, 'git://github.com/markusn/coveralls-erl.git', {:branch, 'master'}}},
   {:exometer_core, '.*',
    {:git, 'git://github.com/Feuerlabs/exometer_core', {:branch, 'master'}}}],
  erl_opts: [:debug_info, :fail_on_warning,
   {:parse_transform, :lager_transform}, {:verbosity, :trace}],
  cover_enabled: true], from: "/Users/jose/OSS/checkel/rebar.config",
 manager: :rebar3,
 opts: [lock: {:git, "git://github.com/boundary/bear.git",
   "119234548783af19b8ec75c879c5062676b92571", [tag: "0.8.2"]}, env: :prod,
  checkout: "/Users/jose/OSS/checkel/deps/bear",
  build: "/Users/jose/OSS/checkel/_build/test/lib/bear",
  dest: "/Users/jose/OSS/checkel/deps/bear",
  git: "git://github.com/boundary/bear.git", tag: "0.8.2"], requirement: ~r/.*/,
 scm: Mix.SCM.Git, status: :compile, top_level: false}

Notice it has a bunch of information leaked from exometer_core. We will continue investigating.

[josevalim]

Here is a PR that fixes the issue: if folks can give it a try, it will be appreciated: https://github.com/elixir-lang/elixir/pull/5687

[josevalim]

FWIW, it works against both apps provided.

[xadhoom]

For me is ok, it works on both the test app and on my main app.

Great, thanks!

[tony612]

I still have the error using the PR:

===> Compiling hpack
===> Compiling src/huffman.erl failed
src/huffman.erl:none: undefined parse transform 'lager_transform'

** (Mix) Could not compile dependency :hpack, "/Users/tony/.mix/rebar3 bare compile --paths "/Users/tony/.../sabre/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile hpack", update it with "mix deps.update hpack" or clean it with "mix deps.clean hpack"

And I found that it works(including the stable Elixir 1.4) after changing deps from this

    [{:grpc, github: "tony612/grpc-elixir"}]

to this

    [
      {:grpc, github: "tony612/grpc-elixir"},
      {:hpack, github: "joedevivo/hpack", override: true},
      {:goldrush, override: true} # without this, the error will be in goldrush instead of hpack
    ]

My test project only adds grpc as a dependence. This deps.tree is:

sabre
└── grpc (https://github.com/tony612/grpc-elixir.git)
    ├── exprotobuf ~> 1.2.0 (Hex package)
    │   └── gpb ~> 3.24 (Hex package)
    ├── chatterbox (https://github.com/tony612/chatterbox.git)
    │   ├── lager ~r/2.2.3/ (git://github.com/basho/lager.git)
    │   │   └── goldrush ~r/.*/ (git://github.com/DeadZen/goldrush.git)
    │   └── hpack ~r/.*/ (git://github.com/joedevivo/hpack.git)
    └── cowboy (https://github.com/tony612/cowboy.git)
        ├── cowlib ~r/.*/ (https://github.com/ninenines/cowlib)
        └── ranch ~r/.*/ (https://github.com/ninenines/ranch)

[tony612]

Sorry, I used the wrong mix. It works for me now. Thanks!

btw, mix will be 1.4 by using bin/mix directly. So I have to change elixir path in bin/mix

[umatomba]

Now it is correct.
_build/dev/lib/bear/mix.rebar.config:

{deps,[]}.
{erl_opts,[debug_info]}.
{cover_enabled,true}.
{overrides,[]}.

thanks!