Elixir: Protocol implementations lost in umbrellas

[michalmuskala]

Environment

• Elixir & Erlang versions (elixir --version):

Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Elixir 1.5.0-dev (1822cdbc8)

• Operating system: macOS

Current behaviour

When working inside the umbrella and alternating between running mix tasks from the umbrella root and one of the applications, it seems that the compiler tends to "lose" some protocol implementations. I suspect this might be due to the fact that one application in the umbrella depends on a package that implements the protocol, while others don't. This most commonly happens with the String.Chars protocol for the Postgrex.Query type.

The issue is "fixed" by purging the _build folder and recompiling everything.

I tried creating a reproducible case multiple times but never managed to successfully create one - that's why I waited with reporting the issue, but maybe someone else will have better luck.
I face the issue regularly - sometimes even couple times a day. I've also seen other people repeatedly report similar problems. The issue is definitely present in 1.4 and current master, I'm not sure about earlier versions.

Expected behaviour

Protocols are consolidated correctly.

[josevalim]

@michalmuskala next time the issue happens, please try to run mix compile.protocols --force and let us know if that is enough to fix the issue.

[lukaszsamson]

I noticed the same problem of vanishing protocols for Plug.Exception implementations in Phoenix umbrella application. From time to time the protocol would not be found and the default error handler would be triggered. It made no difference wether the protocol implementation was in web or lib directory.
Forcing rebuild of the protocol by modifying the .ex file would temporarily resolve the issue on development machines. Unfortunately, the issue also occurred in build production releases (via distillery).
I had to refactor the app to not use error protocols at all.

Elixir versions: 1.3.4, 1.4.2
Os: Windows, OSX (run via mix), Linux (built release)

[wojtekmach]

I've just got this:

     ** (Protocol.UndefinedError) protocol Phoenix.HTML.FormData not implemented for #Ecto.Changeset<action: nil, changes: %{}, errors: [plan_financial_policy: {"can't be blank", []}], data: #Plan<>, valid?: false>

I got this when running tests from umbrella root after after a long rebasing session, so mtimes from different files were likely changed. I'm on Elixir 1.4.2/OTP 19.3.

MIX_ENV=test mix compile.protocols --force did fix the issue.

Btw, I didn't see this error until now for at least a few months, before that I saw it sporadically. I've used rm -rf _build but also learned before that just doing mix compile --force (which I assume does compile.protocols too) is enough.

[lukerollans]

I am not sure if this is very useful information, but I can reliably reproduce this anytime I write a new test which uses assert_raise. Forcing a recompile as above fixes the issue, of course.

I don't have any new error messages or code to show apart from what's already in #5989.

If I can help further at all, please let me know!

[josevalim]

Hi @lukerollans. Thank you for the mechanism to reproduce the issue, really really important! Just to make sure I won't miss anything, can you please provide the steps to reproduce? Like git clone, cd, mix test and so on.

[josevalim]

Hi everyone. Can anyone provide a mechanism to reproduce this? Otherwise there isn't much we can do to fix it, unfortunately.

[lukerollans]

I'll put aside some time after work tonight to try and create a repo with steps to reproduce

[lukerollans]

Tried to reproduce for a couple of hours with a new umbrella app, was un-successful. Will attempt to do so again shortly

[josevalim]

@lukerollans if you can consistently reproduce in your project, I would also be fine to do it over a Skype session or if you give me access (which you can revoke as soon as it is fixed). :)

[lukerollans]

@josevalim Just heading to bed now, tomorrow I'll jump back in to that project and see if it's still easy for me to reproduce. If so, that might be an option!

[kelvinst]

Well, I have one example here that repeats it easily. Maybe it can help in this issue.

Just run the following commands to download and start the server I've built as an example repeating the error:

git clone [email protected]:kelvinst/prototest_umbrella.git
cd prototest_umbrella
mix deps.get
(cd apps/prototest && mix compile)
mix phx.server

Then open http://localhost:4000 and you should see the protocol not implemented error. The reason is quite obvious as you can see here in the lib where it's happening: we compiled the code for the app prototest, but then, when we loaded prototest_web, as the money lib protocol implementation was not recompiled, and we have a compilation time if there, it used the already consolidated Phoenix.HTML.Safe protocol, without the implementation for Money.

If we force a full recompilation of the protocols from inside umbrella's root, the page works correctly.

This is an easy reproduction of the error with no dependency on ecto and other databases, but you can see here that phoenix_ecto's problem (mentioned by @wojtekmach) is quite the same as money's.

[michalmuskala]

I don't think ensure_compiled? is the culprit. I notice the error most of the time with String.Chars protocol, which is built-in and never subject to a conditional compilation.

The reproduction repo is awesome!

[kelvinst]

Well, String.Chars itself might not be subject to a conditional compilation, but maybe the implementations that fail for you are. Such as the errors I mentioned, what is subject to the conditional compilations there was the defimpl for that protocols.

Anyways, I'll try to repeat your String.Chars errors locally, I have a theory, and if I'm right I'll get in touch explaining it soon.

[kelvinst]

@michalmuskala No success repeating your problem. :/

Sorry for the bothering, just thought my problems could be related.

[wojtekmach]

@kelvinst nice one, I tried your repo and can reproduce the error. On work project, I'm able to reliably reproduce the error with Phoenix.HTML.FormData for Ecto.Changeset.

While trying to get the minimal report case, I got the String.Chars one; instructions in the repo: https://github.com/wojtekmach/umbrella_proto_bug

[wojtekmach]

I verified that the fix works on both work project as well as minimal repro! 🎉

[aikrikunov95]

How can I apply the fix on my computer?
*I have installed elixir from precompiled package on Ubuntu.

[whatyouhide]

@aikrikunov95 this will be released when Elixir v1.5 is released (which should be soon™). There's a chance Elixir v1.4.5 will be released as well and maybe this can be backported but in both cases, the solution will be to wait for a new version to be out. Alternatively, you can clone the Elixir repo and compile Elixir yourself or you can use something like kiex.

[fishcakez]

We will definitely have a v1.4.5 as v1.4.4 has logging issues on OTP 20.0-rc2.

[josevalim]

It is unlikely we will backport this fix though because it changes the consolidation path and that may get some people by surprise if they were not relying on Mix.Project.consolidation_path. We need to be very conservative with changes on patch branches because if we need to do a security release, everyone needs to jump to the latest patch version without breaking anything at all.

[DavidOliver]

I seem to have come across this issue when running tests for an app in an umbrella project (Phoenix 1.3-generated) in Elixir 1.5.1.:

** (Protocol.UndefinedError) protocol Phoenix.HTML.FormData not implemented for #Ecto.Changeset...

Deleting _build worked around it, and things seem to be fine again. Let me know if I should provide more information or share the project privately. Thanks.

[chulkilee]

I hit this problem too.

• Ubuntu 16.04
• Elixir 1.5.2 / Erlang/OTP 20 (installed esl-erlang and elixir from erlang solution repo)
• mix 1.5.2
• distillery 1.5.2
• ecto 2.2.6
• postgrex 0.13.3

mix compile.protocols --force
** (exit) an exception was raised:
    ** (Protocol.UndefinedError) protocol String.Chars not implemented for %Postgrex.Query{columns: ["status", "count"], name: "ecto_162", param_formats: [], param_oids: [], param_types: [], ref: #Reference<0.3129991069.1824784386.259765>, result_formats: [:binary, :binary], result_oids: [1043, 20], result_types: [Postgrex.Extensions.Raw, Postgrex.Extensions.Int8], statement: "SELECT o0.\"status\", count(o0.\"id\") FROM \"XXX\" AS o0 GROUP BY o0.\"status\"", types: {Ecto.Adapters.Postgres.TypeModule, #Reference<0.3129991069.1824915457.259840>}}. This protocol is implemented for: Atom, BitString, Date, DateTime, Decimal, Ecto.Date, Ecto.DateTime, Ecto.Time, Float, Integer, List, NaiveDateTime, Time, URI, Version, Version.Requirement
        (elixir) lib/string/chars.ex:3: String.Chars.impl_for!/1
        (elixir) lib/string/chars.ex:22: String.Chars.to_string/1
        (ecto) lib/ecto/adapters/sql.ex:613: Ecto.Adapters.SQL.log/4
        (db_connection) lib/db_connection.ex:1178: DBConnection.log/6
        (ecto) lib/ecto/adapters/postgres/connection.ex:73: Ecto.Adapters.Postgres.Connection.prepare_execute/5
        (ecto) lib/ecto/adapters/sql.ex:256: Ecto.Adapters.SQL.sql_call/6
        (ecto) lib/ecto/adapters/sql.ex:426: Ecto.Adapters.SQL.execute_and_cache/7
        (ecto) lib/ecto/repo/queryable.ex:133: Ecto.Repo.Queryable.execute/5

I switched branches back and forth - most branches have ecto in dep (since some apps are using Ecto.Schema for API request building), only one branch has postgrex (phoenix app). As a result at some point I had to recomplile ecto to pick up postgrex on compile time. Don't remember how I did that..

Anyway, mix compile.protocols --force didn't fix the problem. Removing _build and rebuilding from scratch fixed the problem.

[LukeWood]

I actually had this same problem when building a release inside docker.

details: I was using https://github.com/jamesotron/heap and the Enumerable protocol was being lost.

Adding mix compile.protocols --force to my list of mix commands fixed the problem.

[esambo]

I had the same (Protocol.UndefinedError) protocol Phoenix.HTML.FormData not implemented for #Ecto.Changeset error. I added the _phoenix_ecto_ while inside one of the umbrella child apps, but running mix test in the umbrella parent app (outside of the apps directory) still causes that error.

My environment:

• erlang 20.1.7
• elixir 1.5.2
• phoenix 1.3.0
• ecto 2.2.7
• phoenix_ecto 3.3.0

Following did NOT resole it:

• mix compile.protocols --force
• mix compile --force
• mix clean

My Fix

I can fix it with:

• rm _build/test/consolidated/Elixir.Phoenix.HTML.FormData.beam

I have attached a zip file that includes both the _bad_ (that errors, before I deleted it) and _good_ (after iI regenerated it) beam file:

• Archive.zip

[wojtekmach]

@esambo try upgrading to Elixir v1.6. I haven't seen this particular bug for a really long time.

[r6203]

I've run into the same problem on Elixir 1.6.4.

mix compile.protocols --force fixed it.

[josevalim]

Folks, I am sorry to hear the problem still exists in some cases but we really need a mechanism to reproduce it, otherwise there isn't much we can do.

Also please try to include the operating system and the OTP version on reports.

[mrkaspa]

I'm having the same problem I'm making a crud but the models live in another project in the umbrella

[drazafsky]

I recently ran into this issue on a new Phoenix 1.3.2 umbrella project. Similar to @mrkaspa my models live in a separate repo from my phoenix app. Unlike others I wasn't able to fix the issue by removing _build or forcing protocols to recompile.

Erlang/OTP 19 [erts-8.2] [source] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]

My application was created with the --no-ecto and --no-brunch options. I then created a new ecto app with mix phx.new.ecto.

I was able to solve the issue by adding {:phoenix_ecto, "~>3.3.0"} to my ecto app's deps.

[tcoopman]

I've had a similar error with path dependencies (not an umbrella project) on: elixir 1.7.0-otp-21, on ArchLinux.

mix compile.protocols --force did not fix it.

removing the _build/test folder fixed it.

I've been able to reproduce the error, by renaming my module. After the rename, I got the same error about the missing protocol. When I undid the rename, everything worked again.

[josevalim]

Thanks @tcoopman! Can you please provide a small application that reproduces the error? It should be a matter of running mix new umbrella --umbrella; cd umbrella/apps; mix new a; mix new b and defining a protocol. Then please provide the steps to reproduce the error (like compile from root, rename the protocol, compile from root).

Those will steps will help us a lot in making sure we have the correct fix for this.

Thank you!

[tcoopman]

I'll see if I can reproduce it this weekend on a small repo.

[tcoopman]

ok, it was actually easy to reproduce.

see: https://github.com/tcoopman/elixir-protocol-bug

steps to reproduce:

1. clone the repo
2. cd b
3. mix test (this works)
4. in a/lob/first.ex rename Foo to something else, also in b/test/b_test.exs
5. mix test and now you get this error

 > mix test                                                                                                                         13.9s  Fri 27 ==> a
Compiling 1 file (.ex)
Generated a app
==> b


  1) test bug (BTest)
     test/b_test.exs:4
     ** (Protocol.UndefinedError) protocol A.BugProtocol not implemented for %A.Bar{}. This protocol is implemented for: A.Foo
     code: assert "works" == A.BugProtocol.foo(A.Bar.create())
     stacktrace:
       (a) /home/thomas/Workspace/elixir/elixir-protocol-bug/a/lib/a.ex:1: A.BugProtocol.impl_for!/1
       (a) /home/thomas/Workspace/elixir/elixir-protocol-bug/a/lib/a.ex:2: A.BugProtocol.foo/1
       test/b_test.exs:5: (test)



Finished in 0.02 seconds
1 test, 1 failure

Randomized with seed 684346

mix compile.protocols --force does not fix it (not when running in a, neither in b). Removing _build/test fixes it.

[josevalim]

@tcoopman fantastic.

[josevalim]

Closing in favor of #8037. Thanks @tcoopman for the app, it helped a lot! :heart: :green_heart: :blue_heart: :yellow_heart: :purple_heart: