Absinthe: Error after introducing a subscription: badfun

[samsondav]

If submitting a bug, please provide the following:

Environment

• Elixir version (elixir -v):

Erlang/OTP 20 [erts-9.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Elixir 1.6.3 (compiled with OTP 19)

• Absinthe version (mix deps | grep absinthe):

* absinthe 1.4.11 (Hex package) (mix)
  locked at 1.4.11 (absinthe) b4b4e112
* absinthe_metrics 0.9.0 (Hex package) (mix)
  locked at 0.9.0 (absinthe_metrics) 59b608f6
* absinthe_plug 1.4.4 (Hex package) (mix)
  locked at 1.4.4 (absinthe_plug) f93db435
* absinthe_phoenix 1.4.2 (Hex package) (mix)
  locked at 1.4.2 (absinthe_phoenix) cb84c81b

• Client Framework and version (Relay, Apollo, etc):

"@absinthe/socket-apollo-link@^0.1.11":
  resolved "https://registry.yarnpkg.com/@absinthe/socket-apollo-link/-/socket-apollo-link-0.1.11.tgz#8dde213db3cf6e73419693c409ba00c96214a0e6"
    apollo-link "1.0.0"
apollo-link-ws@^1.0.8:
  resolved "https://registry.yarnpkg.com/apollo-link-ws/-/apollo-link-ws-1.0.8.tgz#ac1de8f29e92418728479a9a523af9f75b9ccc8b"
    apollo-link "^1.2.2"

Socket link is created with:

  const absintheSocketLink = createAbsintheSocketLink(
    AbsintheSocket.create(
      new PhoenixSocket(process.env.GRAPHQL_SOCKET_URI, {
        params: { client_id: clientId, token: auth.jwt },
      }),
    ),
  );

Actual behavior

We are seeing this error in production occasionally:

exit: ** (exit) {:badfun, #Function<1.30929525/1 in Eggl.Schema>}
  File "lib/absinthe/subscription/local.ex", line 60, in Absinthe.Subscription.Local.get_docs/4
  File "lib/absinthe/subscription/local.ex", line 14, in anonymous fn/4 in Absinthe.Subscription.Local.publish_mutation/3
  File "lib/enum.ex", line 1899, in Enum."-reduce/3-lists^foldl/2-0-"/3
  File "lib/absinthe/subscription/local.ex", line 13, in Absinthe.Subscription.Local.publish_mutation/3
  File "lib/task/supervised.ex", line 88, in Task.Supervised.do_apply/2
  File "proc_lib.erl", line 247, in :proc_lib.init_p_do_apply/3
  Module "Elixir.Absinthe.Subscription.Proxy", in anonymous fn/0 in Absinthe.Subscription.Proxy.handle_info/2

Alternative stacktrace:

I  15:37:45.702 [error] Task #PID<0.4267.0> started from #PID<0.2783.0> terminating

I  ** (BadFunctionError) expected a function, got: #Function<1.30929525/1 in Eggl.Schema>

I      (absinthe) lib/absinthe/subscription/local.ex:60: Absinthe.Subscription.Local.get_docs/4

I      (absinthe) lib/absinthe/subscription/local.ex:14: anonymous fn/4 in Absinthe.Subscription.Local.publish_mutation/3

I      (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3

I      (absinthe) lib/absinthe/subscription/local.ex:13: Absinthe.Subscription.Local.publish_mutation/3

I      (elixir) lib/task/supervised.ex:88: Task.Supervised.do_apply/2

I      (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

I  Function: #Function<1.113447438/0 in Absinthe.Subscription.Proxy.handle_info/2>

I      Args: []

Relevant Schema/Middleware Code

The sporadic errors started after adding this subscription:

```elixir
subscription do
field :close_io_lead_updated, :close_io_lead do
arg(:id, non_null(:id))

config(fn %{id: id}, _info ->
  {:ok, topic: id}
end)

trigger([:update_close_io_lead], topic: fn %{id: id} -> [id] end)

end
end

[benwilson512]

I have absolutely no idea :(. If it was a function clause error or something then my guess would have been that something was being passed in that didn't match %{id: id} but I don't know what bad fun means.

[samsondav]

@benwilson512 We're still seeing this. We're running a cluster of three nodes, I wonder if somehow the Phoenix pubsub is referencing a function and when we upgrade a node, this function no longer exists?

[benwilson512]

Ah, you're using hot code reloading? We have done exactly zero testing with hot code loading and Absinthe.

[samsondav]

Not using hot code reloading. We are using a cluster of dockerised containers. My theory is that if we add to the cluster using a new version of code, Phoenix pubsub has a reference to an invalid function on either the old nodes or the new ones.

[benwilson512]

Yeah I'm not sure, cause with subscriptions we aren't passing actual functions between nodes. All that gets passed between nodes are the actual published data, not any of the subscriptions themselves.

[drueck]

We just got this error today for the first time. We're running multiple instances on EC2 of our elixir backend which uses Absinthe and Phoenix, communicating with Phoenix.PubSub.Redis, and I think this error occurred right around the time of a deploy, so we may have had different versions of the code on each instance for a brief period of time. We do use subscriptions, although the change that was deployed did not make a change to any of the subscriptions. Just curious @samphilipd are you guys using Redis for PubSub, or PG2? I wonder if it's related to the PubSub mechanism doing something weird during an upgrade?

[benwilson512]

@drueck when you say upgrades are you referring to distillery / edeliver hot upgrades?

[drueck]

No, sorry, I just meant deploying a new release. We aren't using hot upgrades.

[maartenvanvliet]

Same error here, we also use PubSub.Redis and it happened during a deploy. Also not using hot upgrades

* absinthe 1.5.0-alpha.1 (Hex package) (mix)
  locked at 1.5.0-alpha.1 (absinthe) f605b754
* absinthe_plug 1.4.6 (Hex package) (mix)
  locked at 1.4.6 (absinthe_plug) ac5d2d3d
* absinthe_phoenix 1.4.3 (https://github.com/absinthe-graphql/absinthe_phoenix.git) (mix)

[benwilson512]

@maartenvanvliet @drueck @samphilipd without some mechanism to reproduce there isn't much we can do here in the short term on this.

[samsondav]

You can work around by using a module and function instead of an anonymous function in subscriptions. Reliably solves it for us.

[benwilson512]

Thanks for the update @samphilipd, that's good to know, and helps provide some insight into diagnosing the underlying issue.

[chungwong]

I believe this issue is relating how a request be routed to nodes with different releases
Environment
start two separate nodes

PORT=4001 iex --sname "4001" -S mix phx.server
PORT=4000 iex --sname "4000" -S mix phx.server

your frontend(GraphiQL) is listening to a url, say localhost/graphql which points to the following in Nginx

upstream st-backend {
server localhost:4000;
server localhost:4001;
}

Now make a changes to your Elixir source code and save, restart one of the phx.server to have the latest changes while keeping the old node without applying any changes.

Trigger the subscription and you will see the error.

[maartenvanvliet]

You can work around by using a module and function instead of an anonymous function in subscriptions. Reliably solves it for us.

subscription do
  field :close_io_lead_updated, :close_io_lead do
    arg(:id, non_null(:id))

    config(fn %{id: id}, _info ->
      {:ok, topic: id}
    end)

    trigger([:update_close_io_lead], topic: fn %{id: id} -> [id] end)
  end
end

@samphilipd when you say to use a module and function, do you mean the anonymous function the trigger or the config? Or both?

[benwilson512]

@maartenvanvliet both. In fact you'd probably be best off avoiding anonymous functions entirely. Notably &Module.function/3 forms are not anonymous functions, they are "function captures".

[maartenvanvliet]

Great, thanks!