Ecto: DBConnection cursors has a growing list of references causing Out Of Memory exceptions

[gheorghina]

Environment

• Elixir version (elixir -v): 1.9.4
• Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): MariaDB RDS
• Ecto version (mix deps): 3.0
• Database adapter and version (mix deps): myxql 0.3.0
• Operating system: ubuntu:18.04

Current behavior

I noticed the cursors map inside the DBConnection processes keeps growing, causing OOM after a couple of days.
We are using both Ecto.stream and Ecto.transactions - which might cause this.

Examples:
#Reference<0.3571010832.1130627074.258072> => {:column_defs,
[
{:column_def, ...},
{...},
...
], 292},
#Reference<0.3571010832.1130364930.175393> => {:column_defs,
[
{:column_def, "uuid", ...},
{:column_def, ...},
{...},
...
], 92},
#Reference<0.3571010832.1131151361.167112> => {:column_defs,
[{:column_def, "id", ...}, {:column_def, ...}, {...}, ...], 462},
#Reference<0.3571010832.1130364930.213027> => {:column_defs,
[{:column_def, ...}, {...}, ...], 116},
#Reference<0.3571010832.1130889218.13300> => {:column_defs,
[{...}, ...], 302},
#Reference<0.3571010832.1130889218.3795> => {:column_defs, [...], ...},
#Reference<0.3571010832.1131151362.170979> => {:column_defs, ...},
#Reference<0.3571010832.1131413506.206247> => {...},
...
},

Expected behavior

Expected to either get cleaned-up automatically or to be able to trigger the cleanup of old entries somehow

I searched a bit for similar scenarios, but I could not find something relevant.
Thank you in advance.

[josevalim]

Hi @gheorghina! Thank you for the report. Can you please provide a minimal application with steps to reproduce the error? This will help us ensure we are fixing the bug you are running into.

[gheorghina]

yes, sure, I can create a small app, until then, what I managed to do was to reduce the complex scenario we have and still reproduce the issue to:
have a recurrent call to a function which does only this:

def run() do
    current_date = DateTime.utc_now()

    query =
      from(m in Model,
        where: m.status != "expired",
        where: fragment("TIMESTAMPDIFF(SECOND, ?, ?)", ^current_date, m.expires_at) < 0
      )

    stream =
      query
      |> Repo.stream()
      |> Task.async_stream(fn _model -> :ok end)

    Repo.transaction(fn -> Stream.run(stream) end)
  end

[gheorghina]

https://github.com/gheorghina/dbconn_cursors_leak

[wojtekmach]

just to get this out of the way, could you update myxql to v0.4.0 and check again?

[gheorghina]

I just updated it, and the behavior is still the same. I will leave it also over a longer period of time.

[gheorghina]

I just updated it, and the behavior is still the same. I will leave it also over a longer period of time.

I confirm the issue is still there after myxql update to v0.4.0

[josevalim]

This has been fixed on myxql master. Can you please give it a try? Thank you!

[chulkilee]

For the record - https://github.com/elixir-ecto/myxql/commit/082febbe5941a6e430680d5c133bc0e74469554d

[gheorghina]

Thank you for being this fast!
I just tested it and the issue no longer reproducing 🎉