Ecto: Possbile DBConnection regression in Ecto 3.0

[ontofractal]

Environment

• Elixir version: 1.7.3
• Database and version: PostgreSQL 10
• Ecto version (mix deps): 3.0.3
• Database adapter and version (mix deps): 0.14.0

• Operating system: Linux techiomes 4.18.0-11-generic #12-Ubuntu SMP Tue x86_64 GNU/Linux

• pool_size: 50

• pool_timeout: 300_000
• timeout: 300_000

Current behavior

for r <- records do
    spawn(fn -> process_asset_record(r) end)
end

process_asset_record function executes an unremarkable insert query.
records is a list with several thousand elements.

After upgrading to ecto_sql 3.0 this code raises a following exception:

[error] Process #PID<0.1371.0> raised an exception
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 112ms
    (ecto_sql) lib/ecto/adapters/sql.ex:595: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql) lib/ecto/adapters/sql.ex:528: Ecto.Adapters.SQL.execute/5
    (ecto) lib/ecto/repo/queryable.ex:147: Ecto.Repo.Queryable.execute/4
    (ecto) lib/ecto/repo/queryable.ex:18: Ecto.Repo.Queryable.all/3
    (ecto) lib/ecto/repo/queryable.ex:66: Ecto.Repo.Queryable.one/3

Expected behavior

Ecto 2.2 works as expected, no exceptions, all queries complete normally.

Workaround

After some tinkering, I've been able to successfully use the following workaround:

for r <- records do
  spawn(fn -> process_asset_record(r) end)
  Process.sleep(50)
end

If sleep duration is too low (1-5ms), exceptions reappear.

[josevalim]

What is your DBConnection version? Also, how many records do you have?

[ontofractal]

hi @josevalim!

• DBConnection version is 2.0.2
• records length is 2762

[josevalim]

Thanks @ontofractal! This is our bad. pool_timeout has no effect anymore in favor of a much improved queue system.

There are two options:

• :queue_target
  
  
  
  • :queue_interval
  
  

Our goal is to stay under :queue_target for :queue_interval. In case we can't reach that, then we double the :queue_target. If we go above that, then we start dropping messages. This is great because it allows us to better plan for overloads.

In other words, don't see the pool_timeout, instead set :queue_target. The default is 50ms. You can try out the double.

I will update the CHANGELOGs and docs accordingly.

[ontofractal]

Got it. Thanks for the fast response, @josevalim!

[josevalim]

I have updated the docs and CHANGELOG. Thank you!