Npgsql: Memory leak on oversized buffer allocation

[tghamm]

Steps to reproduce

Read from a table using async/await that has approximately 950 columns with pooling set to false. This issue is not replicable in tables with only a handful of columns, so it appears this has something to do with reading from large tables. It also completely disappears if pooling is turned on.

Sample Code:

class Program
    {
        static void Main(string[] args)
        {
            Task.Run(async () =>
            {
                for (var x = 0; x < 1000; x++)
                {
                    var res = await GetDataObject().ConfigureAwait(false);
                }
                GC.Collect();
            });


            Console.ReadLine();
        }


        public static async Task<List<DataObject>> GetDataObject()
        {
            var list = new List<DataObject>();
            using (var context = new NpgsqlConnection("User ID=postgres;Password=<pw>;Host=localhost;Port=5432;Database=demand;Pooling=false;"))
            {
                await context.OpenAsync().ConfigureAwait(false);

                var query = "select * from us_demand limit 1000;";

                using (var command = new NpgsqlCommand(query, context))
                {
                    command.CommandTimeout = 360;


                    using (var dr = await command.ExecuteReaderAsync().ConfigureAwait(false))
                    {
                        while (await dr.ReadAsync().ConfigureAwait(false))
                        {
                            var dObj = new DataObject {Id = (string) dr["ID"]};

                            list.Add(dObj);
                        }
                    }
                }
            }

            return list;
        }

        public class DataObject
        {
            public string Id { get; set; }
        }
    }

The issue

I would expect the memory usage to hold pretty consistently with the above code. When running with pooling set to TRUE, memory holds at ~31mb. With pooling set to FALSE, memory continuously climbs and is not released, settling in around 350mb. This number ultimately continually grows until OOMs occur if it runs unchecked.

Memory snapshot after run at 100 iterations with Pooling off:

Memory Snapshot of same test with Pooling on:

Further technical details

Npgsql version: 4.0.10
PostgreSQL version: 9.4/9.6
Operating system: Windows 10/Ubuntu 16.04
.Net Version: 4.5.2

Please let me know what additional information I can provide to be helpful.

Thanks!

[YohDeadfall]

Reproduced. The issue happens on non-sequential access only.

[YohDeadfall]

It's not about column count, but oversized buffer allocation when an awaitable socket is used. There were lost few disposes of SocketAsyncEventArgs which internally uses a pinned overlapped. In .NET Framework a pinned overlapped cannot be freed until Dispose is invoked, but this issue is fixed in .NET Core.

[tghamm]

@YohDeadfall sure, I'll scope it out later this weekend and report back. Thanks for taking the time to do a deep dive. Much appreciated. More to come within the next 72 hours or so.

Thanks!

[tghamm]

@YohDeadfall I've confirmed your commit rectifies what we're seeing on our end. Looks fixed. Let me know if I can be helpful in any further way. Thanks!