Marten: Supporting Large Objects

[jokokko]

Tracking if & how PostgreSQL Large Object facility could be exposed / abstracted via Marten (akin to something like RavenFS or GridFS).

Rationale:

• Storing binary artifacts via Marten
• Possibility of having streamed access to binaries
• Not having to pollute JSON serializations with base64 encoded byte arrays

Large object identifiers could then conveniently be tracked within the JSON side of things.

[jokokko]

An absolutely elementary example that maybe conveys some idea:

public static class DocumentSessionExtensions
{
    public static DocumentMetadata StoreFile(this IDocumentSession session, Stream fileData)
    {
        if (fileData == null)
        {
            throw new ArgumentNullException(nameof(fileData));
        }

        var manager = new NpgsqlLargeObjectManager(session.Connection);

        var oid = manager.Create();

        using (var stream = manager.OpenReadWrite(oid))
        {
            fileData.CopyTo(stream);
        }

        return new DocumentMetadata(oid);
    }
}

[jeremydmiller]

There is @schotime 's MartenFS project that we could use either as is, or just reverse engineer. From talking to Adam, I don't think it would be too bad to make MartenFS an addon to Marten if we don't want the big object functionality directly in Marten.

My personal preference would be to just make it an OOTB feature so we don't have to spend a lot of energy on extensibility points.

Regardless, doing this feature is going to force some structural changes to how UpdateBatch works today to handle updates that cannot be executed as part of a batched command.

[mdissel]

As long as it's stored within the same transaction of the SaveChanges(..) we don't have to support it within a batched command, at least for a start.

[mdissel]

Just bumping the issue to see if more people need this.

[roblthegreat]

I'm all for this feature. I really have a need to store some binary files as attachments / directly linked to JSON documents and would be part of a single transaction. If either the file doesn't get stored / updated correctly or the document fails an update, then both need to be rolled back.

[mysticmind]

Closing this. No recent activity,

[mdissel]

Retrying to get this on the list. To support the NpgsqlLargeObjectManager we need an explicit transaction from the existing DocumentSession and that is currently not possible. The static DocumentSession.ForTransaction(transaction) is an option but that is no option when using dependency injection..

[jokokko]

@mdissel Any rough ideas how the API or interaction would look like? Using TransactionScope won't work for you? Interested as I myself might also benefit from this feature.

[mdissel]

See the discussion in gitter.