Litedb: FileStorage.Upload from stream does not seem to be working

[ghost]

Hey David,

I have an issue with uploading images to LiteDB using the FileStorage.Upload method. Below is the code snipped that is failing:

using( var image = Image.FromFile("C:\\Temp\\testfile.jpg").GetThumbnailImage(200, 200, null, IntPtr.Zero))
using (var stream = new MemoryStream())
{
    image.Save(stream, ImageFormat.Jpeg);
    Database.FileStorage.Upload(Guid.NewGuid().ToString(), "testfile.jpg", stream);
}

When I debug I see that the stream contains data, but it does not seem to get written to the database. I see the document in the database with the filename, Id, etc. But the chunk-size remains 0.

Using OpenWrite and saving the image to the LiteFileStream directly does work, however, here I get an error when trying to read from the collection while OpenWrite is still writing to the database:

An exception of type 'System.IO.IOException' occurred in LiteDB.dll but was not handled in user code

Additional information: The requested operation cannot be performed on a file with a user-mapped section open.

Stack trace:

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.SetLengthCore(Int64 value) at System.IO.FileStream.SetLength(Int64 value) at LiteDB.FileDiskService.SetLength(Int64 fileSize) in FileDiskService.cs:line 129 at LiteDB.FileDiskService.ClearJournal(UInt32 lastPageID) in FileDiskService.cs:line 191 at LiteDB.TransactionService.PersistDirtyPages() in TransactionService.cs:line 79 at LiteDB.LiteEngine.Transaction[T](String collection, Boolean addIfNotExists, Func`2 action) in LiteEngine.cs:line 1192 at LiteDB.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonType autoId) in LiteEngine.cs:line 886 at LiteDB.LiteEngine.Insert(String collection, BsonDocument doc, BsonType autoId) in LiteEngine.cs:line 873 at LiteDB.LiteStorage.OpenWrite(String id, String filename, BsonDocument metadata) in LiteStorage.cs:line 37

Ok, so actually I have 2 problems it seems. So either I am doing something very wrong or there really is a problem with the Upload method. Any ideas?

Sincerely,
Glenn

[ghost]

This issue might be related to issue: #1022 but I'm not sure

[ghost]

As per ticket #1022 I also tried disabling the journal. After disabling the journal the error regarding the System.IO.IOException was not thrown anymore.

[ghost]

After further analysis and hitting upon your Concurrency wiki entry (https://github.com/mbdavid/LiteDB/wiki/Concurrency) I realized the problem was, that I was opening multiple instances of the database which were conflicting. Once I fixed this I was not getting the "System.IO.IOException" error anymore.

The drawback is, that if I have a process that is writing a lot of entries to the database, and another that is trying to read from the database, the read process is very slow and takes quite a while. I suppose because the file is being locked for writing? Any idea on how to speed up the performance here?

In regards to the FileStorage.Upload method: Still did not get it to work

[mbdavid]

Hi @stitch10925, sorry for long delay. Is your stream in Position = 0 (at beginning) when you call Upload method?

[ghost]

Hey David,

I tried both scenario's. Following the code example above:

(Update: sorry for the formatting. For some reason the code block is not working properly)

Memory stream is at position 5123 (Save image to stream, then upload):

using( var image = Image.FromFile("C:\Temp\testfile.jpg").GetThumbnailImage(200, 200, null, IntPtr.Zero))
using (var stream = new MemoryStream())
{
image.Save(stream, ImageFormat.Jpeg);
Database.FileStorage.Upload(Guid.NewGuid().ToString(), "testfile.jpg", stream);
}

Memory stream is at position 0 (Setup upload, then save image):

using( var image = Image.FromFile("C:\Temp\testfile.jpg").GetThumbnailImage(200, 200, null, IntPtr.Zero))
using (var stream = new MemoryStream())
{
Database.FileStorage.Upload(Guid.NewGuid().ToString(), "testfile.jpg", stream);
image.Save(stream, ImageFormat.Jpeg);
}

Both yielded the same result in the database: Length 0, chunks 0. Also flushing the memory stream did not work.

[mbdavid]

Try set position to 0

using( var image = Image.FromFile("C:\Temp\testfile.jpg").GetThumbnailImage(200, 200, null, IntPtr.Zero))
using (var stream = new MemoryStream())
{
image.Save(stream, ImageFormat.Jpeg);

// your stream must be at begin position to be read by litedb
stream.Position = 0;

Database.FileStorage.Upload(Guid.NewGuid().ToString(), "testfile.jpg", stream);
}

[ghost]

Alright, so loading the image into the stream, setting the stream to position 0, then uploading to the database (as you described in your previous post) worked.

Is setting the stream position to 0 something that makes sense adding to the Upload() method so that anyone using the method does not need to set the stream position to 0 every time?

[mbdavid]

You always need be sure that you stream are at beginning position. LiteDB will read you stream from current position to end. If your stream already at end, there is nothing to read.

In you case, the problem is image.Save that read first stream (your image) to create a thumbnail. This method Save run over your memory stream (to save the bytes) and do not get back to first position (to be saved on litedb).

Its not about how LiteDB upload but most about how Stream works. If you write direct your image (with no thumbnail) where is not need set position = 0 because already in begin.

c# using (var stream = new FileStream("yourphoto.png")) { // stream position are in position 0 because do not start reading yet Database.FileStorage.Upload(Guid.NewGuid().ToString(), "testfile.jpg", stream); // now, if you check position here, you will see Position = Length }

[rolandh]

This is not intuitive, I spent 3 hours debugging the same issue. I've never had to manually set my stream.Position = 0 in any other libraries.

Maybe you could check if stream.Position == stream.Length or if the number of bytes copies is 0 and throw a useful exception to the user?

I've made a small set of extension methods to handle a few un-intuitive things like this.

` public static class LiteDatabaseExtensions
{

    public static LiteFileInfo UploadSafe(this LiteStorage db, string id, string filename, Stream stream)
    {
        stream.Position = 0;
        return db.Upload(MakeSafe(id), MakeSafe(filename), stream);
    }

    public static bool DeleteSafe(this LiteStorage db, string id)
    {
        return db.Delete(MakeSafe(id));
    }

    public static LiteFileStream OpenReadSafe(this LiteStorage db, string id)
    {

        return db.OpenRead(MakeSafe(id));
    }

    public static bool ExistsSafe(this LiteStorage db, string id)
    {
        return db.Exists(MakeSafe(id));
    }


    private static string MakeSafe(string id)
    {
        id = id.Replace(@"\", "_");
        id = id.Replace(":", "-");
        return id;
    }
}`