Meshcentral: Out of Memory when downloading 140+MB file from file explorer

[songunsw]

Hi guys,

When downloading 140+ MB exe file from 'Files' tab, the browser crashes with 'Out of Memory' message.

The bug can be reproduced on my local MESH server and also on the https://meshcentral.com/ .

You may file explore the PC which runs the MESH server, and it takes less time to download and reproduce.
When the download progress bar reach 100%, the page crashes saying Error code: out of memory.

I had some debug and found the issue happens when converting 200mb size string to the blob, the for loop explodes the memory consumption, and crashes the browser.

function data2blob(data) {

var bytes = new Array(data.length);

for (var i = 0; i < data.length; i++) bytes[i] = data.charCodeAt(i);

return new Blob([new Uint8Array(bytes)]);

}

I made some fix and is working well with 1+GB file download now. The code is as following. I avoided the duplicate byte array to reduce memory, and In addition, I grouped the file for every 100mb, because there is a string length limit which is around 500mb which throw exception when the string is too big. Maybe some more improvement can be done?

function p13downloadfile(x, y, z, tag) {
if (xxdialogMode || downloadFile || !files) return;
downloadFile = { path: decodeURIComponent(x), file: decodeURIComponent(y), size: z, tsize: 0, data: '', state: 0, id: Math.random(), tag: tag }
if(z > 100000000)
{
downloadFile.isBigFile = true;
downloadFile.datas = [];
}
//console.log('p13downloadFileCancel', downloadFile);
files.sendText({ action: 'download', sub: 'start', id: downloadFile.id, path: downloadFile.path });
setDialogMode(2, "Download File", 10, p13downloadFileCancel, '

' + downloadFile.file + '

');
}

    // Called by the transport when binary data is received
    function p13gotDownloadBinaryData(data) {
        if (!downloadFile || downloadFile.state == 0) return;
        if (data.length > 4) {
            downloadFile.tsize += (data.length - 4); // Add to the total bytes received
            if(downloadFile.isBigFile == true)
            {
                var index = downloadFile.tsize / 100000000; //group for each 100mb
                if(downloadFile.datas.length <= index)
                    downloadFile.datas.push('');
                downloadFile.datas[downloadFile.datas.length - 1] += data.substring(4); // Append the data
            }
            else
                downloadFile.data += data.substring(4); // Append the data
            Q('d2progressBar').value = downloadFile.tsize; // Change the progress bar
        }
        if ((ReadInt(data, 0) & 1) != 0) { // Check end flag
            if (downloadFile.tag == 'viewer') {
                // View the file in the dialog box
                setDialogMode(4, EscapeHtml(downloadFile.file), 3, p13editSaveBack, null, downloadFile.file);
                QS('dialog').width = 'auto';
                QS('dialog').bottom = '80px';
                QS('dialog').top = QS('dialog').left = QS('dialog').right = '100px';
                Q('d4editorarea').value = downloadFile.data;
                downloadFile = null;
            } else {
                // Save the file to disk
                if(downloadFile.isBigFile == true)
                {
                    saveAs(newData2blob(downloadFile.datas), downloadFile.file); downloadFile = null; setDialogMode(0); // Save the file
                }
                else
                    saveAs(data2blob(downloadFile.data), downloadFile.file); downloadFile = null; setDialogMode(0); // Save the file
            }
        } else {
            files.sendText({ action: 'download', sub: 'ack', id: downloadFile.id }); // Send the ACK
        }
    }

    function newData2blob(datas) {
        var count=0;
        for (var i = 0; i < datas.length; i++)
            count+=datas[i].length;
        var bytes = new Uint8Array(count);
        var k = 0;
        for (var j = 0; j < datas.length; j++)
        {
            var data = datas[j];
            for (var i = 0; i < data.length; i++) bytes[k++] = data.charCodeAt(i);
            data = '';
            datas[j] = '';
        }
        return new Blob([bytes]);
    }

[Ylianst]

I just published MeshCentral v0.6.22 with a fix for this, the fix is not the one above. Instead, when you click on a file, it will now link to the MeshCentral server that will signal the device and stream the file just like a normal HTTPS file download. As a result of this change, the download size is not almost unlimited. The browser will save data to a file as it receives it and there is no more memory buffering.

Let me know if this works for you.

[songunsw]

Wow, You guys are legendary. It does not sound a small change, and it's done within one day. I think it should surely resolve the download limit once for all.
Does this change require Mesh Agent client upgrade? from the changeset it seems like a Server side only change?

I will surely try it out. Thanks @Ylianst

[Ylianst]

Thanks. Indeed it was a big change, but I have known about this for a long time and it had been on my todo list forever. Took 8 hours of focused work.

When you update the server, the agents are automatically updated if needed. In this case both server and meshcore.js need changing. When you update the server, the new meshcore.js will be pushed to all agents. The agent binary will stay the same. As background, there is a small JavaScript engine that runs on all agents and so, for many updates we don't need to change the entire agent.

[PathfinderNetworks]

Thank you for this change. The file downloads from remote devices is now MUCH faster and starts pretty much instantaneously. Huge improvement!