Core: Repair script for encrypted files that were not marked as encrypted + wrong size

[PVince81]

Due to https://github.com/owncloud/core/issues/17594, files uploaded to a shared folder in 8.1.0 through Webdav have the wrong file size and cannot be downloaded properly.
The core issue has been fixed but the bogus files still exist on my system.

Of course I could just reupload them, but for people who have more than a few files it will be tedious to reupload them or ask several users to reupload some files.

I suggest to have a repair step that rescans encrypted files and updates the matching oc_filecache entry with the correct size and encrypted flag.

Some kind of files:scan but for encryption.

@schiesbn @DeepDiver1975 what do you think ?

[PVince81]

Side note: The new sync client 2.0.0~rc1 is less tolerant and fails the download every time. Maybe the download routine was changed and it now always fails on timeout.

[PVince81]

Just tried setting "encrypted" to 1, but it seems it's not enough.
The size column needs to be adjusted too, which I can't do with a simple SQL command.

Deleting or overwriting the files with the sync client also doesn't work.

However deleting through the web UI works, so I was able to reupload the borken files.
Fortunately it was only 10 files in a single folder.

[xn--nding-jua]

Hello,

here is my proposal for a recovery-script. After the configuration is done in the header of the file it will do the following:

1. connect to the ownCloud encryption system
2. connect to the mySQL database
3. steps through the folder given in the header (recursively with sub-folders)
4. compares the filesize stored in database with the real encrypted file. If sizes are the same, the database has the wrong value
5. if the filesize is wrong (database===encrypted file), the specific file will be decrypted, stored to the tmp-directory and the filesize of the decrypted file will be stored to the database. The tmp-file will be deleted afterwards.

Be careful:
a) script will not test, if the encryption-app is enabled or not
b) will run only with Encryption App 2.0
c) database will be read for each file individually. maybe someone can optimizeit or speed it up

How to use?

1. copy the code into a file called fixwrongsizedfiles.php
2. edit the header of the file (usernames, passwords, folder, etc)
3. connect to server by ssh and run "php fixwrongsizedfiles.php"
4. the script will run over all files within the scope of the given folder and updates the SQL-DB whenever necessary

<?php

// set php-language to German/UTF-8
$loc_de = setlocale(LC_ALL, "de_DE.UTF-8");

// ownCloud setup
$oc_datadir = '/owncloud/data/';
$oc_tmpdir = '/tmp/';
$oc_username = "User1";
$oc_password = "Trustno1";
$db_server = "localhost";
$db_user = "DBUSER";
$db_name = "DBNAME";
$db_password = "Trustno1";

// check all files within the following directory (with sub-directories)
// enter the folder as you see them within owncloud-webui
$searchfolder='Folder1/Subfolder1/';
$fix_encryption_flag='1'; // 1=please fix it, 0=dont touch it


require_once 'lib/base.php';

////////////////////////// ////////////////////////////////
// debug-messages
///////////////////////////////////////////////////////////
/*
echo "ownCloud-Data directory: " .$oc_datadir . PHP_EOL;
echo "ownCloud-User: " . $oc_username . PHP_EOL;
echo "Fixing size for file: " . $oc_filepath . PHP_EOL;
echo "Using SQL-DB: " . $db_name . PHP_EOL;
echo PHP_EOL;
*/

////////////////////////// ////////////////////////////////
// Open the desired file and decrypt it to the tmp-directory
///////////////////////////////////////////////////////////

// first get users private key and decrypt it
try{
    $session = new \OCA\Encryption\Session(\OC::$server->getSession());
    $userSession = \OC::$server->getUserSession();
    $crypt = new \OCA\Encryption\Crypto\Crypt(\OC::$server->getLogger(), $userSession, \OC::$server->getConfig());

    $encryptedUserKey = file_get_contents($oc_datadir . $oc_username . '/files_encryption/OC_DEFAULT_MODULE/' . $oc_username . '.privateKey');
    $decryptedUserKey = $crypt->decryptPrivateKey($encryptedUserKey, $oc_password);
} catch (\Exception $ex) {
    echo $ex->getMessage() . PHP_EOL;
}

$userpath=$oc_datadir . $oc_username . '/files/';

try{
    // connect to mySQL
    $db = mysql_connect($db_server, $db_user, $db_password);
    if(!$db)
    {
        // no connection to db
        echo "ERROR: Failed to connect to DB: " . mysqli_connect_error() . PHP_EOL;
    }else{
        // select database
        mysql_select_db($db_name) or die("ERROR: Failed to switch to db " . $db_name);

        //$folderhandle=opendir(utf8_encode($oc_datadir . $oc_username . "/files/" . $searchfolder));
        //while (false !== ($file = readdir($folderhandle)))
        $di = new RecursiveDirectoryIterator(utf8_encode($oc_datadir . $oc_username . "/files/" . $searchfolder));
        foreach (new RecursiveIteratorIterator($di) as $file)
        {
            // define the file to check
            if(file_exists($file) && (filetype($file)==="file"))
            {
                $oc_filepath = str_replace($userpath, '', $file);
                $oc_filepath_raw = utf8_decode($oc_filepath);

                echo $oc_filepath . PHP_EOL;

                $dbresult = mysql_query("SELECT * FROM `" . $db_name .  "`.`oc_filecache` WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                $numberofrows = mysql_num_rows($dbresult);

                if($numberofrows>1)
                {
                    // more than one entry found -> exception
                    echo "ERROR: More than one DB-entry found for the file files/" . $oc_filepath . PHP_EOL;
                }elseif($numberofrows===0){
                    // more than one entry found -> exception
                    echo "ERROR: No DB-entry found for the file files/" . $oc_filepath . PHP_EOL;
                }else{
                    // only one entry found -> OK
                    $row = mysql_fetch_object($dbresult);


                    $encryptedFilesize = filesize($oc_datadir . $oc_username . '/files/' . $oc_filepath);

                    if($fix_encryption_flag==='1'){
                        if(intval($row->encrypted)===0){
                            echo "INFO: Encryption-flag is 0. So we have to fix it." . PHP_EOL;
                            $update = mysql_query("UPDATE `" . $db_name .  "`.`oc_filecache` Set `encrypted` = '1' WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                        }
                    }

                    // compare encrypted file and stored filesize. if they match, we have to fix it
                    if(intval($row->size)===intval($encryptedFilesize)) {
                        echo "INFO: Filesize matches encrypted filesize. So we have to fix it." . PHP_EOL;


                        // we need to decrypt the file-key, therefore we use our private key and the share key
                        $shareKey = file_get_contents($oc_datadir . $oc_username . '/files_encryption/keys/files/' . $oc_filepath . '/OC_DEFAULT_MODULE/' . $oc_username . '.shareKey');
                        $encryptedKeyfile = file_get_contents($oc_datadir . $oc_username . '/files_encryption/keys/files/' . $oc_filepath . '/OC_DEFAULT_MODULE/fileKey');
                        try {
                            $decryptedKeyfile = $crypt->multiKeyDecrypt($encryptedKeyfile, $shareKey, $decryptedUserKey);
                        } catch (\Exception $ex) {
                            echo $ex->getMessage() . PHP_EOL;
                        }

                        // read and decrypt the encrypted file
                        $filehandle = fopen($oc_datadir . $oc_username . '/files/' . $oc_filepath, 'r');
                        $encryptedContent = '';
                        $decryptedContent = '';

                        // read header (means removing it for decrypting the file)
                        $encryptedContent = fread($filehandle, 8192);
                        // read data
                        while ($encryptedContent = fread($filehandle, 8192)) {
                            // Decrypt the block
                            $decryptedBlock = $crypt->symmetricDecryptFileContent($encryptedContent, $decryptedKeyfile);
                            // A that point you can write the decrypted block to the decrypted file
                            // instead of appending it to $decryptedContent. But I don't fear memory
                            // limits and I choose to append the block to my string and write the
                            //  file at the end
                            $decryptedContent .= $decryptedBlock;
                        }
                        fclose($filehandle);

                        // write the content to the tmp-folder
                        // delete an existing temporary file
                        if (file_exists($oc_tmpdir . basename($oc_filepath)))
                        {
                            unlink($oc_tmpdir . basename($oc_filepath));
                        }
                        file_put_contents($oc_tmpdir . basename($oc_filepath), $decryptedContent);

                        // read the real filesize of the file
                        $decryptedFilesize = filesize($oc_tmpdir . basename($oc_filepath));
                        //echo "Size of encrypted file: " . $encryptedFilesize . " Byte" . PHP_EOL;
                        //echo "Size of decrypted file: " . $decryptedFilesize . " Byte" . PHP_EOL;

                        // delete the temporary file
                        if (file_exists($oc_tmpdir . basename($oc_filepath)))
                        {
                            unlink($oc_tmpdir . basename($oc_filepath));
                        }

                        // update DB-entry
                        if(intval($row->size)===intval($decryptedFilesize))
                        {
                            // stored size is the expected size of the dencrypted file -> nothing to do
                            echo "INFO: Correct filesize of " . $row->size . " Bytes found in DB. Nothing to do here." . PHP_EOL;
                        }elseif(intval($row->size)===intval($encryptedFilesize))
                        {
                            // stored size is the expected size of the encrypted file -> correct the DB-entry
                            echo "INFO: Wrong filesize of " . $row->size . " Bytes found in DB. Will be fixed to " . $decryptedFilesize . " Bytes" . PHP_EOL;

                            $update = mysql_query("UPDATE `" . $db_name .  "`.`oc_filecache` Set `size` = '" . $decryptedFilesize . "' WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                            //echo "UPDATE `" . $db_name .  "`.`oc_filecache` Set size = '" . $decryptedFilesize . "' WHERE `path` LIKE 'files/" . $oc_filepath . "'" . PHP_EOL;
                        }else{
                            // size is not as expected -> exception
                            echo "ERROR: The filesize stored in the DB (" . $row->size . " Bytes) is not as expected (" . $encryptedFilesize . " Bytes)!" . PHP_EOL;
                        }
                    }else{
                        echo "INFO: Filesize seems to be OK. Nothing to do here." . PHP_EOL;
                    }
                }
            }
        }
        //closedir($folderhandle);
        mysql_close($db);
    }
} catch (\Exception $ex) {
    echo $ex->getMessage() . PHP_EOL;
}

?>

[PVince81]

@xn--nding-jua thanks. Ideally the script should be an actual "./occ" command.

See how "encrypt_all" is implemented https://github.com/owncloud/core/pull/18423/files#diff-064da27f0f8697102da5a1e016dd1193R33
The command is registered here: https://github.com/owncloud/core/pull/18423/files#diff-9375d4ac4c933da2a6eb29113cc84c0dR60

[PVince81]

@schiesbn can you check its logic ?

[MorrisJobke]

A bug with incorrect sizes occured: #19667

[nickbooties]

Thanks for the script @xn--nding-jua !
I had previously bypassed the download/sync problem by commenting out the pieces of code which send the content-length header, both in DAV and core (Not Ideal!)
Is it true that since a users password is required to decrypt their own files, there is no way to run this as admin across all files on the server? I have quite a few users and it's not really practical to ask for each of their passwords in order to repair the database.

[xn--nding-jua]

@nickbooties I had a look at the occ-decryptAll-command and here (https://github.com/owncloud/core/blob/master/apps/encryption/lib/crypto/decryptall.php#L83) the password for each user is requested. So it seems that it is not possible to decrypt the files without entering the user-password. If the user has activated the recoverykey it is possible to use the recovery-password to decrypt the keys. So this is a limitation in favor of security of the user files.

[nickbooties]

@xn--nding-jua yeah that's as I suspected, thank you again for the help.

[chriseqipe]

thx to xn--nding-jua for his approach for a repair script.
I adapted it a little bit to our environment, because the script runs very slow on big oc instances with a million of records in table (like condition always scan the complete table).
I changed all queries to use oc indexes (storage and path_hash) to speed up the traversal and make use of config.php to include information how to connect to the database (only mysql).

<?php

// set php-language to German/UTF-8
$loc_de = setlocale(LC_ALL, "de_DE.UTF-8");

// ownCloud setup
$oc_tmpdir = '/tmp/';
$oc_username = "ausername";
$oc_password = "verysecret";
$oc_storage = 11; // have a look at table storages

// check all files within the following directory (with sub-directories)
// enter the folder as you see them within owncloud-webui
// NEEDS ending slash!!!
$searchfolder = '/';
$fix_encryption_flag = '1'; // 1=please fix it, 0=dont touch it


require_once('lib/base.php');
include('config/config.php');
$db_name = $CONFIG['dbname'];
$db_user = $CONFIG['dbuser'];
$db_password = $CONFIG['dbpassword'];
$db_server = $CONFIG['dbhost'];
$oc_datadir = $CONFIG['datadirectory'];
$oc_tblfilecache = $CONFIG['dbtableprefix'].'filecache';

////////////////////////// ////////////////////////////////
// debug-messages
///////////////////////////////////////////////////////////
/*
echo "ownCloud-Data directory: " .$oc_datadir . PHP_EOL;
echo "ownCloud-User: " . $oc_username . PHP_EOL;
echo "Fixing size for file: " . $oc_filepath . PHP_EOL;
echo "Using SQL-DB: " . $db_name . PHP_EOL;
echo PHP_EOL;
*/

////////////////////////// ////////////////////////////////
// Open the desired file and decrypt it to the tmp-directory
///////////////////////////////////////////////////////////

// first get users private key and decrypt it
try {
    $session = new \OCA\Encryption\Session(\OC::$server->getSession());
    $userSession = \OC::$server->getUserSession();
    $crypt = new \OCA\Encryption\Crypto\Crypt(\OC::$server->getLogger(), $userSession, \OC::$server->getConfig());

    $encryptedUserKey = file_get_contents($oc_datadir . $oc_username . '/files_encryption/OC_DEFAULT_MODULE/' . $oc_username . '.privateKey');
    $decryptedUserKey = $crypt->decryptPrivateKey($encryptedUserKey, $oc_password);
} catch (\Exception $ex) {
    echo $ex->getMessage() . PHP_EOL;
}

$userpath = $oc_datadir . $oc_username . '/files';

try {
    // connect to mySQL
    $db = mysql_connect($db_server, $db_user, $db_password);
    if (!$db) {
        // no connection to db
        echo "ERROR: Failed to connect to DB: " . mysqli_connect_error() . PHP_EOL;
    } else {
        // select database
        mysql_select_db($CONFIG['dbname']) or die("ERROR: Failed to switch to db " . $CONFIG['dbname']);
        mysql_query('SET names utf8');
        //$folderhandle=opendir(utf8_encode($oc_datadir . $oc_username . "/files/" . $searchfolder));
        //while (false !== ($file = readdir($folderhandle)))
        $di = new RecursiveDirectoryIterator(utf8_encode($oc_datadir . $oc_username . "/files" . $searchfolder));
        foreach (new RecursiveIteratorIterator($di) as $file) {
            // define the file to check
            if (file_exists($file) && (filetype($file) === "file")) {
                $oc_filepath = str_replace($userpath, '', $file);
                $oc_filepath_raw = utf8_decode($oc_filepath);

                #echo $oc_filepath . PHP_EOL;

                $path_hash = md5('files' . $oc_filepath);
                $query = sprintf('select * from %s where storage = %d and path_hash = "%s"', $oc_tblfilecache, $oc_storage, $path_hash);
                $dbresult = mysql_query($query);
                $numberofrows = mysql_num_rows($dbresult);

                if ($numberofrows > 1) {
                    // more than one entry found -> exception
                    echo "ERROR: More than one DB-entry found for the file files" . $oc_filepath . PHP_EOL;
                } elseif ($numberofrows === 0) {
                    // more than one entry found -> exception
                    echo "ERROR: No DB-entry found for the file files" . $oc_filepath . PHP_EOL;
                } else {
                    // only one entry found -> OK
                    $row = mysql_fetch_object($dbresult);


                    $encryptedFilesize = filesize($oc_datadir . $oc_username . '/files' . $oc_filepath);

                    if ($fix_encryption_flag === '1') {
                        if (intval($row->encrypted) === 0) {
                            echo $oc_filepath . PHP_EOL;
                            echo "INFO: Encryption-flag is 0. So we have to fix it." . PHP_EOL;
                            # $update = mysql_query("UPDATE `" . $db_name .  "`.`filecache` Set `encrypted` = '1' WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                            $query = sprintf('UPDATE %s set encrypted = 1 where storage = %d and path_hash = "%s"', $oc_tblfilecache, $oc_storage, $path_hash);
                            $update = mysql_query($query);
                        }
                    }

                    // compare encrypted file and stored filesize. if they match, we have to fix it
                    if (intval($row->size) === intval($encryptedFilesize)) {
                        echo $oc_filepath . PHP_EOL;
                        echo "INFO: Filesize matches encrypted filesize. So we have to fix it." . PHP_EOL;

                        // we need to decrypt the file-key, therefore we use our private key and the share key
                        $shareKey = file_get_contents($oc_datadir . $oc_username . '/files_encryption/keys/files' . $oc_filepath . '/OC_DEFAULT_MODULE/' . $oc_username . '.shareKey');
                        $encryptedKeyfile = file_get_contents($oc_datadir . $oc_username . '/files_encryption/keys/files' . $oc_filepath . '/OC_DEFAULT_MODULE/fileKey');
                        try {
                            $decryptedKeyfile = $crypt->multiKeyDecrypt($encryptedKeyfile, $shareKey, $decryptedUserKey);
                        } catch (\Exception $ex) {
                            echo $ex->getMessage() . PHP_EOL;
                        }

                        // read and decrypt the encrypted file
                        $filehandle = fopen($oc_datadir . $oc_username . '/files' . $oc_filepath, 'r');
                        $encryptedContent = '';
                        $decryptedContent = '';

                        // read header (means removing it for decrypting the file)
                        $encryptedContent = fread($filehandle, 8192);
                        // read data
                        while ($encryptedContent = fread($filehandle, 8192)) {
                            // Decrypt the block
                            $decryptedBlock = $crypt->symmetricDecryptFileContent($encryptedContent, $decryptedKeyfile);
                            // A that point you can write the decrypted block to the decrypted file
                            // instead of appending it to $decryptedContent. But I don't fear memory
                            // limits and I choose to append the block to my string and write the
                            //  file at the end
                            $decryptedContent .= $decryptedBlock;
                        }
                        fclose($filehandle);

                        // write the content to the tmp-folder
                        // delete an existing temporary file
                        if (file_exists($oc_tmpdir . basename($oc_filepath))) {
                            unlink($oc_tmpdir . basename($oc_filepath));
                        }
                        file_put_contents($oc_tmpdir . basename($oc_filepath), $decryptedContent);

                        // read the real filesize of the file
                        $decryptedFilesize = filesize($oc_tmpdir . basename($oc_filepath));
                        //echo "Size of encrypted file: " . $encryptedFilesize . " Byte" . PHP_EOL;
                        //echo "Size of decrypted file: " . $decryptedFilesize . " Byte" . PHP_EOL;

                        // delete the temporary file
                        if (file_exists($oc_tmpdir . basename($oc_filepath))) {
                            unlink($oc_tmpdir . basename($oc_filepath));
                        }

                        // update DB-entry
                        if (intval($row->size) === intval($decryptedFilesize)) {
                            // stored size is the expected size of the dencrypted file -> nothing to do
                            // echo "INFO: Correct filesize of " . $row->size . " Bytes found in DB. Nothing to do here." . PHP_EOL;
                        } elseif (intval($row->size) === intval($encryptedFilesize)) {
                            // stored size is the expected size of the encrypted file -> correct the DB-entry
                            echo "INFO: Wrong filesize of " . $row->size . " Bytes found in DB. Will be fixed to " . $decryptedFilesize . " Bytes" . PHP_EOL;

                            $query = sprintf('UPDATE %s set size = %d where storage = %d and path_hash = "%s"', $oc_tblfilecache, $decryptedFilesize, $oc_storage, $path_hash);
                            $update = mysql_query($query);
                            //echo "UPDATE `" . $db_name .  "`.`filecache` Set size = '" . $decryptedFilesize . "' WHERE `path` LIKE 'files/" . $oc_filepath . "'" . PHP_EOL;
                        } else {
                            // size is not as expected -> exception
                            echo "ERROR: The filesize stored in the DB (" . $row->size . " Bytes) is not as expected (" . $encryptedFilesize . " Bytes)!" . PHP_EOL;
                        }
                    } else {
                        // echo "INFO: Filesize seems to be OK. Nothing to do here." . PHP_EOL;
                    }
                }
            }
        }
        //closedir($folderhandle);
        mysql_close($db);
    }
} catch (\Exception $ex) {
    echo $ex->getMessage() . PHP_EOL;
}

[brunodegoyrans]

Hello

To make the second script working, I had to add a slash to $oc_datadir:
$oc_datadir = $CONFIG['datadirectory'] + '/';

[brunodegoyrans]

I have tons of "ERROR: No DB-entry found for the file.
I look at the sql query:
select * from oc_filecache where storage = 11 and path_hash = "...."

storage = 11 does correspond to the line in top of script:
oc_storage = 11; // have a look at table storages

I don't know what means this parameter?
and I remembered what @chriseqipe said : "I adapted it a little bit to our environment".
I wondered if this 2nd script isn't too specific?

[brunodegoyrans]

If you handle huge files, you may encounter FATAL ERROR allowed memory exhausted.

This is because the above scripts write once to tmp.
I have changed the 1st script in order to write to tmp, block by block.

So this the version 3 of the script:

<?php

// set php-language to German/UTF-8
$loc_de = setlocale(LC_ALL, "de_DE.UTF-8");

// ownCloud setup
$oc_datadir = '/owncloud/data/';
$oc_tmpdir = '/tmp/';
$oc_username = "User1";
$oc_password = "Trustno1";
$db_server = "localhost";
$db_user = "DBUSER";
$db_name = "DBNAME";
$db_password = "Trustno1";

// check all files within the following directory (with sub-directories)
// enter the folder as you see them within owncloud-webui
$searchfolder='/';
$fix_encryption_flag='1'; // 1=please fix it, 0=dont touch it


require_once 'lib/base.php';

////////////////////////// ////////////////////////////////
// debug-messages
///////////////////////////////////////////////////////////
/*
echo "ownCloud-Data directory: " .$oc_datadir . PHP_EOL;
echo "ownCloud-User: " . $oc_username . PHP_EOL;
echo "Fixing size for file: " . $oc_filepath . PHP_EOL;
echo "Using SQL-DB: " . $db_name . PHP_EOL;
echo PHP_EOL;
*/

////////////////////////// ////////////////////////////////
// Open the desired file and decrypt it to the tmp-directory
///////////////////////////////////////////////////////////

// first get users private key and decrypt it
try{
    $session = new \OCA\Encryption\Session(\OC::$server->getSession());
    $userSession = \OC::$server->getUserSession();
    $crypt = new \OCA\Encryption\Crypto\Crypt(\OC::$server->getLogger(), $userSession, \OC::$server->getConfig());

    $encryptedUserKey = file_get_contents($oc_datadir . $oc_username . '/files_encryption/OC_DEFAULT_MODULE/' . $oc_username . '.privateKey');
    $decryptedUserKey = $crypt->decryptPrivateKey($encryptedUserKey, $oc_password);
} catch (\Exception $ex) {
    echo $ex->getMessage() . PHP_EOL;
}

$userpath=$oc_datadir . $oc_username . '/files/';

try{
    // connect to mySQL
    $db = mysql_connect($db_server, $db_user, $db_password);
    if(!$db)
    {
        // no connection to db
        echo "ERROR: Failed to connect to DB: " . mysqli_connect_error() . PHP_EOL;
    }else{
        // select database
        mysql_select_db($db_name) or die("ERROR: Failed to switch to db " . $db_name);

        //$folderhandle=opendir(utf8_encode($oc_datadir . $oc_username . "/files/" . $searchfolder));
        //while (false !== ($file = readdir($folderhandle)))
        $di = new RecursiveDirectoryIterator(utf8_encode($oc_datadir . $oc_username . "/files/" . $searchfolder));
        foreach (new RecursiveIteratorIterator($di) as $file)
        {
            // define the file to check
            if(file_exists($file) && (filetype($file)==="file"))
            {
                $oc_filepath = str_replace($userpath, '', $file);
                $oc_filepath_raw = utf8_decode($oc_filepath);

                echo $oc_filepath . PHP_EOL;

                $dbresult = mysql_query("SELECT * FROM `" . $db_name .  "`.`oc_filecache` WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                $numberofrows = mysql_num_rows($dbresult);

                if($numberofrows>1)
                {
                    // more than one entry found -> exception
                    echo "ERROR: More than one DB-entry found for the file files/" . $oc_filepath . PHP_EOL;
                }elseif($numberofrows===0){
                    // more than one entry found -> exception
                    echo "ERROR: No DB-entry found for the file files/" . $oc_filepath . PHP_EOL;
                }else{
                    // only one entry found -> OK
                    $row = mysql_fetch_object($dbresult);


                    $encryptedFilesize = filesize($oc_datadir . $oc_username . '/files/' . $oc_filepath);

                    if($fix_encryption_flag==='1'){
                        if(intval($row->encrypted)===0){
                            echo "INFO: Encryption-flag is 0. So we have to fix it." . PHP_EOL;
                            $update = mysql_query("UPDATE `" . $db_name .  "`.`oc_filecache` Set `encrypted` = '1' WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                        }
                    }

                    // compare encrypted file and stored filesize. if they match, we have to fix it
                    if(intval($row->size)===intval($encryptedFilesize)) {
                        echo "INFO: Filesize matches encrypted filesize. So we have to fix it." . PHP_EOL;


                        // we need to decrypt the file-key, therefore we use our private key and the share key
                        $shareKey = file_get_contents($oc_datadir . $oc_username . '/files_encryption/keys/files/' . $oc_filepath . '/OC_DEFAULT_MODULE/' . $oc_username . '.shareKey');
                        $encryptedKeyfile = file_get_contents($oc_datadir . $oc_username . '/files_encryption/keys/files/' . $oc_filepath . '/OC_DEFAULT_MODULE/fileKey');
                        try {
                            $decryptedKeyfile = $crypt->multiKeyDecrypt($encryptedKeyfile, $shareKey, $decryptedUserKey);
                        } catch (\Exception $ex) {
                            echo $ex->getMessage() . PHP_EOL;
                        }

                        // delete an existing temporary file
                        if (file_exists($oc_tmpdir . basename($oc_filepath)))
                        {
                            unlink($oc_tmpdir . basename($oc_filepath));
                        }

                        // read and decrypt the encrypted file, write the decrypted file
                        $fh_r = fopen($oc_datadir . $oc_username . '/files/' . $oc_filepath, 'r');
                        $fh_w = fopen($oc_tmpdir . basename($oc_filepath), 'w');

                        $encryptedContent = '';
                        $decryptedContent = '';

                        // read header (means removing it for decrypting the file)
                        $encryptedContent = fread($fh_r, 8192);
                        // read data
                        while ($encryptedContent = fread($fh_r, 8192)) {
                            // Decrypt the block
                            $decryptedBlock = $crypt->symmetricDecryptFileContent($encryptedContent, $decryptedKeyfile);
                            // A that point you can write the decrypted block to the decrypted file
                            // instead of appending it to $decryptedContent. But I don't fear memory
                            // limits and I choose to append the block to my string and write the
                            //  file at the end
                            //$decryptedContent .= $decryptedBlock;

                            // Write decrypted content to tmp file
                             fwrite($fh_w, $decryptedBlock);
                        }
                        fclose($fh_r);
                        fclose($fh_w);

                        // write the content to the tmp-folder
                        //file_put_contents($oc_tmpdir . basename($oc_filepath), $decryptedContent);

                        // read the real filesize of the file
                        $decryptedFilesize = filesize($oc_tmpdir . basename($oc_filepath));
                        //echo "Size of encrypted file: " . $encryptedFilesize . " Byte" . PHP_EOL;
                        //echo "Size of decrypted file: " . $decryptedFilesize . " Byte" . PHP_EOL;

                        // delete the temporary file
                        if (file_exists($oc_tmpdir . basename($oc_filepath)))
                        {
                            unlink($oc_tmpdir . basename($oc_filepath));
                        }

                        // update DB-entry
                        if(intval($row->size)===intval($decryptedFilesize))
                        {
                            // stored size is the expected size of the dencrypted file -> nothing to do
                            echo "INFO: Correct filesize of " . $row->size . " Bytes found in DB. Nothing to do here." . PHP_EOL;
                        }elseif(intval($row->size)===intval($encryptedFilesize))
                        {
                            // stored size is the expected size of the encrypted file -> correct the DB-entry
                            echo "INFO: Wrong filesize of " . $row->size . " Bytes found in DB. Will be fixed to " . $decryptedFilesize . " Bytes" . PHP_EOL;

                            $update = mysql_query("UPDATE `" . $db_name .  "`.`oc_filecache` Set `size` = '" . $decryptedFilesize . "' WHERE `path` LIKE 'files/" . $oc_filepath_raw . "'");
                            //echo "UPDATE `" . $db_name .  "`.`oc_filecache` Set size = '" . $decryptedFilesize . "' WHERE `path` LIKE 'files/" . $oc_filepath . "'" . PHP_EOL;
                        }else{
                            // size is not as expected -> exception
                            echo "ERROR: The filesize stored in the DB (" . $row->size . " Bytes) is not as expected (" . $encryptedFilesize . " Bytes)!" . PHP_EOL;
                        }
                    }else{
                        echo "INFO: Filesize seems to be OK. Nothing to do here." . PHP_EOL;
                    }
                }
            }
        }
        //closedir($folderhandle);
        mysql_close($db);
    }
} catch (\Exception $ex) {
    echo $ex->getMessage() . PHP_EOL;
}

?>

[chriseqipe]

Based on the first script presented here I made a experimental oc app, which can be used by end user. You can find it here:
The script expects php apc cache to be enabled!

https://gitlab.eqipe.ch/xwave-gmbh/ocfixwrongsizedfiles

This way end user can may fix the database by himself.
Please use on your own risks. Make Backup before running it.
The script works for us as expected.

[brunodegoyrans]

This script had partially solved issues on my owncloud,
but not all.

I think it's because how this script tries to detect if the size stored in database is right or not.
It compares encrypted file and stored filesize. if they match, script has to fix it (because encrypted filesize is always different compared to stored file).
Probably, there are some cases where filesizes are different, and the script thinks it's OK.
But the filesize in database is not correct.

To solve that issue, the script shall not compare the filesizes,
but it should do all the treatments whatever.
But it might take a long, long, loooong time.

[chriseqipe]

@brunodegoyrans
I think you mentioned a correct objection: I added a "force check ALL" checkbox.
You may test it.
Again: make backup before running the thing...

[Ninos]

@chriseqipe thx I'll wait until this app is finished and then try it out.

PS: Before I did not used encryption and had the same problem..

[chriseqipe]

@Ninos haven't tested without encryption enabled...
But seems to do it's work in our env. Will now also detect broken files, only containing the encryption header. We are currently investigating why this also happens some time (oc 8.1.4).

[Ninos]

K thx for response. Is there a roadmap atm for that oc application?

[PVince81]

@cmonteroluque @schiesbn many people are having the problems where encrypted files got the wrong size, mostly caused by migration. It would be good to provide the script as a repair step or occ command, ideally backportable to 8.1 because that's where people started having issues.

Another option is to make it a separate script/app, but that would be less integrated in ownCloud's APIs.

Setting to OC 9 to have a look in the scope of 9.

[cmonteroluque]

@PVince81 ok to have a look, yeah

[chriseqipe]

@cmonteroluque the problem why a occ command is not a practical solution, is that you need to have the password from the user having problems with wrong sized files. Only the owner of the files should do the repair. Private key is encrypted with users password and will be decrypted when user logs into the web client. Decrypted private key then resides inside the user session (how cool that is is another discussion about how meaningful the whole oc encryption really is btw)...

[PVince81]

Another idea is a button on the "Personal" page "repair encrypted files".
But this will very likely cause PHP request timeout issues due to the long running process.
Then there's the idea to make the button schedule a background job that would run with cron, but then it would also need the user's password to be available / stored in the background job data... not so good.

[crhbetz]

Hi, I'm coming over from #21720 - I used @chriseqipe 's app to solve my file size related problem there, however I found that encrypted files I had on the owncloud instance prior to upgrading to encryption 2 had another, shorter header than the one the app/script is using to identify encrypted files. I changed lines 217 and 246 to

if (preg_match("/^HBEGIN:cipher:AES-256-CFB:HEND/", $content) || preg_match("/^HBEGIN:oc_encryption_module:OC_DEFAULT_MODULE:cipher:AES-256-CFB:HEND/", $content)) {

to be able to fix the file size of both migrated and newly uploaded files. I'm sure this is not the elegant way to achieve this, but I never really learned PHP, and it works, at least ...
Maybe you'll want to consider checking for both/more header formats if you want to include the script into future releases.

[PVince81]

Alternative solution: for those where "encrypted" is true but the database's size is wrong (and is equal to the file-on-disk size), here is a ticket/PR that would automatically repair the sizes at download time: https://github.com/owncloud/core/issues/21751 and the PR for encryption: https://github.com/owncloud/core/pull/22008 (needs more testing though)

[PVince81]

The way it works is that you need the user to download the file once for the repair to happen. That first download will fail because when we know the correct size the file was already fully streamed to the client, so the first download fails. Subsequent downloads will then work properly.
One idea would be to have sync clients take care of the download/redownload in such situations.

[PVince81]

In 8.2.3 and 9.0.0 there is a fix that will automatically fix the size of encrypted files: https://github.com/owncloud/core/pull/22579

[PVince81]

forgot to mention that the fixing of the size is done while downloading the file, at download time.

[mSys-mislav]

@PVince81
I've tried to test the problems mentioned in my topic here: https://github.com/owncloud/core/issues/22042

but this is still not fixed in newer version of OC - I must still go in database and change flag to 1, then I can download the file and it's not corrupted. Before that I get an error that file doesn't even exist and it's corrupted once download (this was update from 8.2.2. to version 9 - I'm not even sure why it didn't update to 8.2.3. in the first place).

Also, in the new PDF manual, there is still no proper documentation on how to do this / restore procedure. Encryption is one of the most important things in the owncloud, to make it secure and that's not working.

[PVince81]

@mislav-eu you're right, reopening to look into how/where to automatically set the flag.

Maybe the encryption code could try and detect if there are encryption keys available for that file at scan time.

[PVince81]

@mislav-eu please raise a separate ticket for this, unrelated to size repair

[brunodegoyrans]

Hello

After few months of peace, I have to come here once again. Several file size errors came again.
I launch my fixwrongfilesized.php and it discovers errors. But this time, I obtain a php error when it tries to repair:
PHP Fatal error: Call to a member function t() on null in /var/www/owncloud/apps/encryption/lib/crypto/crypt.php on line 483

An idea of what is the reason?

I have read somewhere that this repair would be automatically launch during download? It is not the case apparently.

[PVince81]

@brunodegoyrans the repair at download time only kicks in if the detected size that's in the database matches the encrypted size instead of unencrypted size, or if that size is -1. We didn't make it repair in other cases because the repair code is expensive and would slow down all downloads.

As a workaround, if you know which files have wrong sizes you can try setting them to -1 in the database. Subsequent downloads will recompute their sizes.

[PVince81]

On some setups there might be too many files to be able to do a quick fix like this.

So it would be good to have a "occ encryption:rescan-sizes" command based on the script above. That script would require either the user's password or a recovery key to be able to verify and fix invalid sizes.

[PVince81]

We discovered another potential limitation which might not make this possible in all situations: when setting back the "encrypted" flag to 1 manually: 1 is not always the correct value. That value is used as "encrypted file version" and increases after every overwrite. So if the file that was restored from backup had a different version, it won't be decryptable. Detecting the version is not possible except by iterating over all values...

Ideal when restoring files from backup is really to also find the matching oc_filecache entry and adjust accordingly.