Yii2: ActiveRecord attribute processing with binary data

[bologer]

What steps will reproduce the problem?

• I use 2amigos widget to upload (this issue is not related to them)
• I have Document controller where I process the data recevied from ajax call made by the widget, it look something like the following:

...
public function actionUpload() {
    $model = new UploadedFiles();
    $model->image = UploadedFile::getInstance($model, 'image');

    if ($model->image) {

        $_fileName = Html::encode(str_replace(' ', '_', $model->image->name));

        $fp = fopen($model->image->tempName, 'r');
        $content = fread($fp, filesize($model->image->tempName));
        fclose($fp);

        $model->name = $_fileName;
        $model->type = $model->image->type;
        $model->size = $model->image->size;

        $model->content = $content;
        $model->client_id = Yii::$app->user->identity->id;

        if ( $model->save() ) {
        ...

• I dont save the file on the server, but fread the content of tempName and save the record in the database
• Most of the images are uploaded normally without any problem. For example this is fine and this one takes about 30 secs or more to process.

AJAX response:

General: 
Request URL:http://domain.com/document/upload
Request Method:POST
Status Code:200 OK
Remote Address:127.0.0.1:80
Referrer Policy:no-referrer-when-downgrade

Response headers: 
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:91
Content-Type:text/html; charset=UTF-8
Date:Thu, 28 Sep 2017 08:24:08 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Keep-Alive:timeout=5, max=100
Pragma:no-cache
Server:Apache/2.4.7 (Ubuntu)
Vary:Accept-Encoding
X-Debug-Duration:75
X-Debug-Link:/debug/default/view?tag=59ccb1a812bc4
X-Debug-Tag:59ccb1a812bc4
X-Powered-By:PHP/5.5.9-1ubuntu4.22

Request headers: 
Accept:*/*
Accept-Encoding:gzip, deflate
Accept-Language:en,en-US;q=0.8,es;q=0.6,ru;q=0.4
Cache-Control:no-cache
Connection:keep-alive
Content-Length:178290
Content-Type:multipart/form-data; boundary=----WebKitFormBoundary0tlIAE5C8mf09xof
Cookie:o_p=404d5763beee9d0a2fe49cdf8d45d0ac343a47889c5294de1f32c5bf9fcbc6afa%3A2%3A%7Bi%3A0%3Bs%3A3%3A%22o_p%22%3Bi%3A1%3Bi%3A1506331028%3B%7D; _csrf-client=6b9dca009d2a9f31b9d296378af9737a123bf7215ace29e912d690d83888269aa%3A2%3A%7Bi%3A0%3Bs%3A12%3A%22_csrf-client%22%3Bi%3A1%3Bs%3A32%3A%22%D7%01%1AC%E0%C0%DB%E4.%81%9E%D6h%E2%FC%24%96%84%C2%F1%DB%09%C3C%27%A8%3A%FB%ACIg%DF%22%3B%7D; lang=6f86f2618f98de21705486f24c082ce7931d5a8d6ed0b3eecfda3471a576d000a%3A2%3A%7Bi%3A0%3Bs%3A4%3A%22lang%22%3Bi%3A1%3Bs%3A5%3A%22en-US%22%3B%7D; _csrf-backend=0bd65720524cb5b04c35985bdf580731de6f662c06fcce4c02db89de7a74fe1ea%3A2%3A%7Bi%3A0%3Bs%3A13%3A%22_csrf-backend%22%3Bi%3A1%3Bs%3A32%3A%22%BE%D0%AE%98%40%0E%29%9C%28%9D%EAFL%F5%FC%B2%2B%AB%25%0F%C7%AC%C3Y%F0%1F%7E%27-14N%22%3B%7D; o=5d9ee1b694332ca26bcfbf4c5df7f79ceb36322fa7f426f315e735ac17b0e6b1a%3A2%3A%7Bi%3A0%3Bs%3A1%3A%22o%22%3Bi%3A1%3Bi%3A1506502698%3B%7D; sc-backend=h04j7h7s2dbq3qdo69uf832a55; sc-client=9auoc3itlqcc5lpl9ghcvbjig6
Host:doamain.com
Origin:http://domain.com
Pragma:no-cache
Referer:http://domain.com/profile/update
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.79 Chrome/61.0.3163.79 Safari/537.36
X-CSRF-Token:yuoJPBfEcNBnFpHxzz0K7dkUB6HQu53aNtDjDc7j2q4d6xN_9wSrNEmXDyen3_bJT5DFUAuyXpkReNn2Yqq9cQ==
X-Requested-With:XMLHttpRequest

Request payload: 
------WebKitFormBoundary0tlIAE5C8mf09xof
Content-Disposition: form-data; name="UploadedFiles[image]"; filename="small.jpg"
Content-Type: image/jpeg


------WebKitFormBoundary0tlIAE5C8mf09xof--

Debugging

I was debugging the code for long time and tried to find the source of the problem and I found it:

• when I send ajax request to process the file, the whole action "stucks" on $model->save() method, which I believe does some processing of the binary data and it takes about 30-50 secs. I can say that by running the top command and seeing almost 90% CPU load by the apache.

What is the expected result?

Processing of the binary attributes should be fast, I believe that ActivreRecord does some extra processing of the inputs, which takes extra time.

What do you get instead?

Long processing of form data.

| Q | A
| ---------------- | ---
| Yii version | 2.0.12
| PHP version | 5.5+
| Operating system | Ubuntu 14.04

[TerraSkye]

Do u have a html purifier on it?

[bologer]

@TerraSkye nope, I figured out the problem by using base64_encode() and base64_decode().
Before inserting/getting the data.

Any soltion to this issue? Or maybe it is something wrong on my side that I do not know about?

[yii-bot]

Thank you for your question.
In order for this issue tracker to be effective, it should only contain bug reports and feature requests.

We advise you to use our community driven resources:

• The support forum
• The IRC chat room
• The Gitter chat room
• The Definitive Guide to Yii

If you are confident that there is a bug in the framework, feel free to provide information on how to reproduce it. This issue will be closed for now.

_This is an automated comment, triggered by adding the label question._

[bologer]

@samdark this is not the question. Try it yourself you will get the same bug. I solved it only by using base64_encode()/decode().

[samdark]

Yii by itself doesn't do any binary processing.

[rob006]

@samdark This could be related to query logging?

@bologer Did you try to disable logging and profiling for DB?
http://www.yiiframework.com/doc-2.0/yii-db-connection.html#$enableLogging-detail
http://www.yiiframework.com/doc-2.0/yii-db-connection.html#$enableProfiling-detail

[bologer]

@rob006 just tested it and it is actually the database query logging. Added db to the list of exceptions and it does not make anymore problems. I believe, this is a bug.

[samdark]

@bologer, @rob006 any idea about the fix?

[bologer]

@samdark do you really need to display binary data in the logs? does it make any sense?
I used base64_encode() to encode binary data before inserting it in db, and base64_decode() retrieve the original data back. I don't know if this is the real fix or there is a better solutiuon.

[samdark]

1. Sometimes yes. Especially if I've got something wrong at the server, cannot reproduce issue locally and want to debug it.
2. How do I know if it's binary or just a long string?

[bologer]

@samdark don't know whether it is about the stirng length or the binary data. But I know for sure that it happens on some of the images (not all). I used 2-3 different images, one of the smallest ones usually fails, this is something that I noticed.

[samdark]

A set of code, DB schema and image to reproduce the issue could be helpful. Would you be able to put it together starting with basic project template?

[bologer]

@samdark, here is the information:

Yes, I would be able to put it in basic tempalte, but a bit later, maybe for now you can look at the code below.

Db schema:

CREATE TABLE IF NOT EXISTS `uploaded_files` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `client_id` int(250) NOT NULL,
  `name` varchar(255) COLLATE utf8_bin NOT NULL,
  `type` varchar(30) COLLATE utf8_bin NOT NULL,
  `size` int(11) NOT NULL,
  `comment` varchar(1024) COLLATE utf8_bin DEFAULT NULL,
  `content` mediumblob NOT NULL,
  `deleted_at` int(11) NOT NULL,
  `updated_at` int(11) NOT NULL,
  `created_at` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `client_id_deleted_at_created_at` (`client_id`,`deleted_at`,`created_at`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=459 ;

Upload action:

public function actionUpload()
{
    try {
        $model = new UploadedFiles(['scenario' => UploadedFiles::SCENARIO_CLIENT]);

        $model->image = UploadedFile::getInstance($model, 'image');

        if ($model->image) {

            $_fileName = Html::encode(str_replace(' ', '_', $model->image->name));

            $fp = fopen($model->image->tempName, 'r');
            $content = fread($fp, filesize($model->image->tempName));
            fclose($fp);

            $model->name = $_fileName;
            $model->type = $model->image->type;
            $model->size = $model->image->size;

            $model->content = $content;
            $model->client_id = Yii::$app->user->identity->id;

            if ($model->save()) {

                return Json::encode([
                    'files' => [
                        'status' => 1,
                        'errors' => [],
                        'name' => Html::encode($_fileName),
                        'timestamp' => time()
                    ],
                ]);
            } else {

                if ($model->hasErrors()) {
                    $errors = [];

                    foreach ($model->errors as $key => $error) {
                        $errors[] = $error;
                    }

                    return Json::encode([
                        'files' => [
                            'status' => 0,
                            'errors' => $errors,
                            'timestamp' => time()
                        ]
                    ]);
                }
            }
        }

        return Json::encode([
            'files' => [
                'status' => 0,
                'errors' => [Yii::t('frontend', 'unable_to_upload_the_file')],
                'timestamp' => time()
            ]
        ]);
    } catch (Exception $exception) {
        throw new GuiException();
    }
}

View file:

<?php

use yii\helpers\Html;
use dosamigos\fileupload\FileUpload;
use yii\grid\GridView;
use yii\widgets\Pjax;

?>

<?= FileUpload::widget([
    'model' => $modelUpload,
    'attribute' => 'image',
    'url' => ['document/upload'],
    'clientEvents' => [
        'fileuploaddone' => "function (e, data) {

            var result = JSON.parse(data.result);
            var files = result.files;

            if( files.status === 1 ) {
                $('.upload-err').remove(); // Remove all previous errors 

                $.pjax.reload({container:'#uploaded-files-widget-pjax'});

                $.each(result.files, function (index, file) {                            
                    $(document).on('pjax:end', function() {
                        var jAdded = $('table#uploaded-files-table tbody tr:first-child');

                        jAdded.css({
                            'background-color': 'lightgreen'
                        });

                        setTimeout(function() {
                            jAdded.removeAttr('style');
                        }, 1000);
                    });
                });

                $('#progress').hide(1000);
            } else {
                $('#progress').hide(1000); // Hide progress bar 
                $('#progress .progress-bar').css( 'width', '0%' );

                $('.upload-err').remove(); // Remove all previous errors 
                var errs = files.errors; // get list of errors 

                if( errs.length >= 1 ) {
                    for(var i = 0; i <= errs.length; i++) {
                        var errEl = $('<div>')
                            .addClass('upload-err text-danger')
                            .text(errs[i]);

                            $('#progress').before(errEl);
                    }
                } else {
                    $('.upload-err').remove(); // Remove all previous errors
                    $('#progress').hide();
                }
            }                            
        }",
        'fileuploadfail' => "function(e, data) {
            $('#progress').hide(1000); 
            $('#progress .progress-bar').css( 'width', '0%' );  
        }",
        'fileuploadprogressall' => "function(e, data) {
            var progress = parseInt(data.loaded / data.total * 100, 10);
            $('#progress').show();
            $('#progress .progress-bar').css( 'width', progress + '%' );                        
        }",
    ],
]); ?>

<br><br>
<div class="progress" id="progress" style="display: none">
    <div class="progress-bar" role="progressbar" aria-valuenow="0" aria-valuemin="0"
         aria-valuemax="100">
        <span class="sr-only"></span>
    </div>
</div>

<?php Pjax::begin([
    'options' => [
        'id' => 'uploaded-files-widget-pjax'
    ]
]); ?>
<?= GridView::widget([
    'dataProvider' => $dataProvider,
    'layout' => '{items}{pager}',
    'pager' => [
        'firstPageLabel' => 'First',
        'lastPageLabel' => 'Last'
    ],
    'tableOptions' => [
        'class' => 'table table-striped table-bordered',
        'id' => 'uploaded-files-table'
    ],
]); ?>
<?php Pjax::end(); ?>

Images

• All good
• This one fails

[samdark]

How's your logger configured?

[rob006]

@bologer Can you save DB logs for these queries and compare them? It is hard to say what is going on here.

[schmunk42]

@bologer Which database and version do you use? Do you see the binary string in the INSERT query?

It's a wild guess, but recently we had some serious issues with Unicode characters and MySQL databases ... and since the blob works fine, but the query logging has issues, I thought it could be related.

You might search for 4 byte unicode characters in the file you are uploading

Turns out MySQL’s utf8 charset only partially implements proper UTF-8 encoding. It can only store UTF-8-encoded symbols that consist of one to three bytes; encoded symbols that take up four bytes aren’t supported.

https://mathiasbynens.be/notes/mysql-utf8mb4

[samdark]

@schmunk42 utf8 encoding never made it slow. It was just messing characters up.

[bologer]

I close this as it was misconfiguration of the logs. Basically it was because binary data was saved into logs which took quite some time for some reason.

Thank you very much for your help @samdark, @schmunk42 and @rob006