Prestashop: Virtual Product > 500mb slow download speed and download fails

[digitalhuman]

Describe the bug

We serve virtual product of > 500mb. There are 2 things happening. The download is extremely slow (45kb/sec) which in regular download cases exceeds 100mb/sec. The second problem is that the download fails often because the PHP engine times out. Apparently the download is served with PHP in Prestashop instead of by the webserver.

Expected behavior

Download speed should be the same as webserver downloads and should not be interrupted.

Steps to Reproduce

Steps to reproduce the behavior:

1: Create a 600mb file
2: Add as virtual product
3: Download file with 10 different connection all at the same time
4: Make sure you use different accounts, different computers, different internet connections
5: Most will fail to download or the file will be corrupt

Additional information

• PrestaShop version: 1.7.6.4
• PHP version: 7.2.27
• nGinx: 1.16.1

PHP-FPM config

; Note: This value is mandatory.
pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 150

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 10

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 5

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 10

; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
;pm.process_idle_timeout = 10s;

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries. For
; endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0
pm.max_requests = 1000

; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
request_terminate_timeout = 20m

PHP.ini

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 60

; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 256M

; Default timeout for socket based streams (seconds)
; http://php.net/default-socket-timeout
default_socket_timeout = 60

Nginx config

user nginx;
worker_processes auto;
worker_rlimit_nofile 60000;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.fedora.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 4000;
    multi_accept on;
    use epoll;
}

http {

    charset utf-8;
    server_tokens       off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" **$request_time/$upstream_response_time**';

    access_log  /var/log/nginx/access.log main;

    sendfile                    on;
    tcp_nopush                  on;
    tcp_nodelay                 on;
    keepalive_timeout           600;
    types_hash_max_size         2048;
    keepalive_requests          100000;
    reset_timedout_connection   on;
    send_timeout                600;
    client_body_timeout         600;
    client_max_body_size        800M;

    gzip                        on;
    gzip_static                 on;
    gzip_http_version           1.1;
    gzip_comp_level             8;
    gzip_min_length             128;
    gzip_vary                   on;
    gzip_proxied                any;
    gzip_buffers                16 8k;
    gzip_types
        application/atom+xml
        application/javascript
        application/json
        application/rss+xml
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/x-web-app-manifest+json
        application/xhtml+xml
        application/xml
        font/opentype
        image/png
        image/jpeg
        image/svg+xml
        image/x-icon
        text/css
        text/plain
        text/javascript
        text/x-component;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    fastcgi_cache_path          /var/nginx/cache_store levels=1:2 keys_zone=TCC:512m max_size=64m inactive=10m;
    fastcgi_cache_key           "$scheme$request_method$host$request_uri$http_cookie";
    fastcgi_cache_valid         200 15m;
    fastcgi_cache               TCC;
    fastcgi_cache_lock          on;
    fastcgi_cache_use_stale     error timeout invalid_header updating http_500;
    fastcgi_ignore_headers      Cache-Control Expires Set-Cookie;
    fastcgi_buffering           off;
    fastcgi_request_buffering   off;
    fastcgi_buffers             8 16k;
    fastcgi_buffer_size         32k;

Nginx Host config

server
{

# IPv6
# listen [::]:80;

# SSL Ipv4 & v6
listen 37.97.150.193:443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;

# Your SSL Certificates, don't forget to take a look at Certbot (https://certbot.eff.org)
include /etc/nginx/ssl_params.conf;

include /etc/nginx/letsencrypt.conf;

server_name ******;

error_log /var/log/nginx/prestashop.error_log;
access_log /var/log/nginx/prestashop.access_log;

root /home/vhost/domain.com/shop;

index index.php index.html;

# to control the amount that can be uploaded.
client_max_body_size 800M;

# set admin folder name
set $admin_dir /adminjemoeder;

location ~ /admin.*/(sell|api|common|_wdt|modules|improve|international|configure|addons|_profiler|product|combination|specific-price)/(.*)$ {
    #rewrite ^/admin[a-zA-Z][a-zA-Z]/index.php(.*)$ /index.php$1;

    try_files $uri $uri/ /index.php?q=$uri&$args $admin_dir/index.php$is_args$args;
}

# Cloudflare / Max CDN fix
location ~* \.(eot|otf|ttf|woff|woff2)$ {
    add_header Access-Control-Allow-Origin *;
}

# Do not save logs for these
location = /favicon.ico {
    log_not_found off;
    access_log off;
}

location = /robots.txt {
    auth_basic off;
    allow all;
    log_not_found off;
    access_log off;
}

location / {
    limit_req zone=mylimit burst=12 delay=8;

    # Redirect pretty urls to index.php
    try_files $uri $uri/ /index.php?$args;

    # Images
    rewrite ^/([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$1$2$3.jpg last;
    rewrite ^/([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$1$2$3$4.jpg last;
    rewrite ^/([0-9])([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$3/$1$2$3$4$5.jpg last;
    rewrite ^/([0-9])([0-9])([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$3/$4/$1$2$3$4$5$6.jpg last;
    rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$3/$4/$5/$1$2$3$4$5$6$7.jpg last;
    rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$1$2$3$4$5$6$7$8.jpg last;
    rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$7/$1$2$3$4$5$6$7$8$9.jpg last;
    rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$7/$8/$1$2$3$4$5$6$7$8$9$10.jpg last;
    rewrite ^/c/([0-9]+)(-[.*_a-zA-Z0-9-]*)(-[0-9]+)?/.+.jpg$ /img/c/$1$2$3.jpg last;
    rewrite ^/c/([a-zA-Z_-]+)(-[0-9]+)?/.+.jpg$ /img/c/$1$2.jpg last;

    # AlphaImageLoader for IE and fancybox
    rewrite ^images_ie/?([^/]+)\.(jpe?g|png|gif)$ js/jquery/plugins/fancybox/images/$1.$2 last;

    # Web service API
    rewrite ^/api/?(.*)$ /webservice/dispatcher.php?url=$1 last;
}

# Allow access to the ACME Challenge for Let's Encrypt
location ~ /\.well-known\/acme-challenge {
  allow all;
}

# Block all files with these extensions
location ~ \.(md|tpl)$ {
  deny all;
}

# File security
# .htaccess .DS_Store .htpasswd etc
location ~ /\. {
    deny all;
}
# Source code directories
location ~ ^/(app|bin|cache|classes|config|controllers|docs|localization|override|src|tests|tools|translations|travis-scripts|vendor|var)/ {
    deny all;
}
# Prevent exposing other sensitive files
location ~ \.(yml|log|tpl|twig|sass)$ {
    deny all;
}

# Prevent injection of php files
location /upload {
    location ~ \.php$ {
        deny all;
    }
}
location /img {
    add_header Cache-Control public;
    expires 1d;

    location ~ \.php$ {
        deny all;
    }
}

location ~ \.php$ {
    # Verify that the file exists, redirect to index if not
    try_files $fastcgi_script_name /index.php$uri&$args;
    fastcgi_index  index.php;
    include fastcgi_params;

    add_header X-Node 6 always;

    fastcgi_read_timeout 60m;
    fastcgi_send_timeout 60m;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 256 16k;
    fastcgi_max_temp_file_size 0;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_keep_conn on;

    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param SCRIPT_NAME $fastcgi_script_name;

    fastcgi_pass unix://var/run/php-fpm/php-production.sock;
}

}

[prestashop-issue-bot[bot]]

Thanks for opening this issue! We will help you to keep its state consistent

[khouloudbelguith]

Hi @digitalhuman,

Thanks for your report.
Ping @PierreRambaud what do you think? it is a performance issue.

Thanks!

[digitalhuman]

@khouloudbelguith @PierreRambaud Please keep in mind that we have > 1000 concurrent users on our platform. Our next product release could even mean > 10000 concurrent users + downloads. We can't release the product this way unless we have a solution for the mentioned issues. We are now forced to use a CDN with the risk of piracy of our virtual product.

[matks]

I just had a look at controller responsible for sending file content to browser, it uses echo fget($file)
https://github.com/PrestaShop/PrestaShop/blob/develop/controllers/front/GetFileController.php

Using readfile might optimize performance but it will not do Download speed should be the same as webserver downloads and should not be interrupted.

@digitalhuman the reason why PHP is responsible for the process is because there are multiple checks (is the customer logged in ? is the order paid ? how many times the product has been downloaded and does it comply with max. amount of download ?). I dont think you can have the performance of webserver download because this logic needs to be executed.

You could "hack" this process by performing logic checks and, at the end, redirecting users to the right filepath (so download would be done by webserver) but smart customers will be able to notice and reuse the final URL.

[matks]

I dont think there exists a relevant (or at least something that does need weeks of complex engineering) solution using PHP and that can handle > 10000 concurrent users + downloads other than an expensive infrastructure.

(by the way congratulations for the success of your products!)

The CDN seems actually like a good solution although indeed the right security must be set configured.
CDN _are_ built to handle massive quantity of simultaneous users 😁 it makes sense to use one.

Can you elaborate on We are now forced to use a CDN with the risk of piracy of our virtual product. ?

[digitalhuman]

hi @matks the problem is that the installer could be shared and the current does not have a serial key validation logic build in. Anyone with the correct URL can download the file. I think it would be good for Prestashop to look further into this and see if there can be a fix. I have some options in mind that I maybe, after evaluation will suggest of course.
However I wanted to see if the community had already experience with this issue.

[digitalhuman]

I just had a look at controller responsible for sending file content to browser, it uses echo fget($file)
https://github.com/PrestaShop/PrestaShop/blob/develop/controllers/front/GetFileController.php

Using readfile might optimize performance but it will not do Download speed should be the same as webserver downloads and should not be interrupted.

@digitalhuman the reason why PHP is responsible for the process is because there are multiple checks (is the customer logged in ? is the order paid ? how many times the product has been downloaded and does it comply with max. amount of download ?). I dont think you can have the performance of webserver download because this logic needs to be executed.

You could "hack" this process by performing logic checks and, at the end, redirecting users to the right filepath (so download would be done by webserver) but smart customers will be able to notice and reuse the final URL.

Hi @matks

Well why read the file at all? Why not force a redirect to do a download to a random 'download' file. You could even on the fly change the filename every once in a file. Yes I know disk I/O but still. Even without renaming, if you have a random SHA filename it would already hard to figure out.

[digitalhuman]

@matks I do believe that readfile has much better performance. Or we could also implement resumeable downloads by using the Content-Range header. I will do some testing and research, I will let you know.

[matks]

Well why read the file at all? Why not force a redirect to do a download to a random 'download' file. You could even on the fly change the filename every once in a file. Yes I know disk I/O but still. Even without renaming, if you have a random SHA filename it would already hard to figure out.

If I am not wrong, in order to be sure that user cannot bypass validations (to avoid him giving the link to someone else or downloading it more times than accepted) you need to:

1. perform PHP logic to validate download (user logged in, order is paid, download quota is not reached ...)
2. create the single-user file-to-be-downloaded with unique SHA filename
3. wait for user to download file*
4. when download has been completed (hard to be sure it's 100% complete on user side) remove the file-to-be-downloaded with unique SHA filename

One tricky part is that 4. will happen outside of PHP scope so you need the webserver, apache or nginx, to notify PHP about the successfull download
Another tricky part is "how long do we wait in step 3" ? I guess after some time you consider the "download timeslot" is over and you'll create another link + file

Moreover all this process must be resilient to failures:

• failure to create the file at step 2 (bad read/right permissions ? corrupted file ? out of disk usage), at least being able to detect failure and tell admin
• failure to detect download happened at step 4
• thinking file was downloaded at step 4 but because of network issue it actually failed, so user has not the file but you think he has it
• failure to remove the file at step 4
• issues with disk memory

Which is why I said

I dont think there exists a relevant (or at least something that does need weeks of complex engineering) solution

It starts to look like a complex process 😅

[matks]

@matks I do believe that readfile has much better performance. Or we could also implement resumeable downloads by using the Content-Range header. I will do some testing and research, I will let you know.

This seems to be a lot more simple to implement and actually quite promising 😄

I also believe it is possible to stream files through PHP 🤔 however I did some research, I did not find a standard solution. I see multiple solutions, many of them explained on stackoverflow, but no "this is THE way to stream php files". But I'm quite sure PHP Streams have very nice feature for this usecase

[digitalhuman]

@matks yeah I think the last suggestion is quite promising. I found this one: https://www.media-division.com/php-download-script-with-resume-option/. I have done exactly this before while building a YouTube proxy service.
I don't know how this will work under heavy load though. I think we should just stresstest a POC to proof this. The Streams class looks indeed what I used before there were OOP classes :) LOL /yeah i am old.

[davidglezz]

Years ago I modified my AttachmentController (Prestashop 1.6.1.1) to allow continue failed downloads. It is not well tested and did not go into production because it did not have time, but it can give an idea of how it works. The key is range headers like Content-Range

<?php

/*
 * 2007-2015 PrestaShop
 *
 * NOTICE OF LICENSE
 *
 * This source file is subject to the Open Software License (OSL 3.0)
 * that is bundled with this package in the file LICENSE.txt.
 * It is also available through the world-wide-web at this URL:
 * http://opensource.org/licenses/osl-3.0.php
 * If you did not receive a copy of the license and are unable to
 * obtain it through the world-wide-web, please send an email
 * to [email protected] so we can send you a copy immediately.
 *
 * DISCLAIMER
 *
 * Do not edit or add to this file if you wish to upgrade PrestaShop to newer
 * versions in the future. If you wish to customize PrestaShop for your
 * needs please refer to http://www.prestashop.com for more information.
 *
 *  @author PrestaShop SA <[email protected]>
 *  @copyright  2007-2015 PrestaShop SA
 *  @license    http://opensource.org/licenses/osl-3.0.php  Open Software License (OSL 3.0)
 *  International Registered Trademark & Property of PrestaShop SA
 */

class AttachmentControllerCore extends FrontController {

    public function postProcess() {
        $a = new Attachment(Tools::getValue('id_attachment'), $this->context->language->id);
        if (!$a->id)
            Tools::redirect('index.php');

        /*if (!Context::getContext()->customer->isLogged())
            Tools::redirect('index.php?controller=authentication');*/

        Hook::exec('actionDownloadAttachment', array('attachment' => &$a));

        if (ob_get_level() && ob_get_length() > 0)
            ob_end_clean();

        @set_time_limit(0);
        self::smartReadFile(_PS_DOWNLOAD_DIR_ . $a->file, utf8_decode($a->file_name), $a->mime);
        exit;
    }

    public static function smartReadFile($location, $filename, $mimeType = 'application/octet-stream') {
        if (!file_exists($location)) {
            header("HTTP/1.0 404 Not Found");
            return;
        }

        $size = filesize($location);
        $time = date('r', filemtime($location));

        $fm = @fopen($location, 'rb');
        if (!$fm) {
            header("HTTP/1.0 505 Internal server error");
            return;
        }

        $begin = 0;
        $end = $size;

        if (isset($_SERVER['HTTP_RANGE'])) {
            if (preg_match('/bytes=\h*(\d+)-(\d*)[\D.*]?/i', $_SERVER['HTTP_RANGE'], $matches)) {
                $begin = intval($matches[0]);
                if (!empty($matches[1]))
                    $end = intval($matches[1]);
            }
        }

        if ($begin > 0 || $end < $size)
            header('HTTP/1.0 206 Partial Content');
        else
            header('HTTP/1.0 200 OK');

        header("Content-Type: $mimeType");
        header('Cache-Control: public, must-revalidate, max-age=0');
        header('Pragma: no-cache');
        header('Accept-Ranges: bytes');
        header('Content-Length:' . ($end - $begin));
        header("Content-Range: bytes $begin-$end/$size");
        // inline | attachment
        header("Content-Disposition: attachment; filename=$filename");
        header("Content-Transfer-Encoding: binary\n");
        header("Last-Modified: $time");
        header('Connection: close');

        $cur = $begin;
        fseek($fm, $begin, 0);

        while (!feof($fm) && $cur < $end && (connection_status() == 0)) {
            print fread($fm, min(1024 * 16, $end - $cur));
            $cur += 1024 * 16;
        }
    }

}

Of course this is just to give a general idea and not a final solution to help create good implementation.

[digitalhuman]

ooh interesting

[PierreRambaud]

I like the idea of continuous download, and also add check to be sure the download can take more than the PHP process (timeout limit). But if you're using Nginx, you need to configure it to be sure the process can continue, it's important for security to be sure the connected user is able to download the virtual product and not everyone :)