Docker: Upload file rate very slow

[mgscreativa]

Hi! I have setup nextcloud-fpm image with nginx image proxy on a LAN homebrew server (core i7, 16 gb ram, 120GB ssd, etc). The nextcloud is accesses through an Apache server as proxy in the same server.

Everything works just fine but sync client in my desktop is really slow at uploading files, it uploads a bunch of files and then pauses for a short time. Files being uploaded aren't big (ttf, png, dll files, and so on).

What can I do to improve upload rate?

Screen recording (be patient, animated GIF but upload rate is soooo slow that it may be like a static GIF...)

My setup is as follows

docker-compose.yml:

version: '3'

services:
  app:
    image: nextcloud:stable-fpm
    restart: always
    container_name: nextcloud
    volumes:
      - /media/data/nextcloud/data/html:/var/www/html
      - /media/data/nextcloud/data/apps:/var/www/html/custom_apps
      - /media/data/nextcloud/data/config:/var/www/html/config
      - /media/data/nextcloud/data/cloud_data:/var/www/html/data

  web:
    build: ./docker/web
    restart: always
    hostname: 'cloud.mgscreativa.com.ar'
    container_name: nginx-nextcloud
    ports:
      - 3080:80
    volumes:
      - /media/data/nextcloud/data/html:/var/www/html:ro
    depends_on:
      - app

./docker/web/Dockerfile

FROM nginx:alpine

COPY nginx.conf /etc/nginx/nginx.conf

./docker/web/nginx.conf

worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    upstream php-handler {
        server app:9000;
    }

    server {
        listen 80;

        # Add headers to serve security related headers
        # Before enabling Strict-Transport-Security headers please read into this
        # topic first.
        # add_header Strict-Transport-Security "max-age=15768000;
        # includeSubDomains; preload;";
        #
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header X-Robots-Tag none;
        add_header X-Download-Options noopen;
        add_header X-Permitted-Cross-Domain-Policies none;
        add_header Referrer-Policy no-referrer;

        root /var/www/html;

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # The following 2 rules are only needed for the user_webfinger app.
        # Uncomment it if you're planning to use this app.
        #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
        #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
        # last;

        location = /.well-known/carddav {
            return 301 $scheme://$host/remote.php/dav;
        }
        location = /.well-known/caldav {
            return 301 $scheme://$host/remote.php/dav;
        }

        # set max upload size
        client_max_body_size 10G;
        fastcgi_buffers 64 4K;

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Uncomment if your server is build with the ngx_pagespeed module
        # This module is currently not supported.
        #pagespeed off;

        location / {
            rewrite ^ /index.php$request_uri;
        }

        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
            deny all;
        }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
            deny all;
        }

        location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
            fastcgi_split_path_info ^(.+\.php)(/.*)$;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_path_info;
            # fastcgi_param HTTPS on;
            #Avoid sending the security headers twice
            fastcgi_param modHeadersAvailable true;
            fastcgi_param front_controller_active true;
            fastcgi_pass php-handler;
            fastcgi_intercept_errors on;
            fastcgi_request_buffering off;
        }

        location ~ ^/(?:updater|ocs-provider)(?:$|/) {
            try_files $uri/ =404;
            index index.php;
        }

        # Adding the cache control header for js and css files
        # Make sure it is BELOW the PHP block
        location ~ \.(?:css|js|woff|svg|gif)$ {
            try_files $uri /index.php$request_uri;
            add_header Cache-Control "public, max-age=15778463";
            # Add headers to serve security related headers (It is intended to
            # have those duplicated to the ones above)
            # Before enabling Strict-Transport-Security headers please read into
            # this topic first.
            # add_header Strict-Transport-Security "max-age=15768000;
            #  includeSubDomains; preload;";
            #
            # WARNING: Only add the preload option once you read about
            # the consequences in https://hstspreload.org/. This option
            # will add the domain to a hardcoded list that is shipped
            # in all major browsers and getting removed from this list
            # could take several months.
            add_header X-Content-Type-Options nosniff;
            add_header X-XSS-Protection "1; mode=block";
            add_header X-Robots-Tag none;
            add_header X-Download-Options noopen;
            add_header X-Permitted-Cross-Domain-Policies none;
            add_header Referrer-Policy no-referrer;

            # Optional: Don't log access to assets
            access_log off;
        }

        location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
            try_files $uri /index.php$request_uri;
            # Optional: Don't log access to other assets
            access_log off;
        }
    }

}

NextCloud config

<?php
$CONFIG = array (
  'instanceid' => 'oc8o4useipus',
  'passwordsalt' => 'DLmLNK4uHvVbXBaM1QdW2MVHWalYEL',
  'secret' => 'P9E5IOdvt1kkoWjmFqBTneOOg5nclf09/qxuhGBdOXij/4rD',
  'trusted_domains' => 
  array (
    0 => 'cloud.mgscreativa.com.ar',
    1 => 'lucas.mgscreativa.com.ar',
    2 => 'localhost',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '15.0.5.3',
  'overwrite.cli.url' => 'https://cloud.mgscreativa.com.ar',
  'overwriteprotocol' => 'https',
  'overwritehost' => 'cloud.mgscreativa.com.ar',
  'trusted_proxies' => array('10.0.0.77'),
  'dbname' => 'XXXXXX',
  'dbhost' => '10.0.0.77',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'XXXXXX',
  'dbpassword' => 'XXXXXX',
  'installed' => true,
  'logtimezone' => 'America/Argentina/Buenos_Aires',
  'forcessl' => false,
  'maintenance' => false,
  'default_language' => 'es',
  'loglevel' => 5,
);

Apache proxy configuration

<Proxy />
  Order deny,allow
  Allow from all
</Proxy>

ProxyPreserveHost On
ProxyPass /.well-known/acme-challenge/ !
ProxyPass / http://lucas.mgscreativa.com.ar:3080/
ProxyPassReverse / http://lucas.mgscreativa.com.ar:3080/

[cseelmann]

Are you sure your are not uploading via your external connection back to your local server and your line is the bottleneck?

I use pihole as local DNS and just poined my external nextcloud hostname to the local LAN ip, therefore all my devices in my LAN / Wifi will use the LAN ip when I am at home and the internet ip when outside. I must add I am using a tunnel ipv4 as my local link only has ipv6 ds-lite and my DNS points to the ipv4 tunnel.

[mgscreativa]

Hi @cseelmann thanks for taking time to review this!

In my case, on my desktop (Ubuntu Desktop 18.04) I have mapped my LAN server with hosts file. Ping gives me correct LAN IP of my server si I don't think the NC client is goint outside my lan to do this, it's something to do with the docker container I guess, maybe something I need to configure better... really don't know

[mgscreativa]

I have noticed that the files are uploaded in batches of 3 at a time, so, the pattern is, upload 3 files, then pauses very long time for no reason, then upload another 3 files, and so on...

[marceljd]

Does your database have enough resources (memory, cpu)? Especially with small files this could cause serious delays. The same goes for the proxy. Could you increase memory and cpu? Your hardware should be more than sufficient. And using Redis could also help.

[mgscreativa]

Hi @marceljd how can I double check this resources thing? What config files should I touch in my server? I have setup NC to use external DB, the one in the host server...

[marceljd]

If your database is on the host server the memory and cpu should not be a problem, unless you restricted them in any way. But it also means that all communication is routed outside of your docker environment, so I think the proxy is too restricted.

You can do 2 things now:

• only look at your proxy first (you should probably give the proxy more memory / cpu anyway)
• expand your nextcloud installation with a database and Redis (to reduce the traffic through the proxy)

What command did you use to start the proxy? Can you give more CPU / memory?

I would advise you to start a database and a Redis container next to NC, that will work best.

[mgscreativa]

Thanks @marceljd ! I gave it a shot to include MariaDB in the docker-compose.yml file, but it got worse, even after adding Redis to it...Seems that the connection to the mariadb and NC got through the internet, really weird! My router WAN light went crazy after that!

Tried the same setup using Redis but leaving the DB connection to my host and that's the best I will get I think, I also disabled all Activity feeds so the only table currently updating is oc_filecache.

This is my final docker-compose.yml and it got fairly well whit this...see screencast

version: '3'

services:
  redis:
    image: redis:alpine
    restart: always
    container_name: nextcloud-redis

  app:
    image: nextcloud:stable-fpm
    restart: always
    container_name: nextcloud-fpm
    volumes:
      - /media/data/nextcloud/data/fpm/html:/var/www/html
      - /media/data/nextcloud/data/fpm/apps:/var/www/html/custom_apps
      - /media/data/nextcloud/data/fpm/config:/var/www/html/config
      - /media/data/nextcloud/data/fpm/cloud_data:/var/www/html/data
    environment:
      - REDIS_HOST=redis
    depends_on:
      - redis

  web:
    build: ./docker/web
    restart: always
    container_name: nextcloud-nginx
    ports:
      - 3080:80
    volumes:
      - /media/data/nextcloud/data/fpm/html:/var/www/html:ro
    depends_on:
      - app

  cron:
    image: nextcloud:stable-fpm
    restart: always
    container_name: nextcloud-cron-fpm
    volumes:
      - /media/data/nextcloud/data/fpm/html:/var/www/html
    entrypoint: /cron.sh
    depends_on:
      - redis

[marceljd]

Why didn't you create a database inside the docker-compose file? Like Redis you can create a database and point to db or database or something like that in the Nextcloud container.

Personally I am using the Nextcloud Apache image with MySQL and Redis, and it is working really smooth and fast. You will have 3 containers then: Nextcloud web (Apache), DB and Redis.

[mgscreativa]

Hi @marceljd thanks for taking a time to chek this issue.

I finally decided to try your suggestion and add MariaDB to my compose file, but slowness is still an issue. It seems that NC it's having a hard time syncing folders with lots of small files (txt, ttf, gif, php, etc), this folder has 350GB and 150000 files and it's taking forever to review sync. I had NC installed on the host and it used to work just fine, then I deleted it in favor of Docker image (my bad, should have waited a little longer :( )

So far the process runs like this:

• I create the sync link in NC desktop client (the heavy folder with lots of small files)
• It starts to sync really well, it hits a directory tree with lots of photos and videos
• It goes really well until GB 40 aprox.
• At that point it starts to hit folders with small files and it slows down from 3 hs to 7 days and then noting.
• After restarting my desktop, the sync process never starts, see screencast

Look the screen cast, at capture time, I's been 1 hour of "Searching for changes...".

My compose file now looks like this:

version: '3'

services:
  db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    container_name: nextcloud-mariadb
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=XXXXXX
    env_file:
      - ./docker/db.env

  redis:
    image: redis:alpine
    restart: always
    container_name: nextcloud-redis

  app:
    image: nextcloud:stable-apache
    restart: always
    container_name: nextcloud-apache
    ports:
      - 3080:80
    volumes:
      - /media/data/nextcloud/data/apache/html:/var/www/html
      - /media/data/nextcloud/data/apache/apps:/var/www/html/custom_apps
      - /media/data/nextcloud/data/apache/config:/var/www/html/config
      - /media/data/nextcloud/data/apache/cloud_data:/var/www/html/data
    environment:
      - REDIS_HOST=redis
      - MYSQL_HOST=db
    env_file:
      - ./docker/db.env
    depends_on:
      - db
      - redis

  cron:
    image: nextcloud:stable-apache
    restart: always
    container_name: nextcloud-cron-apache
    volumes:
      - /media/data/nextcloud/data/apache/html:/var/www/html
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis

volumes:
  db:

[marceljd]

My experience is that many small files cause a big load on the database. When you start this setup it will be slower as normal as well because Redis has to get started slowly. I think you should try this for a few days and then decide what you want to do. After my initial upload (which took days) it is working fast, but I did not try to upload many small files after the first upload.

[marceljd]

I see the update now, I would really let it run for a few days. My sync was 500 GB and it took a long time (days). Question is if your PC or the server (or both) have trouble keeping up. What does your PC task manager say about the load of the CPU and disk?

[mgscreativa]

Ohh, I see...so in your opinion I should leave it as is, and see what happens in the comming days...

[marceljd]

Yes, at least that is what I did and it works fine now. Just be sure that your PC is still connected to the server and doing things, the rest will be ok when you just let it do its things...

[mgscreativa]

Well, you where right, now the sync time got down to 3 hours left...seems that NC it's having a hard time syncing small files...

[mgscreativa]

Finally I think I got it, I was able to recover my old NC installation from a server disk image and notices that I tweaked the .user.ini file inside NC with this vaules:

upload_max_filesize=1G
post_max_size=1G
memory_limit=1024M
mbstring.func_overload=0
always_populate_raw_post_data=-1
default_charset='UTF-8'
output_buffering=Off
register_globals=Off
max_execution_time=6000
short_open_tag=Off
date.timezone='America/Argentina/Buenos_Aires'
display_errors=Off
html_errors=Off
error_reporting=E_ALL
log_errors=On
error_log='/var/www/html/data/php.log'
magic_quotes_gpc=Off
expose_php=Off

Applying that to the docker image .user.ini file gives me this! (which is very acceptable...)

[LitvK]

You should add these lines to nginx config:

        keepalive_timeout 10m;
        proxy_connect_timeout  20s;
        proxy_send_timeout  600s;
        proxy_read_timeout  150s;

        client_body_buffer_size 70m;
        client_header_buffer_size 50k;
        large_client_header_buffers 2 50k;
        client_max_body_size 0;

The most important line is client_body_buffer_size 70m;. It cerates 70Mb in memory buffer for every connection. If this buffer is not enough for your file, nginx will create temporary file to store data. Then it will read it and send to apache.

I think that is the reason why your nginx-nextcloud is so slow.

[LitvK]

This variable has "8k" by default, so nginx will create temporary file for every uploaded file which is biger then 8Kb. :-D

[iguy0]

I had a huge performance increase by using @LitvK nginx's config.

Thanks

[montvid]

I am testing official nextcloud 18, stable, latest docker images with gvfs webdav + rsync -vPh 1 file and i get slow uploads comparing to rclone "serve webdav". Using debian 10, btrfs, docker 20.10.0~3-0~debian-buster. See:
NC stable "sent 1.39G bytes received 35 bytes 28.69M bytes/sec"
NC latest "sent 1.39G bytes received 35 bytes 29.29M bytes/sec"
NC 18 "sent 1.39G bytes received 35 bytes 38.12M bytes/sec"
rclone "sent 1.39G bytes received 35 bytes 61.84M bytes/sec"
Could someone fix the official images to get better upload speeds?