Invidious: High CPU Usage (again)

[tmiland]

CPU Usage is back up again. Mentioned earlier in #426 and #225

CPU load averages: | 1.67 (1 mins) , 1.68 (5 mins) , 1.66 (15 mins)
-- | --

ID | Owner | CPU | Command
-- | -- | -- | --
15675 | invidious | 46.6 % | /home/invidious/invidious/invidious -o invidious.log

Config.yml:

---
channel_threads: 1
feed_threads: 1
db:
  user: kemal
  password: 
  host: 
  port: 5432
  dbname: invidious
full_refresh: false
https_only: true
domain: 
use_pubsub_feeds: true
hmac_key: 
use_feed_events: true
default_home: Trending
feed_menu:
- Top
- Trending
- Popular
- Subscriptions
top_enabled: true
captcha_enabled: true
login_enabled: true
registration_enabled: true
statistics_enabled: true
admins:
- 
external_port: 443
default_user_preferences:
  annotations: false
  annotations_subscribed: false
  autoplay: false
  captions:
  - 
  - 
  - 
  comments:
  - youtube
  - 
  continue: false
  continue_autoplay: true
  dark_mode: false
  latest_only: false
  listen: false
  local: false
  locale: en-US
  max_results: 40
  notifications_only: false
  quality: hd720
  redirect_feed: false
  related_videos: true
  sort: published
  speed: 1.0
  thin_mode: false
  unseen_only: false
  video_loop: false
  volume: 100
dmca_content: []
check_tables: true
cache_annotations: false
hsts: true

[omarroth]

I'm having trouble reproducing with the above config on master. You might try setting either channel_threads and/or feed_threads to 0 to see if it's an issue with either of those jobs.

$ ./invidious -v
{
  "name": "invidious",
  "version": "0.18.0-eecf76c",
  "branch": "master"
}

built with crystal build src/invidious.cr --release and

$ crystal -v
Crystal 0.29.0 (2019-06-06)

LLVM: 6.0.1
Default target: x86_64-pc-linux-gnu

[tmiland]

Still the same with:

channel_threads: 0
feed_threads: 1

and:

channel_threads: 1
feed_threads: 0

./invidious -v
{
  "name": "invidious",
  "version": "0.18.0-de8d244",
  "branch": "On branch master"
}

crystal -v
Crystal 0.29.0 [fbfe8b62f] (2019-06-05)

LLVM: 4.0.0
Default target: x86_64-unknown-linux-gnu

Tiny bit better with both set to 0

channel_threads: 0
feed_threads: 0

ID | Owner | CPU | Command
-- | -- | -- | --
20559 | invidious | 31.8 % | /home/invidious/invidious/invidious -o invidious.log

[omarroth]

I'm having trouble finding de8d244, is it in a fork?

[tmiland]

I'm having trouble finding de8d244, is it in a fork?

That's correct. https://github.com/invidiou-sh/invidious/commit/de8d2442d22beab0f56b760006822d2b5dae319b

[tmiland]

"Solved" by adding:

CPUAccounting=true
CPUQuota=25%

to invidious.service.

Like so:

[Unit]
Description=Invidious (An alternative YouTube front-end)
After=syslog.target
After=network.target

[Service]
RestartSec=2s
Type=simple

User=invidious
Group=invidious

WorkingDirectory=/home/invidious/invidious
ExecStart=/home/invidious/invidious/invidious -o invidious.log
CPUAccounting=true
CPUQuota=25%

Restart=always

[Install]
WantedBy=multi-user.target

CPU load averages: | 0.67 (1 mins) , 0.99 (5 mins) , 1.41 (15 mins)
-- | --

ID | Owner | CPU | Command
-- | -- | -- | --
4719 | invidious | 23.7 % | /home/invidious/invidious/invidious -o invidious.log

[omarroth]

It appears you've already found a solution, but if you'd like to reopen I'd be happy to continue debugging this.

[tmiland]

It appears you've already found a solution, but if you'd like to reopen I'd be happy to continue debugging this.

That'd be great. I have not a clue why this would be happening.

Not really a solution though 😬

Currently on the exact same setup on the other server (same OS, repo, crystal version (Crystal 0.29.0 [fbfe8b62f] (2019-06-05)) etc):

ID | Owner | CPU | Command
-- | -- | -- | --
17598 | invidious | 2.2 % | /home/invidious/invidious/invidious -o invidious.log

But on the new server, it is both high CPU (46.6 % +- if not limited) and Memory usage.

ID | Owner | Size | Command
-- | -- | -- | --
12889 | invidious | 3.96 GB | /home/invidious/invidious/invidious -o invidious.log

Seen it as high as 20+ GB... While on the other server it's 254.20 MB currently.

PostgreSQL-DB resides on the new host, which both instances are using a.t.m.

[omarroth]

Thanks. I was able to reproduce this locally as well, see if 3be1c9261f12b44a8e84597bc59d1fe3d8504f87 fixes the issue for you.

[tmiland]

No problem. Alright, updated and now looks like this:

CPU load averages: | 0.78 (1 mins) , 1.51 (5 mins) , 1.63 (15 mins)
-- | --

ID | Owner | CPU | Command
-- | -- | -- | --
6099 | invidious | 33.3 % | /home/invidious/invidious/invidious -o invidious.log

ID | Owner | Size | Command
-- | -- | -- | --
6099 | invidious | 983.96 MB | /home/invidious/invidious/invidious -o invidious.log

[omarroth]

I'm a bit stumped, mind providing shard.lock for the currently running instance?

[tmiland]

I'm a bit stumped, mind providing shard.lock for the currently running instance?

Me too 😳

Here it is:
```
version: 1.0
shards:
db:
github: crystal-lang/crystal-db
version: 0.5.1

exception_page:
github: crystal-loot/exception_page
version: 0.1.2

kemal:
github: kemalcr/kemal
version: 0.25.2

kilt:
github: jeromegn/kilt
version: 0.4.0

pg:
github: will/crystal-pg
version: 0.16.1

radix:
github: luislavena/radix
version: 0.3.9

sqlite3:
github: crystal-lang/crystal-sqlite3
version: 0.12.0

[omarroth]

Only difference I have is in crystal-db, which is pinned to the latest commit. Mind running shards update && shards install and recompiling?

[tmiland]

Only difference I have is in crystal-db, which is pinned to the latest commit. Mind running shards update && shards install and recompiling?

Done on both instances. Still same content in shard.lock

Tiny bit better a.t.m

ID | Owner | CPU | Command
-- | -- | -- | --
29073 | invidious | 29.4 % | /home/invidious/invidious/invidious -o invidious.log

ID | Owner | Size | Command
-- | -- | -- | --
29073 | invidious | 2.10 GB | /home/invidious/invidious/invidious -o invidious.log

[omarroth]

You might try with:

top_enabled: false

[tmiland]

You might try with:

top_enabled: false

It's about the same, though it's down by 18 % or so.

ID | Owner | CPU | Command
-- | -- | -- | --
7328 | invidious | 28.6 % | /home/invidious/invidious/invidious -o invidious.log

I'll let it sit until tomorrow, and I'll post an update.

Thanks so far :+1:

[tmiland]

Here's how it looks currently:

ID | Owner | CPU | Command
-- | -- | -- | --
7328 | invidious | 31.1 % | /home/invidious/invidious/invidious -o invidious.log

ID | Owner | Size | Command
-- | -- | -- | --
7328 | invidious | 20.36 GB | /home/invidious/invidious/invidious -o invidious.log

[omarroth]

Mind commenting out:
https://github.com/omarroth/invidious/blob/ca79e81b39563b7fac0f8d8d895068999b89d245/src/invidious.cr#L135
https://github.com/omarroth/invidious/blob/ca79e81b39563b7fac0f8d8d895068999b89d245/src/invidious.cr#L176-L180
https://github.com/omarroth/invidious/blob/ca79e81b39563b7fac0f8d8d895068999b89d245/src/invidious.cr#L183-L187

and rebuilding to see if one of those might be causing the issue?

[tmiland]

ID | Owner | CPU | Command
-- | -- | -- | --
26149 | invidious | 41.4 % | /home/invidious/invidious/invidious -o invidious.log

It's still about the same. Commented out all three.

[omarroth]

Does CPU usage jump immediately after starting? Do you still encounter it with no load?

[tmiland]

Does CPU usage jump immediately after starting?

Tried to stop invidious, CPU went down, and back up again when it was started.

Do you still encounter it with no load?

Turning on nginx maintenance mode, it drops down to:

ID | Owner | CPU | Command
-- | -- | -- | --
31960 | invidious | 5.8 % | /home/invidious/invidious/invidious -o invidious.log

[haizrul]

Turning on nginx maintenance mode

Is there any alternative for apache? Having the same problem.

[tmiland]

Turning on nginx maintenance mode

Is there any alternative for apache? Having the same problem.

The issue was, that the instance was under constant attack, and mitigating the attacks lowered the CPU usage.

[haizrul]

constant attack

You mean DDOS attack?

mitigating the attacks lowered the CPU usage.

What did you do to prevent this?

[tmiland]

You mean DDOS attack?

Not constantly no, but should put in place the right tools to mitigate this as well.

What did you do to prevent this?

Have done a lot of things.

• Added blacklists in both firewall and nginx.
• Added tools like DDOS Deflate and stopspam with some modifications to add my own blacklists.
• Blocking bad bots with Nginx Bad Bot and User-Agent Blocker
• Blocking fake google bots with findfakebots.sh (IPs are auto added to custom blacklist)
• Monitoring access logs with traffic analyser
• Blocked Amazon IP's, which was the most attacking IP's
  Made a custom script for that as well:

Save as aws_blocklist.sh

#!/usr/bin/env bash


## Author: Tommy Miland (@tmiland) - Copyright (c) 2019

if [[ "$EUID" -ne 0 ]]; then
  echo -e "Sorry, you need to run this as root"
  exit 1
fi

#------------------------------------------------------------------------------#
#
# MIT License
#
# Copyright (c) 2019 Tommy Miland
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#
#------------------------------------------------------------------------------#

# To be used with Nginx Bad Bot and User-Agent Blocker
# https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker

# Add to /etc/nginx/bots.d/blacklist-ips.conf
# Amazon IP subnets
# include snippets/IPv4_aws_blacklist.conf;
# include snippets/IPv6_aws_blacklist.conf;

NGINX_PATH=/etc/nginx
# Detect absolute and full path as well as filename of this script
cd "$(dirname $0)"
CURRDIR=$(pwd)
SCRIPT_FILENAME=$(basename $0)
cd - > /dev/null
sfp=$(readlink -f "${BASH_SOURCE[0]}" 2>/dev/null || greadlink -f "${BASH_SOURCE[0]}" 2>/dev/null)
if [ -z "$sfp" ]; then sfp=${BASH_SOURCE[0]}; fi
SCRIPT_DIR=$(dirname "${sfp}")
# Icons used for printing
ARROW='➜'
DONE='✔'
ERROR='✗'
WARNING='⚠'
# Colors used for printing
RED='\033[0;31m'
BLUE='\033[0;34m'
BBLUE='\033[1;34m'
GREEN='\033[0;32m'
ORANGE='\033[0;33m'
DARKORANGE="\033[38;5;208m"
CYAN='\033[0;36m'
DARKGREY="\033[48;5;236m"
NC='\033[0m' # No Color
# Text formatting used for printing
BOLD="\033[1m"
DIM="\033[2m"
UNDERLINED="\033[4m"
INVERT="\033[7m"
HIDDEN="\033[8m"
# Command arguments
blocklist_type=`echo "$1"`

# Check input
check_input() {
  echo ""
  echo -e "${ORANGE}${INVERT}${WARNING}${BOLD} AWS Nginx Blocklist Generator ${NC}"
  echo ""
  echo -e "${ORANGE}${ARROW} Usage:${NC}${GREEN} ./aws_blocklist.sh [nginx/ngxblocker] ${NC}"
  echo ""
}

INPUT_CHECK=$(check_input)

#check command input
if [[ -z "$1" ]];
then
  echo -e "${INPUT_CHECK}"
  exit 0
fi

curl -o aws.json https://ip-ranges.amazonaws.com/ip-ranges.json 2> /dev/null

## temp file location ##
TEMP_IPv4="$(mktemp /tmp/IPv4_aws_blacklist.XXXXXXXX)"
TEMP_IPv6="$(mktemp /tmp/IPv4_aws_blacklist.XXXXXXXX)"

# Get all IPv4 addresses
IPv4=$(jq -r '.prefixes | .[].ip_prefix' < aws.json > $TEMP_IPv4)
# Get all IPv6 addresses
IPv6=$(jq -r '.ipv6_prefixes | .[].ipv6_prefix' < aws.json > $TEMP_IPv6)

CONF_OUT=$NGINX_PATH/snippets/"$1"_aws_blacklist.conf

# Remove duplicates
gawk -i inplace '!seen[$0]++' "$TEMP_IPv4";
gawk -i inplace '!seen[$0]++' "$TEMP_IPv6";

if [ "$1" == "ngxblocker" ]
then
  # Ngxblocker Blacklist
  awk '{ print "" $1 " 1;" }' "$TEMP_IPv4" > $CONF_OUT;
  awk '{ print "" $1 " 1;" }' "$TEMP_IPv6" >> $CONF_OUT;

elif [ "$1" == "nginx" ]
then
  # Nginx Blacklist

  awk '{ print "deny " $1 ";" }' "$TEMP_IPv4" > $CONF_OUT;
  awk '{ print "deny " $1 ";" }' "$TEMP_IPv6" >> $CONF_OUT;

elif [ "$1" == "csf" ]
then
  CONF_OUT=/etc/csf/csf.deny
  # csf Blacklist
  awk '{ print "" $1 " # Amazon IP subnets - do not delete" }' "$TEMP_IPv4" >> $CONF_OUT;
  awk '{ print "" $1 " # Amazon IP subnets - do not delete" }' "$TEMP_IPv6" >> $CONF_OUT;
else
  echo -e "${INPUT_CHECK}"
fi

# Copy to nginx path
# for f in IPv*.conf; do
#   cp -rp "$f" $NGINX_PATH/snippets/;
# done

rm -f aws.json && rm -f $TEMP_IPv4 && rm -f $TEMP_IPv6

To install Nginx, i recommend nginx-autoinstall which have added option to install bad bot blocker, and other custom modules.

Haven't used Apache since maybe 2005, so i would recommend going with Nginx either running in front of Apache or standalone.

Good luck. :+1:

[haizrul]

Wow, lot of things to do. Thanks for sharing @tmiland . I really appreciate it. 👍