Salt: [BUG] [report v2019.2.3, v3000] 25% memory usage on Raspberry Pi (Ubuntu 64bit)

Created on 20 Feb 2020  路  23Comments  路  Source: saltstack/salt

Description of Issue

I am running the latest release v2019.2.3 on a Raspberry Pi 3 B and it uses nearly 25% of the memory (250MB of 1GB). There are no running task - the minion is "idle".

Why is so much memory needed?

Setup

Saltstack minion on a Raspberry Pi 3 B V1.2

Versions Report

Salt Version:
Salt: 2019.2.3

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.31.0
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.17 (default, Nov 7 2019, 10:07:09)
python-gnupg: Not Installed
PyYAML: 5.1.2
PyZMQ: 17.1.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 5.1.1
ZMQ: 4.3.2

System Versions:
dist: Ubuntu 19.10 eoan
locale: UTF-8
machine: aarch64
release: 5.3.0-1007-raspi2
system: Linux
version: Ubuntu 19.10 eoan

Bug Tests cannot-reproduce team-core
Source
picture sjkummer

All 23 comments

salt-minion-memory-usage-clipped

sjkummer picture sjkummer on 20 Feb 2020

That is a surprising amount of memory. You might see if you can update the number of worker processes to just 1.

I've used minions and masters on my pi before - the only real problem that I saw was that occasionally it would get into a runaway loop that would start eating all my memory, but that's a different issue.

waynew picture waynew on 24 Feb 2020

Hello Wayne, thanks for your hints!

I have tried to disable multithreading as follows, but now effect. I still got 3 minion processes. Maybe, it's a bug in the current release 馃き

multiprocessing: False

process_count_max: 1

reactor_worker_threads: 1

This runaway loop sound bad 馃槙did you find a solution for this? ulimit?

sjkummer picture sjkummer on 25 Feb 2020

Upgrading to version 3000 did not make a difference.
The multithreading settings are reset after ever service restart, see https://github.com/saltstack/salt/issues/56245~~

sjkummer picture sjkummer on 25 Feb 2020

Try disabling some execution modules and grains using the disable_modules and disable_grains minion settings respectively.

You can find the list of loaded modules using the following command:

salt RASPBERRYPI sys.list_modules

Then add the modules you don't need to the minion config:

disable_modules:
  - consul
  - glassfish
  - grafana4
  - jboss7
  - ...
  - vsphere

As for grains, you can find the full list of grains modules here, and also disable the ones you don't need:

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs

This can also considerably speed up the minion response time: https://github.com/saltstack/salt/issues/48773#issuecomment-443599880

max-arnold picture max-arnold on 25 Feb 2020

Thanks for your further hints!

I tried to disable as many modules as possible but it did not make a difference ... now the three salt-minion processes use 27% of the memory 馃檲

My config is as follows now:

master: xxx

multiprocessing: False
process_count_max: 1
reactor_worker_threads: 1

disable_modules:
    - ansible
    - artifactory
    - bigip
    - btrfs
    - buildout
    - composer
    - consul
    - cpan
    - cryptdev
    - data
    - defaults
    - devmap
    - dig
    - django
    - dnsmasq
    - dnsutil
    - drbd
    - event
    - extfs
    - gem
    - genesis
    - git
    - glassfish
    - google_chat
    - grafana4
    - grains
    - hashutil
    - highstate_doc
    - hipchat
    - hosts
    - http
    - incron
    - ini
    - inspector
    - introspect
    - iosconfig
    - ip
    - iptables
    - jboss7
    - jboss7_cli
    - k8s
    - kernelpkg
    - key
    - keyboard
    - kmod
    - locale
    - locate
    - log
    - logrotate
    - lowpkg
    - lvm
    - mandrill
    - match
    - mattermost
    - mine
    - minion
    - modjk
    - mount
    - msteams
    - nagios
    - nagios_rpc
    - namecheap_domains
    - namecheap_domains_dns
    - namecheap_domains_ns
    - namecheap_ssl
    - namecheap_users
    - network
    - nexus
    - nginx
    - nova
    - npm
    - nspawn
    - nxos_api
    - openscap
    - openstack_config
    - opsgenie
    - out
    - pagerduty
    - pagerduty_util
    - pam
    - parallels
    - partition
    - peeringdb
    - pillar
    - pip
    - pkg
    - pkg_resource
    - publish
    - pushover
    - pyenv
    - raid
    - random
    - random_org
    - rbenv
    - rest_sample_utils
    - ret
    - rsync
    - rvm
    - s3
    - s6
    - salt_proxy
    - saltcheck
    - saltutil
    - schedule
    - scsi
    - sdb
    - seed
    - serverdensity_device
    - service
    - shadow
    - slack
    - slsutil
    - smbios
    - smtp
    - solrcloud
    - sqlite3
    - ssh
    - state
    - status
    - statuspage
    - supervisord
    - sys
    - sysctl
    - sysfs
    - syslog_ng
    - system
    - telegram
    - telemetry
    - temp
    - test
    - timezone
    - tls
    - travisci
    - udev
    - uptime
    - user
    - vault
    - vbox_guest
    - virtualenv
    - vsphere
    - x509
    - xfs
    - zabbix
    - zenoss

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs
sjkummer picture sjkummer on 25 Feb 2020

Sorry, small offtop: is progress bar on your screen broken (for top)?

Why do we see 60.9/908.2 MB?

4815162342lost picture 4815162342lost on 25 Feb 2020

@4815162342lost
Sorry, I just toggled the summary display mode (m) ... making the "used" percentage visible
top

sjkummer picture sjkummer on 25 Feb 2020

@sjkummer You disabled too many modules. You'll probably want to run things like state.apply, test.ping, or grains.get. This means you need to enable grains, pillar, pkg, saltutil, service, state, sys, system, test, user and probably other modules. Check the minion log for any errors to see what else is required.

As for 27%, I'm quite surprised. The memory usage should go down. I'll try it on my Pi in a few days.

max-arnold picture max-arnold on 25 Feb 2020

Btw. I also switched from python 2.7 to python 3 ... but no change

sjkummer picture sjkummer on 25 Feb 2020

Ok, it looks like I actually have a PI I can access, and below are my results (Raspbian Buster Lite, Salt 3000 Py3):

Default install

Measurements right after the sudo salt raspberrypi sys.list_modules command was finished.

$ sudo top -p $(pgrep salt-minion | paste -s -d, )
MiB Mem :    926.1 total,    422.4 free,     99.5 used,    404.1 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.    760.2 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 5082 root      20   0   38640  23792  10724 S   0.0   2.5   0:01.33 salt-minion
 5087 root      20   0  100420  38156  11640 S   0.0   4.0   0:22.09 salt-minion
 5089 root      20   0   49148  17872   3836 S   0.0   1.9   0:00.00 salt-minion

With disabled modules/grains

/etc/salt/minion.d/disable.conf

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs

disable_modules:
    - aliases
    - alternatives
    - ansible
    - artifactory
    - bigip
    - btrfs
    - buildout
    - chroot
    - cloud
    - composer
    - consul
    - container_resource
    - cpan
    - cryptdev
    - data
    - dig
    - django
    - dnsmasq
    - drbd
    - etcd
    - ethtool
    - extfs
    - freezer
    - gem
    - genesis
    - git
    - glassfish
    - google_chat
    - grafana4
    - highstate_doc
    - hipchat
    - incron
    - ini
    - inspector
    - introspect
    - iosconfig
    - iptables
    - jboss7
    - jboss7_cli
    - k8s
    - key
    - keyboard
    - kmod
    - locate
    - logrotate
    - lvm
    - mandrill
    - mattermost
    - modjk
    - msteams
    - nagios
    - nagios_rpc
    - namecheap_domains
    - namecheap_domains_dns
    - namecheap_domains_ns
    - namecheap_ssl
    - namecheap_users
    - nexus
    - nfs3
    - nginx
    - nova
    - npm
    - nspawn
    - nxos_api
    - openscap
    - openstack_config
    - opsgenie
    - pagerduty
    - pagerduty_util
    - parallels
    - peeringdb
    - publish
    - pushover
    - pyenv
    - raid
    - random
    - random_org
    - rbenv
    - rest_sample_utils
    - rsync
    - rvm
    - s3
    - s6
    - salt_proxy
    - saltcheck
    - scsi
    - sdb
    - seed
    - serverdensity_device
    - slack
    - smbios
    - smtp
    - solrcloud
    - sqlite3
    - ssh
    - statuspage
    - supervisord
    - syslog_ng
    - telegram
    - telemetry
    - tls
    - travisci
    - vault
    - vbox_guest
    - virtualenv
    - vsphere
    - x509
    - xml
    - xfs
    - zabbix
    - zenoss

Measurements right after the sudo salt raspberrypi sys.list_modules command was finished.

$ sudo top -p $(pgrep salt-minion | paste -s -d, )
MiB Mem :    926.1 total,    424.9 free,     96.9 used,    404.2 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.    762.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 5303 root      20   0   97268  35300  11604 S   0.3   3.7   0:03.46 salt-minion
 5298 root      20   0   38640  23880  10812 S   0.0   2.5   0:01.31 salt-minion
 5305 root      20   0   49148  17900   3840 S   0.0   1.9   0:00.00 salt-minion
 ```

So, the RES memory usage is a bit lower:

In [1]: (23792 + 38156 + 17872) - (35300 + 23880 + 17900)
Out[1]: 2740
```

And the minion response time is significantly reduced.

What I don't understand is why the RES memory consumption is much higher on your screenshots. Maybe Ubuntu 19.10 is less optimized for PI than Raspbian Buster?

max-arnold picture max-arnold on 26 Feb 2020

Hey @max-arnold thanks for your reproduction! That is really interesting. I now disabled exactly the same list, but I still got the same high memory usage.

This really could be related to the Ubuntu 64bit environment.

I also had to install salt using git sudo sh bootstrap-salt.sh -x python3 git v3000 (the python version does not make a difference).

Just one thing: I double checked the modules using sys.list_modules and even tough most modules are correctly disabled, some specific like ansible are still listet. I set that list in both /etc/salt/minion.d/disable.conf and /etc/salt/minion ... is there something wrong with my setup?

sjkummer picture sjkummer on 26 Feb 2020

Just one thing: I double checked the modules using sys.list_modules and even tough most modules are correctly disabled, some specific like ansible are still listet. I set that list in both /etc/salt/minion.d/disable.conf and /etc/salt/minion ... is there something wrong with my setup?

I see the same thing. It happens because sys.list_modules lists virtual module names, and disable_modules expects file names (without the .py extension). So, to really disable the ansible module you need to specify ansiblegate.

/etc/salt/minion.d/disable.conf

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs

disable_modules:
    - aliases
    - alternatives
    - ansiblegate
    - artifactory
    - bigip
    - btrfs
    - zcbuildout
    - chroot
    - cloud
    - composer
    - consul
    - container_resource
    - cpan
    - cryptdev
    - data
    - dig
    - djangomod
    - dnsmasq
    - drbd
    - etcd_mod
    - ethtool
    - extfs
    - freezer
    - gem
    - genesis
    - git
    - glassfish
    - google_chat
    - grafana4
    - highstate_doc
    - hipchat
    - incron
    - ini_manage
    - inspector
    - introspect
    - iosconfig
    - iptables
    - iwtools
    - jboss7
    - jboss7_cli
    - k8s
    - key
    - keyboard
    - kmod
    - locate
    - logrotate
    - lvm
    - mandrill
    - mattermost
    - modjk
    - msteams
    - nagios
    - nagios_rpc
    - namecheap_domains
    - namecheap_domains_dns
    - namecheap_domains_ns
    - namecheap_ssl
    - namecheap_users
    - nexus
    - nfs3
    - nginx
    - nova
    - npm
    - nspawn
    - nxos_api
    - openscap
    - openstack_config
    - opsgenie
    - pagerduty
    - pagerduty_util
    - parallels
    - peeringdb
    - publish
    - pushover_notify
    - pyenv
    - raid
    - random
    - random_org
    - rbenv
    - rest_sample_utils
    - rsync
    - rvm
    - s3
    - s6
    - salt_proxy
    - saltcheck
    - scsi
    - sdb
    - seed
    - serverdensity_device
    - slack_notify
    - smbios
    - smtp
    - solrcloud
    - sqlite3
    - ssh
    - statuspage
    - supervisord
    - syslog_ng
    - telegram
    - telemetry
    - tls
    - travisci
    - vault
    - vbox_guest
    - virtualenv_mod
    - vsphere
    - x509
    - xml
    - xfs
    - zabbix
    - zenoss

max-arnold picture max-arnold on 26 Feb 2020
👍1

And I guess you are right - the increased memory consumption is caused by 64 bit arch.

salt-call --versions-report

Salt Version:
           Salt: 3000

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10.3
         locale: UTF-8
        machine: armv7l
        release: 4.19.97-v7+
         system: Linux
        version: debian 10.3

max-arnold picture max-arnold on 26 Feb 2020
👍1

@max-arnold so you're not seeing the same amount of memory usage?

waynew picture waynew on 17 Mar 2020

@waynew Yes, that is correct. Earlier I posted all the stats for a 32bit architecture.

max-arnold picture max-arnold on 17 Mar 2020

@sjkummer Running on a pi2 with raspbian 10 only used about 50M with the default 3000 install. It does seem possible that the 64-bit nature is what's causing increased memory consumption. Unfortunately I don't have access to a 64-bit pi right now.

waynew picture waynew on 24 Mar 2020

trying to move this forward, I will need to follow up tomorrow or Monday.

sagetherage picture sagetherage on 19 Jun 2020

@sjkummer or @max-arnold are either of you able to open a PR for this? This likely isn't something the Open Core Team can focus on right now, but we can keep it open for the community to pick up. I will leave this as-is and follow up again the next business day.

sagetherage picture sagetherage on 19 Jun 2020

I have no idea how to reduce memory usage (other than switching to a 32-bit OS), so I can't provide a PR

max-arnold picture max-arnold on 23 Jun 2020

Neither do I, sorry.

Update
When using the latest salt version v3001 on Ubuntu 20.04 LTS (arm64), the memory usage looks much better. It's about half as much as before (~100MB)

sjkummer picture sjkummer on 23 Jun 2020

putting this into Blocked for now, but have a reminder set for myself to come back to this ticket this week.

sagetherage picture sagetherage on 27 Jul 2020

this needs a 64-bit RaspberryPi to test, I will attempt to assign this out to get it unblocked.

sagetherage picture sagetherage on 28 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

allyunion picture allyunion  路  3Comments
mooperd picture mooperd  路  3Comments
zieba88 picture zieba88  路  3Comments
Arguros picture Arguros  路  3Comments
layer3switch picture layer3switch  路  3Comments