Salt: Firewalld state cause horrid performance during saltrun.

[hunkeelin]

Some of my formula will have firewalld rules in it like the following:

dokuwiki_fw:
    firewalld.service:
        - name: dokuwiki
        - ports:
            - 8002/tcp

dokuwiki_fw_rule:
    firewalld.present:
        - name: public
        - prune_services: False
        - services:
            - dokuwiki

Some subset of my nodes will have serveral firewalld states to run during an highstate run. Upon inspecting debug I realize the firewalld module is consuming a lot of time. It is inefficient.

E.G

consider the following formula test

doku_default_pw:
    file.managed:
        - name: /tmp/test
        - source: salt://test/testfile

test_fw:
    firewalld.service:
        - name: test
        - ports:
            - 8002/tcp

dokuwiki_fw_rule:
    firewalld.present:
        - name: public
        - prune_services: False
        - services:
            - test

When I run salt-call state.apply test on the minion it takes an average of 10s for a no change occurrence.

local:

Summary for local
------------
Succeeded: 9
Failed:    0
------------
Total states run:     9
Total run time:  12.718 s

When I commented out the firewall lines. it takes significantly less time to run the state.

local:

Summary for local
------------
Succeeded: 7
Failed:    0
------------
Total states run:     7
Total run time:   1.577 s

[garethgreenaway]

Thanks for the report. Sounds like the firewalld module(s) could benefit from the addition of mod_aggregate support.

[nhavens]

@garethgreenaway, thanks for the mod_aggregate suggestion. I didn't know about that, but it could be really helpful in cases in which we need to run multiple firewall changes all at once.

Unfortunately, I don't think mod_aggregate will fix the slowness in the firewalld state module. Here's an example sls file with one invocation of the firewalld.present state function:

add firewall rule to test firewalld performance:
  firewalld.present:
    - name: public
    - ports:
      - 8080/tcp

Here is the output of a state.apply that made a change as well as 2 that did not (altered to hide username, hostname, and domain). Note that it took > 3 seconds to make a simple firewall rule change and > 2.3 seconds to confirm that the machine was already in the desired state:

user@master /srv/salt/test $ sudo salt 'testminion.example.org' state.apply test.fw
testminion.example.org:
----------
          ID: add firewall rule to test firewalld performance
    Function: firewalld.present
        Name: public
      Result: True
     Comment: 'public' was configured.
     Started: 12:24:29.134325
    Duration: 3217.489 ms
     Changes:   
              ----------
              ports:
                  ----------
                  new:
                      - 8080/tcp
                  old:

Summary for testminion.example.org
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   3.217 s
user@master /srv/salt/test $ sudo salt 'testminion.example.org' state.apply test.fw
[sudo] password for user: 
testminion.example.org:
  Name: public - Function: firewalld.present - Result: Clean Started: - 12:44:31.981771 Duration: 2354.323 ms

Summary for testminion.example.org
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   2.354 s
user@master /srv/salt/test $ sudo salt 'testminion.example.org' state.apply test.fw
testminion.example.org:
  Name: public - Function: firewalld.present - Result: Clean Started: - 12:44:37.940333 Duration: 2334.168 ms

Summary for testminion.example.org
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1 
Total run time:   2.334 s
user@master /srv/salt/test $ 

It looks like the firewalld.present state function makes many unnecessary calls to the firewalld execution module, which in turn runs the firewall-cmd command. Here the snip from the minion log file for the firewalld.present in the simple example above:

2018-05-09 13:13:52,076 [salt.state       :1795][INFO    ][30189] Executing state firewalld.present for [public]
2018-05-09 13:13:52,080 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --get-zones --permanent' in directory '/root'
2018-05-09 13:13:52,313 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --get-icmptypes --permanent' in directory '/root'
2018-05-09 13:13:52,532 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-icmp-blocks --permanent' in directory '/root'
2018-05-09 13:13:52,765 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-all --permanent' in directory '/root'
2018-05-09 13:13:52,997 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-ports --permanent' in directory '/root'
2018-05-09 13:13:53,242 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --add-port=8080/tcp --permanent' in directory '/root'
2018-05-09 13:13:53,478 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-forward-ports --permanent' in directory '/root'
2018-05-09 13:13:53,709 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-services --permanent' in directory '/root'
2018-05-09 13:13:53,939 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-interfaces --permanent' in directory '/root'
2018-05-09 13:13:54,184 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-sources --permanent' in directory '/root'
2018-05-09 13:13:54,416 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --zone=public --list-rich-rules --permanent' in directory '/root'
2018-05-09 13:13:54,658 [salt.loaded.int.module.cmdmod:385 ][INFO    ][30189] Executing command '/usr/bin/firewall-cmd --reload' in directory '/root'