Salt: service state no longer working after kernel upgrade

[drewmat]

Description of Issue/Question

service state is no longer working after kernel upgrade in Amazon Linux

Setup

Enable tomcat service:
service.running:
- name: tomcat7
- enable: True
- watch:
- /etc/tomcat7/tomcat7.conf
- /etc/tomcat7/server.xml
- /etc/tomcat7/catalina.properties
- /etc/tomcat7/context.xml
- /etc/tomcat7/logging.properties
- /etc/tomcat7/web.xml
- /etc/tomcat7/tomcat-users.xml

Steps to Reproduce Issue

run:
salt state.apply
results:

...

      ID: Enable tomcat service
Function: service.running
    Name: tomcat7
  Result: False
 Comment: State 'service.running' was not found in SLS 'tomcat'
          Reason: 'service' __virtual__ returned False: No service execution module loaded: check support for service management on Amazon Linux AMI-2018
 Changes:

...

Versions Report

[root@salt-master ~]# salt --versions-report
Salt Version:
Salt: 2018.3.0

Dependency Versions:
cffi: 1.11.5
cherrypy: 3.2.2
dateutil: 2.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Jan 31 2018, 00:17:36)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 14.5.0
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist:
locale: UTF-8
machine: x86_64
release: 4.14.26-46.32.amzn1.x86_64
system: Linux
version: Not Installed

[root@salt-master ~]# salt demo-tc grains.items
demo-tc:
----------
SSDs:
- xvda
biosreleasedate:
08/24/2006
biosversion:
4.2.amazon
cpu_flags:
- fpu
- vme
- de
- pse
- tsc
- msr
- pae
- mce
- cx8
- apic
- sep
- mtrr
- pge
- mca
- cmov
- pat
- pse36
- clflush
- mmx
- fxsr
- sse
- sse2
- ht
- syscall
- nx
- rdtscp
- lm
- constant_tsc
- rep_good
- nopl
- xtopology
- cpuid
- pni
- pclmulqdq
- ssse3
- fma
- cx16
- pcid
- sse4_1
- sse4_2
- x2apic
- movbe
- popcnt
- tsc_deadline_timer
- aes
- xsave
- avx
- f16c
- rdrand
- hypervisor
- lahf_lm
- abm
- cpuid_fault
- invpcid_single
- pti
- fsgsbase
- bmi1
- avx2
- smep
- bmi2
- erms
- invpcid
- xsaveopt
cpu_model:
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
cpuarch:
x86_64
disks:
dns:
----------
domain:
ip4_nameservers:
- 172.31.0.2
ip6_nameservers:
nameservers:
- 172.31.0.2
options:
- timeout:2
search:
- ec2.internal
- 2phase.net
sortlist:
domain:
ec2.internal
fqdn:
ip-172-31-36-244.ec2.internal
fqdn_ip4:
- 172.31.36.244
fqdn_ip6:
gid:
0
gpus:
|_
----------
model:
GD 5446
vendor:
unknown
groupname:
root
host:
ip-172-31-36-244
hwaddr_interfaces:
----------
eth0:
06:d1:34:5f:40:70
lo:
00:00:00:00:00:00
id:
demo-tc
init:
upstart
ip4_gw:
172.31.32.1
ip4_interfaces:
----------
eth0:
- 172.31.36.244
lo:
- 127.0.0.1
ip6_gw:
False
ip6_interfaces:
----------
eth0:
- fe80::4d1:34ff:fe5f:4070
lo:
- ::1
ip_gw:
True
ip_interfaces:
----------
eth0:
- 172.31.36.244
- fe80::4d1:34ff:fe5f:4070
lo:
- 127.0.0.1
- ::1
ipv4:
- 127.0.0.1
- 172.31.36.244
ipv6:
- ::1
- fe80::4d1:34ff:fe5f:4070
kernel:
Linux
kernelrelease:
4.14.33-51.34.amzn1.x86_64
kernelversion:
#1 SMP Fri Apr 13 18:18:26 UTC 2018
locale_info:
----------
defaultencoding:
UTF-8
defaultlanguage:
en_US
detectedencoding:
UTF-8
localhost:
demo-tc.2phase.net
lsb_distrib_codename:
Amazon Linux AMI 2018.03
lsb_distrib_id:
Amazon Linux AMI
lsb_distrib_release:
2018.03
machine_id:
e49ddaf535ec4fdd962883f75a57eafa
manufacturer:
Xen
master:
salt-master.2phase.net
mdadm:
mem_total:
3947
nodename:
demo-tc.2phase.net
num_cpus:
2
num_gpus:
1
os:
Amazon
os_family:
RedHat
osarch:
x86_64
oscodename:
Amazon Linux AMI 2018.03
osfinger:
Amazon Linux AMI-2018
osfullname:
Amazon Linux AMI
osmajorrelease:
2018
osrelease:
2018.03
osrelease_info:
- 2018
- 3
path:
/sbin:/usr/sbin:/bin:/usr/bin
pid:
2792
productname:
HVM domU
ps:
ps -efHww
pythonexecutable:
/usr/bin/python2.7
pythonpath:
- /usr/bin
- /usr/lib/python27.zip
- /usr/lib64/python2.7
- /usr/lib64/python2.7/plat-linux2
- /usr/lib64/python2.7/lib-tk
- /usr/lib64/python2.7/lib-old
- /usr/lib64/python2.7/lib-dynload
- /usr/local/lib64/python2.7/site-packages
- /usr/local/lib/python2.7/site-packages
- /usr/lib64/python2.7/site-packages
- /usr/lib/python2.7/site-packages
- /usr/lib64/python2.7/dist-packages
- /usr/lib64/python2.7/dist-packages/PIL
- /usr/lib/python2.7/dist-packages
pythonversion:
- 2
- 7
- 13
- final
- 0
saltpath:
/usr/lib/python2.7/dist-packages/salt
saltversion:
2018.3.0
saltversioninfo:
- 2018
- 3
- 0
- 0
selinux:
----------
enabled:
False
enforced:
Disabled
serialnumber:
ec2e8b41-fb5a-e703-a1f2-5bd7c0dd1881
server_id:
1030780186
shell:
/bin/sh
swap_total:
0
uid:
0
username:
root
uuid:
ec2e8b41-fb5a-e703-a1f2-5bd7c0dd1881
virtual:
xen
virtual_subtype:
Xen PV DomU
zfs_support:
False
zmqversion:
4.0.5

Any help would be appreciated. Thanks!

[pdarvasi]

+1 this is blocker for us, as we start our process with a "yum update", we cannot use Salt to provision Amazon images

[gtmanfred]

Yup, that is a bug.

Set this in your /etc/salt/minion to work around this issue.

providers:
  service: rh_service

[gtmanfred]

I have fixed this and it will be in 2018.3.1 and 2017.7.6

47271

[drewmat]

I applied the workaround and it is working again. Thanks!

[rallytime]

Closed via #47271

[aphor]

I think the service module should move towards deprecating sysv and upstart services, and start checking for systemd in __virtual__.

[gtmanfred]

The service state module uses whichever virtual module is used for your system, if it is booted with systemd, it will load systemd as service for the execution module.
Then the service state just uses that to handle the services.

There is no need to deprecate anything.

[seeafish]

@gtmanfred I still need use the workaround (i.e. providers: rh_service)in both 2017.7.6 and 2018.3.1 for some reason.

Firstly, here's the installation log entry:
* INFO: Command line: '/tmp/kitchen/bootstrap.sh -P git v2018.3.1'

And here's the error when the workaround isn't in place:
Reason: 'service' __virtual__ returned False: No service execution module loaded: check support for service management on CentOS Linux-7

This is in the log when running it without the workaround, before it errors out:

[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.utils.boto.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto3. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.utils.boto3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3', please fix this.
       [ERROR   ] State 'service.running' was not found in SLS 'salt.minion'
       Reason: 'service' __virtual__ returned False: No service execution module loaded: check support for service management on CentOS Linux-7

       [ERROR   ] State 'service.running' was not found in SLS 'openssh'
       Reason: 'service' __virtual__ returned False: No service execution module loaded: check support for service management on CentOS Linux-7

       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto3_elasticache. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto3_elasticache.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3_elasticache', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto3_route53. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto3_route53.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3_route53', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto3_sns. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto3_sns.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3_sns', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_apigateway. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_apigateway.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_apigateway', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_asg. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_asg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_asg', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cfn. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_cfn.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cfn', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cloudfront. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_cloudfront.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cloudfront', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cloudtrail. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_cloudtrail.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cloudtrail', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cloudwatch. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_cloudwatch.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cloudwatch', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cloudwatch_event. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_cloudwatch_event.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cloudwatch_event', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cognitoidentity. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_cognitoidentity.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cognitoidentity', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_dynamodb. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_dynamodb.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_dynamodb', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_ec2. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_ec2.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_ec2', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elasticache. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_elasticache.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elasticache', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elasticsearch_domain. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_elasticsearch_domain.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elasticsearch_domain', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elb. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_elb.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elb', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elbv2. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_elbv2.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elbv2', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_iam. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_iam.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_iam', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_iot. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_iot.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_iot', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_kinesis. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_kinesis.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_kinesis', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_kms. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_kms.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_kms', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_lambda. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_lambda.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_lambda', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_rds. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_rds.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_rds', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_route53. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_route53.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_route53', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_s3. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_s3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_s3', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_s3_bucket. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_s3_bucket.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_s3_bucket', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_secgroup. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_secgroup.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_secgroup', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_sns. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_sns.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_sns', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_sqs. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.module.boto_sqs.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_sqs', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto3. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.utils.boto3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3', please fix this.
       [ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto. Module will not be loaded: 'module' object has no attribute '__version__'
       [WARNING ] salt.loaded.int.utils.boto.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto', please fix this.
       [WARNING ] org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
       [WARNING ] org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
       'service' __virtual__ returned False: Your OS is on the disabled list

Any idea what's going on?

Edits: Updated to be clearer.

[gtmanfred]

What does your --versions-report look like?

[seeafish]

Salt Version:
           Salt: 2018.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.28.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Apr 11 2018, 07:36:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 4.9.87-linuxkit-aufs
         system: Linux
        version: CentOS Linux 7.5.1804 Core

For reference, running this in kitchen-docker at the moment. Starting to think it's just a docker image thing. I know most of these images shouldn't really be used as proper OS test beds, so happy to concede user error on this one.

[gtmanfred]

That says you are still using 2018.3.1.

yeah if you are in docker, and don't set your exec command to an init system, then you cannot use the service module.

Check out the .kitchen.yml for saltstack/salt, we have each docker setup to handle starting with systemd or upstart.

[aphor]

There might be a good reason to add a note about Docker and containers with
and without init/service subsystem to the docs.

On Fri, Jun 29, 2018, 08:51 Daniel Wallace [e-mail] wrote:

That says you are still using 2018.3.1.

yeah if you are in docker, and don't set your exec command to an init
system, then you cannot use the service module.

Check out the .kitchen.yml for saltstack/salt, we have each docker setup
to handle starting with systemd or upstart.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/47258#issuecomment-401360475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOKu7C7d7MVNnSQ5WZxz-1uHYIf8jy1ks5uBjD1gaJpZM4Tgvn1
.