Awx: Mail Module is not working in AWX but working in ansible CLI

Created on 29 Nov 2018 · 15Comments · Source: ansible/awx

ISSUE TYPE

Bug Report - Mail Module from AWX

COMPONENT NAME

Mail Module is not working when the playbook is triggered from AWX

SUMMARY

Kindly help us if someone resolved this issue.

Playbook:

cat helloworld.yml

hosts: 127.0.0.1
tasks:
- name: Print the Message
  shell: echo "Welcome to Ansible AWX"
  register: printout
- debug:
  msg: "Printed the message"
- mail:
  secure: never
  subject: "Its a Hello World Test email Subject"
  body: |
  Hi
```
Its a Hello World Test email.
```
  to: [email protected]
  charset: utf8

Error:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket.error: [Errno 111] Connection refused
fatal: [127.0.0.1]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/root/.ansible/tmp/ansible-tmp-1543410686.39-84204182712689/AnsiballZ_mail.py\", line 113, in \n _ansiballz_main()\n File \"/root/.ansible/tmp/ansible-tmp-1543410686.39-84204182712689/AnsiballZ_mail.py\", line 105, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/root/.ansible/tmp/ansible-tmp-1543410686.39-84204182712689/AnsiballZ_mail.py\", line 48, in invoke_module\n imp.load_module('__main__', mod, module, MOD_DESC)\n File \"/tmp/ansible_mail_payload_TVBYW8/__main__.py\", line 374, in \n File \"/tmp/ansible_mail_payload_TVBYW8/__main__.py\", line 263, in main\n File \"/usr/lib64/python2.7/smtplib.py\", line 315, in connect\n self.sock = self._get_socket(host, port, se…

The full traceback is:
Traceback (most recent call last):
File "/root/.ansible/tmp/ansible-tmp-1543485386.63-176255315913210/AnsiballZ_mail.py", line 113, in
_ansiballz_main()
File "/root/.ansible/tmp/ansible-tmp-1543485386.63-176255315913210/AnsiballZ_mail.py", line 105, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "/root/.ansible/tmp/ansible-tmp-1543485386.63-176255315913210/AnsiballZ_mail.py", line 48, in invoke_module
imp.load_module('__main__', mod, module, MOD_DESC)
File "/tmp/ansible_mail_payload_adEnWK/__main__.py", line 374, in
File "/tmp/ansible_mail_payload_adEnWK/__main__.py", line 263, in main
File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_so

ENVIRONMENT

AWX version: 2.1.0
AWX install method: docker
Ansible version: 2.7.1
Operating System: Redhat
Web Browser: chrome

STEPS TO REPRODUCE

Execute the playbook again from AWX

EXPECTED RESULTS

Mail has to be sent

ACTUAL RESULTS

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket.error: [Errno 111] Connection refused
fatal: [127.0.0.1]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/root/.ansible/tmp/ansible-tmp-1543410686.39-84204182712689/AnsiballZ_mail.py\", line 113, in \n _ansiballz_main()\n File \"/root/.ansible/tmp/ansible-tmp-1543410686.39-84204182712689/AnsiballZ_mail.py\", line 105, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/root/.ansible/tmp/ansible-tmp-1543410686.39-84204182712689/AnsiballZ_mail.py\", line 48, in invoke_module\n imp.load_module('__main__', mod, module, MOD_DESC)\n File \"/tmp/ansible_mail_payload_TVBYW8/__main__.py\", line 374, in \n File \"/tmp/ansible_mail_payload_TVBYW8/__main__.py\", line 263, in main\n File \"/usr/lib64/python2.7/smtplib.py\", line 315, in connect\n self.sock = self._get_socket(host, port, se…

ADDITIONAL INFORMATION

bug

Source

ChengalReddy

Most helpful comment

Hey @ryanpetrello,

The mail server is implied as localhost when none is specified. I guess the container can't send mail messages from its host. Makes sense.

I added the host: attribute to the module and specified an external SMTP relay, thus solving the issue.

RylandDeGregory on 23 Jun 2020

🎉1 👍1

All 15 comments

@ChengalReddy can you show the output of this command docker exec -it <AWX CONTAINER ID> rpm -q ansible you might be hitting this issue https://github.com/ansible/ansible/pull/49016

ryan012880 on 29 Nov 2018

Here is the output of docker exec -it c2f045e20a2b rpm -q ansible
ansible-2.7.1-1.el7.ans.noarch

ChengalReddy on 29 Nov 2018

@ChengalReddy ok that could be different i was hitting https://github.com/ansible/ansible/pull/49016 so downgrading ansible from 2.7.2 to ansible-2.7.1-1.el7.ans.noarch did a fixed for me, can you specify a host: <not localhost> parameter in the mail module on your playbook see if that would help. If you do not specify any host parameter default will be localhost and that will try to connect to the loopback interface of the container not the loopback interface of the docker host.

ryan012880 on 29 Nov 2018

@ryan012880 - I have performed the steps but still the issue is same. The problem here is it works when I execute the playbook from CLI but not from AWX. Kindly help

sudo docker exec -it c2f045e20a2b rpm -q ansible
ansible-2.7.1-1.el7.ans.noarch
ansible --version

ansible 2.7.1
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/test/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, May 31 2018, 09:41:32) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

[test@temphost ~]$ cat /var/lib/awx/projects/HelloWorld/helloworld.yml

hosts: 127.0.0.1
tasks:
- name: Print the Message
  shell: echo "Welcome to Ansible AWX"
  register: printout
- debug:
  msg: "Printed the message"
- mail:
  secure: never
  subject: "Its a Hello World Test email Subject"
  host: temphost.abc.com
  body: |
  Hi Team,
```
Its a Hello World Test email.
```
  to: [email protected]
  charset: utf8

ChengalReddy on 29 Nov 2018

check if temphost.abc.com can be resolve from the container if yes check if temphost.abc.com is accepting external connection on port 25, if you have a mail relay server in your network you can probably try using that server.

What's the output of this command ss -ltnp | grep :25 from temphost.abc.com ?

ryan012880 on 29 Nov 2018

Hi
How to check the if the external connection is accepting using port 25 in container. Below is the output of
ss -ltnp | grep :25
LISTEN 0 100 127.0.0.1:25 :
LISTEN 0 100 ::1:25 :::*

ChengalReddy on 29 Nov 2018

Hi
How to check the if the external connection is accepting using port 25 in container. Below is the output of
ss -ltnp | grep :25
LISTEN 0 100 127.0.0.1:25 _:_
LISTEN 0 100 ::1:25 :::*

so it seems that temphost.abc.com port 25 is only listening on 127.0.0.1, you can either reconfigure your mail service on temphost.abc.com to bind on 0.0.0.0 which will make it accept connections from other network segments or like I've said use a mail relay server in your network if you have one as the value for host parameter. tnx

ryan012880 on 29 Nov 2018

Hi
I have smtp server for mailing. Can I know where can i set this parameter to allow this connections

ChengalReddy on 29 Nov 2018

Can some one assist here?

ChengalReddy on 30 Nov 2018

This was solved after adding the host: .

ChengalReddy on 4 Dec 2018

I have the same issue. It works from command line but doesn't work from AWX.
What's the fix?

click2chandu on 12 Jul 2019

Mail notification is not working in AWX 11.2.0 but same configuration is working in AWX 6.0.0

ERROR: Email Notification: Notification failed.
[Errno 111] Connection refused

ansible 2.9.9 config file = None configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.6/site-packages/ansible executable location = /usr/local/bin/ansible python version = 3.6.8 (default, Aug 10 2019, 06:54:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

balaramr on 8 Jun 2020

This is also an issue in AWX 12.0.0.

It doesn't look like anyone from the AWX team has ever commented, so I hope we can get it looked at (since it seems to have been an issue since 2018). @ryanpetrello any info you can provide here?

My test playbook:

---
- hosts: localhost
  connection: local
  tasks:
  - name: send email with the report attached
    mail:
      subject: "Report of ClamAV compliance"
      body: "Status on {{ ansible_date_time.date }}"
      attach: test.csv
      from: "ansible@tower.<company>.com"
      to: "ryland_degregory@<company>.com"
    run_once: true

The error I'm receiving:

<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823 && echo ansible-tmp-1592837425.263759-44212-196204416170823="` echo /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823 `" ) && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/notification/mail.py
<127.0.0.1> PUT /var/lib/awx/.ansible/tmp/ansible-local-44150pnj1ha3b/tmp4862218c TO /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/ /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3.6 /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py", line 102, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.notification.mail', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib64/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_mail_payload_prk6xmed/ansible_mail_payload.zip/ansible/modules/notification/mail.py", line 398, in <module>
  File "/tmp/ansible_mail_payload_prk6xmed/ansible_mail_payload.zip/ansible/modules/notification/mail.py", line 283, in main
  File "/usr/lib64/python3.6/smtplib.py", line 251, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib64/python3.6/smtplib.py", line 336, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib64/python3.6/smtplib.py", line 307, in _get_socket
    self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1592837425.263759-44212-196204416170823/AnsiballZ_mail.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.notification.mail', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_mail_payload_prk6xmed/ansible_mail_payload.zip/ansible/modules/notification/mail.py\", line 398, in <module>\n  File \"/tmp/ansible_mail_payload_prk6xmed/ansible_mail_payload.zip/ansible/modules/notification/mail.py\", line 283, in main\n  File \"/usr/lib64/python3.6/smtplib.py\", line 251, in __init__\n    (code, msg) = self.connect(host, port)\n  File \"/usr/lib64/python3.6/smtplib.py\", line 336, in connect\n    self.sock = self._get_socket(host, port, self.timeout)\n  File \"/usr/lib64/python3.6/smtplib.py\", line 307, in _get_socket\n    self.source_address)\n  File \"/usr/lib64/python3.6/socket.py\", line 724, in create_connection\n    raise err\n  File \"/usr/lib64/python3.6/socket.py\", line 713, in create_connection\n    sock.connect(sa)\nOSError: [Errno 101] Network is unreachable\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

RylandDeGregory on 22 Jun 2020

@RylandDeGregory,

I don't have a readily available SMTP server to test with, but this exception in your error message seems pretty clear to me:

OSError: [Errno 101] Network is unreachable

Looking down that Python stack trace, this looks like pretty distinct Python socket code, and the way I interpret this is that your container can't reach your mail host:

 File "/usr/lib64/python3.6/smtplib.py", line 251, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib64/python3.6/smtplib.py", line 336, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib64/python3.6/smtplib.py", line 307, in _get_socket
    self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable