Status-react: Constant "Connecting to peers" on eth.beta fleet

[churik]

Description

Type: Bug

Summary: constant "Connecting to peers" on both mobile and desktop while connecting to eth.beta fleet.
Switching to eth.stage fleet helps to fix this.

Expected behavior

Can send and receive messages

Actual behavior

"Connecting to peers", messaging is not working

Reproduction

• Open Status
• Log in
• Join #status

Additional Information

• Status version: all
• Operating System: all

Logs

geth.log

cc @adambabik @jakubgs

[oskarth]

Duplicate: https://github.com/status-im/status-react/issues/6701 - closing that one, thanks @churik

[oskarth]

Also cc @mandrigin @PombeirP @dshulyak - can we revert whatever deploy we did with the eth.beta fleet so people can chat with Desktop without further action?

Once it is up and running, we should run a post-mortem to understand how this happened and how we can avoid it.

[jakubgs]

It's not a deployment, docker service has issues on eth.beta fleet. Working on it

[oskarth]

Beta up for me now. Thanks J

[churik]

Up for me.

[jakubgs]

This appears to be due to Ansible generating a faulty /etc/docker/daemon.json file which was not valid JSON.

Template: https://github.com/status-im/infra-role-bootstrap/blob/master/templates/daemon.json.j2

The resulting file looked like this:

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-facility": "local6",
    "tag": "{{.DaemonName}}/{{.Name}}{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-facility": "local6",
    "tag": "{{.DaemonName}}/{{.Name}}",
    "labels": "do-ams3",
    "env": "eth,test"
  }
}

Which is clearly wrong.

Page not found · GitHub

[oskarth]

When or how did that file get generated?

[jakubgs]

I was running the infra-role-bootstrap on the eth.beta fleet to add missing users. It hasn't been run there in quite a while and it was missing quite a lot of users. The issue was caused when the JSON file was generated incorrectly by that role and then docker service restarted.

[jakubgs]

Added JSON validation but I need to also add some more sanity check before docker service is restarted:
https://github.com/status-im/infra-role-bootstrap/commit/fbffde55e7f39d2b14f9159497e903247afa378b

[jakubgs]

And here is a proper for for the templating issue:
https://github.com/status-im/infra-role-bootstrap/commit/6b259eafc18aba87df178b0384c188eb182b9e5c

[jakubgs]

My guess is that something in the newest(2.7.0) Ansible version changed about how the {% raw %} and {% endraw %} escape mechanism works and that fucked up the template and the resulting /etc/docker/daemon.json.

Or it might be a bug in Ansible, because as far as I can tell this should still work:
https://docs.ansible.com/ansible/2.7/user_guide/playbooks_advanced_syntax.html

Advanced Syntax — Ansible Documentation

[jakubgs]

Ok, I think I've made it safer. Now the docker service is reloaded rather then restarted, so if the config is bad it just keeps going with the old config. This config should not cause issues in the future