Alertmanager: Alertmanager Web GUI issue

[fchiorascu]

What did you do?
Upgrade alertmanager to 0.14 version.

What did you expect to see?
The Web GUI page loading 100% like below.

What did you see instead? Under which circumstances?
The below page.

Environment

• System information:
  $ uname -srm
  Linux 3.10.0-514.6.2.el7.x86_64 x86_64

$ cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

• Alertmanager version:
  $ /opt/alertmanager/alertmanager --version
  alertmanager, version 0.14.0 (branch: HEAD, revision: 30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)
  build user: root@37b6a49ebba9
  build date: 20180213-08:16:42
  go version: go1.9.2

• Prometheus version:
  $ /opt/prometheus/prometheus --version
  prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user: root@149e5b3f0829
  build date: 20180314-14:15:45
  go version: go1.10

$ /opt/prometheus/promtool check config /opt/prometheus/prometheus.yml
Checking /opt/prometheus/prometheus.yml
SUCCESS: 1 rule files found

Checking /opt/prometheus/all.rules
SUCCESS: 5 rules found

• Alertmanager configuration file:

# cat /opt/alertmanager/alertmanager.yml
global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: '127.0.0.1:25'
  smtp_from: '[email protected]'
  smtp_require_tls: false

route:
  receiver: default
  group_interval: 30s

receivers:
  - name: default
    email_configs:
      - to: '[email protected]'

• Prometheus configuration file:

# cat /opt/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
  scrape_timeout: 30s
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093
rule_files:
  - all.rules
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          alias: prometheus
  - job_name: 'grafana'
    scheme: https
    tls_config:
        insecure_skip_verify: true
    static_configs:
      - targets: ['localhost:3000']
        labels:
          alias: grafana
  - job_name: 'server1'
    static_configs:
      - targets: ['192.168.16.1:9100']
        labels:
          alias: server1
.
.
etc

• Logs:

Alertmanager:
$ journalctl -u alertmanager -f
-- Logs begin at Tue 2018-04-17 12:10:14 UTC. --
May 23 18:16:05 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:05.556723857Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/kube.min.css not found"
May 23 18:16:05 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:05.651949342Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/kube.min.css not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.556424622Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/jquery.min.js not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.559950996Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/moment.min.js not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.560048569Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular.min.js not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.560998418Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-moment.min.js not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.638985204Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/kube.min.css not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.652198483Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular.min.js not found"
May 23 18:16:06 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:06.741735536Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-moment.min.js not found"
May 23 18:16:07 server1 alertmanager[25705]: level=warn ts=2018-05-23T18:16:07.206455107Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/kube.min.css not found"

Prometheus:
$ journalctl -u prometheus -f
-- Logs begin at Tue 2018-04-17 12:10:14 UTC. --
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.254108508Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.254145049Z caller=main.go:222 host_details="(Linux 3.10.0-514.6.2.el7.x86_64 #1 SMP Thu Feb 23 03:04:39 UTC 2017 x86_64 server1 (none))"
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.254159382Z caller=main.go:223 fd_limits="(soft=1024, hard=4096)"
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.256940485Z caller=main.go:504 msg="Starting TSDB ..."
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.256986309Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.831656033Z caller=main.go:514 msg="TSDB started"
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.831730584Z caller=main.go:588 msg="Loading configuration file" filename=/opt/prometheus/prometheus.yml
May 23 17:46:55 server1 prometheus[24259]: level=info ts=2018-05-23T17:46:55.834033582Z caller=main.go:491 msg="Server is ready to receive web requests."
May 23 17:47:00 server1 prometheus[24259]: level=error ts=2018-05-23T17:47:00.835291979Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=0 msg="Error sending alert" err="Post http://localhost:9093/api/v1/alerts: dial tcp [::1]:9093: connect: connection refused"
May 23 17:47:06 server1 prometheus[24259]: level=error ts=2018-05-23T17:47:06.28304693Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=0 msg="Error sending alert" err="Post http://localhost:9093/api/v1/alerts: dial tcp [::1]:9093: connect: connection refused"

[simonpasquier]

Did you install alertmanager from package?

[fchiorascu]

Hi Simon,

From here: https://github.com/prometheus/alertmanager/releases/download/v0.14.0/alertmanager-0.14.0.linux-amd64.tar.gz

Kind Regards

[fchiorascu]

I've deleted and unzipped again, but the same thing.
I was thinking to delete the directory and to create and extract again.
Selinux ?

Any suggestion?

[hoffie]

What version did you run before? There is at least one report about a similar case (#770).

I am not too familiar with the alertmanager internals, but it seems like the requests for ui/app/lib/angular.min.js are no longer relevant in the current code base, which makes this look like some caching problem.

Can you

• try clearing your browser's cache
• try another browser
• try running alertmanager on another address/port and access the web UI there
• tell us which browser you are using

... and report back?

[stuartnelson3]

This looks like your browser has the old javascript cached. Clear the cache and it should load fine.

[fchiorascu]

Yes it seems that there was a problem, I've do the following actions:
•Restarted the PC.
•Restarted AlertManager.
•Cleared all browsers cache and restarted them.

An now I have the GUI that seems to be OK:

The past verison of AlertManager was: 0.6.2 / 2017-05-09.

The logs looks like:

[root@server1 ~]# systemctl status alertmanager
● alertmanager.service - Prometheus Alertmanager
   Loaded: loaded (/usr/lib/systemd/system/alertmanager.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-05-24 04:51:53 UTC; 4 days ago
 Main PID: 942 (alertmanager)
   CGroup: /system.slice/alertmanager.service
           └─942 /opt/alertmanager/alertmanager --config.file=/opt/alertmanager/alertmanager.yml

May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.551306384Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-route.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.554182885Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-sanitize.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.571330066Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-resource.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.585093311Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-moment.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.593992714Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.640874249Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-sanitize.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.68741525Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-route.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.733886426Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-resource.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.780828026Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/angular-moment.min.js not found"
May 28 09:11:28 server1 alertmanager[942]: level=warn ts=2018-05-28T09:11:28.880176613Z caller=web.go:33 msg="Could not get file" err="AssetInfo ui/app/lib/kube.min.css not found"

[fchiorascu]

Do you consider normal the logs for AlertManager status?
Thank you for all support!

[stuartnelson3]

Your logs are normal. Your browser was requesting old, no-longer available files, so the alertmanager backend was reporting that the files could not be found.