Alertmanager: silences rule disappeared when restart alertmanager

[249510573]

when i restart alertmanager, the silence rule disappeared,. how to make rules persistence.

what's the effect of parameters as follow.
--storage.path="data/" Base path for data storage.
--data.retention=120h How long to keep data for.

[simonpasquier]

Which version of AlertManager are you using? Do you run a cluster of AlertManager?

[simonpasquier]

The way you restart AlertManager probably doesn't shutdown the process properly. Can you share the logs around the restart?

[249510573]

the silence as follows

the start commond as follows:
./alertmanager --config.file=./alertmanager.yml --log.level=debug --storage.path=./data

first problem: how to read the file ./data/silences

i kill the process of alertmanager (ps -ef |grep alertmanager)
when i restart alertmanager ,the silence i have configured on the alertmanager web have disappeared .

the log as follows:
level=debug ts=2018-12-21T01:56:33.831104761Z caller=dispatch.go:445 component=dispatcher aggrGroup="{}:{target1="node-exporter"}" msg=Flushing alerts=[InstanceDown[acb4421][active]]
level=info ts=2018-12-21T01:57:13.374501039Z caller=main.go:174 msg="Starting Alertmanager" version="(version=0.15.2, branch=HEAD, revision=d19fae3bae451940b8470abb680cfdd59bfa7cfa)"
level=info ts=2018-12-21T01:57:13.374859866Z caller=main.go:175 build_context="(go=go1.10.3, user=root@3101e5b68a55, date=20180814-10:53:39)"
level=debug ts=2018-12-21T01:57:13.374914354Z caller=cluster.go:143 component=cluster msg="resolved peers to following addresses" peers=
level=info ts=2018-12-21T01:57:13.412823443Z caller=cluster.go:155 component=cluster msg="setting advertise address explicitly" addr=10.221.103.141 port=9094
level=debug ts=2018-12-21T01:57:13.414311152Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01CZ77P6J5R9E9YWY8PTMQSQZJ addr=10.221.103.141:9094
level=debug ts=2018-12-21T01:57:13.415725853Z caller=cluster.go:224 component=cluster msg="joined cluster" peers=0
level=info ts=2018-12-21T01:57:13.41589981Z caller=main.go:322 msg="Loading configuration file" file=./alertmanager.yml
level=info ts=2018-12-21T01:57:13.416753337Z caller=cluster.go:570 component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2018-12-21T01:57:13.423161561Z caller=main.go:398 msg=Listening address=:9093
level=info ts=2018-12-21T01:57:15.417010346Z caller=cluster.go:595 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000150348s
level=debug ts=2018-12-21T01:57:17.417268597Z caller=cluster.go:592 component=cluster msg="gossip looks settled" elapsed=4.000424034s
level=debug ts=2018-12-21T01:57:19.41751052Z caller=cluster.go:592 component=cluster msg="gossip looks settled" elapsed=6.00065818s
level=debug ts=2018-12-21T01:57:21.417759865Z caller=cluster.go:592 component=cluster msg="gossip looks settled" elapsed=8.000915453s
level=info ts=2018-12-21T01:57:23.417972361Z caller=cluster.go:587 component=cluster msg="gossip settled; proceeding" elapsed=10.001131545s

[simonpasquier]

You probably use the SIGKILL signal to stop AlertManager which does a hard shutdown of AlertManager shutdown:

level=debug ts=2018-12-21T01:56:33.831104761Z caller=dispatch.go:445 component=dispatcher aggrGroup="{}:{target1="node-exporter"}" msg=Flushing alerts=[InstanceDown[acb4421][active]]
level=info ts=2018-12-21T01:57:13.374501039Z caller=main.go:174 msg="Starting Alertmanager" version="(version=0.15.2, branch=HEAD, revision=d19fae3bae451940b8470abb680cfdd59bfa7cfa)"

If you kill the process with SIGINT or SIGKILL, you should have a log line with msg="Received SIGTERM, exiting gracefully..." instead.

[createdanew]

So what was the outcome of this ticket? Trying figure out the solution here. I have experienced issues when alertmanager restarts and the rules that i have implemented to silence specific alerts disappear. How is it possible to make the rules persistence?

[zhangjianghua]

How to deal with it？