Homebrew-core: prometheus not able to start with services mode! but manual working fine.

[teochenglim]

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• [x] are reporting a bug others will be able to reproduce and not asking a question or requesting software. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh. To get software added or changed in Homebrew please file a Pull Request
• [ ] have a problem with brew install (or upgrade, reinstall) a single, Homebrew/homebrew-core formula (not cask) on macOS? If it's a general brew problem please file this issue at Homebrew/brew: https://github.com/Homebrew/brew/issues/new/choose. If it's a Linux problem please file this issue at https://github.com/Homebrew/linuxbrew-core/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• [x] ran brew update and can still reproduce the problem?
• [x] ran brew doctor, fixed all issues and can still reproduce the problem?
• [x] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• [ ] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

To help us debug your issue please explain:

• What you were trying to do (and why)

prometheus can't start using service mode, but if i cd into the /usr/local/opt/prometheus and manual run it works.

$ brew install prometheus
$ brew services start prometheus # -> failed

$ cd  /usr/local/opt/prometheus
$ prometheus --config.file ~/.config/prometheus.yml --web.listen-address=127.0.0.1:9090
level=info ts=2019-09-23T08:33:39.699Z caller=main.go:293 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-09-23T08:33:39.699Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.12.0, branch=non-git, revision=non-git)"
level=info ts=2019-09-23T08:33:39.699Z caller=main.go:330 build_context="(go=go1.12.9, [email protected], date=20190818-13:28:33)"
level=info ts=2019-09-23T08:33:39.699Z caller=main.go:331 host_details=(darwin)
level=info ts=2019-09-23T08:33:39.700Z caller=main.go:332 fd_limits="(soft=256, hard=unlimited)"
level=info ts=2019-09-23T08:33:39.700Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-09-23T08:33:39.702Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090
level=info ts=2019-09-23T08:33:39.702Z caller=main.go:654 msg="Starting TSDB ..."
level=info ts=2019-09-23T08:33:39.706Z caller=head.go:509 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-09-23T08:33:39.709Z caller=head.go:557 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
level=info ts=2019-09-23T08:33:39.709Z caller=head.go:557 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
level=info ts=2019-09-23T08:33:39.711Z caller=main.go:669 fs_type=19
level=info ts=2019-09-23T08:33:39.711Z caller=main.go:670 msg="TSDB started"
level=info ts=2019-09-23T08:33:39.711Z caller=main.go:740 msg="Loading configuration file" filename=/Users/teochenglim/.config/prometheus.yml
level=info ts=2019-09-23T08:33:39.713Z caller=main.go:768 msg="Completed loading of configuration file" filename=/Users/teochenglim/.config/prometheus.yml
level=info ts=2019-09-23T08:33:39.714Z caller=main.go:623 msg="Server is ready to receive web requests."

• What happened (include command output)

$ cat /usr/local/var/log/prometheus.err.log
level=info ts=2019-09-23T07:22:55.217Z caller=main.go:293 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-09-23T07:22:55.218Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.12.0, branch=non-git, revision=non-git)"
level=info ts=2019-09-23T07:22:55.219Z caller=main.go:330 build_context="(go=go1.12.9, [email protected], date=20190818-13:28:33)"
level=info ts=2019-09-23T07:22:55.219Z caller=main.go:331 host_details=(darwin)
level=info ts=2019-09-23T07:22:55.219Z caller=main.go:332 fd_limits="(soft=256, hard=unlimited)"
level=info ts=2019-09-23T07:22:55.219Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2019-09-23T07:22:55.220Z caller=query_logger.go:104 component=activeQueryTracker msg="Failed to create directory for logging active queries"
level=error ts=2019-09-23T07:22:55.220Z caller=query_logger.go:82 component=activeQueryTracker msg="Error opening query log file" file=data/queries.active err="open data/queries.active: no such file or directory"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x30a9411, 0x5, 0x14, 0x366c960, 0xc0002ace40, 0x366c960)
    /private/tmp/prometheus-20190818-71766-xl1o61/prometheus-2.12.0/promql/query_logger.go:112 +0x4d2
main.main()
    /private/tmp/prometheus-20190818-71766-xl1o61/prometheus-2.12.0/cmd/prometheus/main.go:361 +0x52d3

• What you expected to happen

$ brew services list # and see the prometheus    started with green color?

• Step-by-step reproduction instructions (by running brew install commands)

$ brew install prometheus
$ brew services start prometheus # -> failed
$ echo "--config.file ~/.config/prometheus.yml --web.listen-address=127.0.0.1:9090" > /usr/local/etc/prometheus.args
$ brew services restart prometheus # -> failed
$ $ cat ~/.config/prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9101']

[teochenglim]

quite a hack and it is working.

1. add working dir at prometheus plist

   WorkingDirectory
   /usr/local/opt/prometheus

$ cat homebrew.mxcl.prometheus.plist
<?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.prometheus</string>
    <key>ProgramArguments</key>
    <array>
      <string>sh</string>
      <string>-c</string>
      <string>/usr/local/opt/prometheus/bin/prometheus $(&lt; /usr/local/etc/prometheus.args)</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <false/>
    <key>WorkingDirectory</key>
    <string>/usr/local/opt/prometheus</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/prometheus.err.log</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/prometheus.log</string>
  </dict>
</plist>

1. Then it can't find home directory, so switch it at /usr/local/etc/

level=error ts=2019-09-23T09:25:07.515Z caller=main.go:733 err="error loading config from \"~/.config/prometheus.yml\": couldn't load configuration (--config.file=\"~/.config/prometheus.yml\"): open ~/.config/prometheus.yml: no such file or directory"

$ cat ~/.config/prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9101']

$ cat /usr/local/opt/prometheus/prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9101']

$ cat /usr/local/etc/prometheus.args
--config.file /usr/local/opt/prometheus/prometheus.yml --web.listen-address=127.0.0.1:9090

1. restarting service and it is working now

$ brew services restart prometheus
$ brew services list
Name          Status  User        Plist         
node_exporter started teochenglim /Users/teochenglim/Library/LaunchAgents/homebrew.mxcl.node_exporter.plist
prometheus    started teochenglim /Users/teochenglim/Library/LaunchAgents/homebrew.mxcl.prometheus.plist

[sonicsandy]

The above solution works for me on Mac OS and brew service start.

[svenagnew]

:+1: for workaround detailed above.

[sandipb]

Note that editing the launchctl file may not work because I think homebrew generates that automatically. So if you do a brew services stop prometheus and then start it, the plist file may be overwritten.

A better way seems to be to do this on installation.

echo "--config.file $HOME/.config/prometheus.yml  \
      --web.listen-address=127.0.0.1:9090 \
      --storage.tsdb.path /usr/local/var/prometheus" \
      > /usr/local/etc/prometheus.args

This will cause the data directory to be created in /usr/local/var/prometheus instead.

Also note that use ~ as home directory in the echo command to generate the args file wasn't working for me. Turns out that prometheus will not expand the tilde for me. Using $HOME made sure that it is expanded during the echo itself.

[SMillerDev]

That config file location seems odd (why not /usr/local/etc/prometheus.yml?) but this seems like a good addition

[sandipb]

That config file location seems odd (why not /usr/local/etc/prometheus.yml?) but this seems like a good addition

Actually that is a better default. And this lets us put all these command line arguments as the default settings! These settings are perfectly reasonable defaults. Why ask the user to do it themselves? Let them have a reasonable working config out of the box which works immediately after installation. They can tweak it later.

[SMillerDev]

Could you make a pull request that sets these options in a post_install step? so everyone can enjoy them by default?

[teochenglim]

oh man.. @SMillerDev @sandipb I was here asking for help..

~/.config/prometheus.yml is not a weird location. Just the prometheus default. But then it is not Homebrew native. Agreed that it shall be at /usr/local/etc/prometheus.yml or $(brew --prefix)/etc/prometheus.yml

I am not so familiar with Ruby, but i guest the code should be like

https://github.com/Homebrew/homebrew-core/blob/master/Formula/prometheus.rb

  def post_install
    system "echo '--config.file $(brew --prefix)/etc/prometheus.yml  \
      --web.listen-address=127.0.0.1:9090 \
      --storage.tsdb.path $(brew --prefix)/var/prometheus' \
      > $(brew --prefix)/etc/prometheus.args"
  end

and maybe also updating the caveat?

  def caveats; <<~EOS
    When used with `brew services`, prometheus' configuration is stored as command line flags in
      #{etc}/prometheus.args
    Example configuration:
      echo "--config.file ~/.config/prometheus.yml" > #{etc}/prometheus.args
  EOS
  end

[SMillerDev]

Yeah, something like that seems reasonable. You can even remove the example in the caveat since that will already be there.

[teochenglim]

https://github.com/Homebrew/homebrew-core/pull/46618 submitted

[teochenglim]

Thanks @sandipb toke your suggestion of storage.tsdb.path and put into pull request at https://github.com/Homebrew/homebrew-core/pull/46618

[sandipb]

Thanks for the fix! :)

[iMichka]

Fixed by #46618