Sidekiq: systemmd production not working

Created on 5 Jun 2019 · 7Comments · Source: mperham/sidekiq

Ruby version:
ruby-2.5.1

Sidekiq version:
sidekiq (5.2.7)

Enviroment: Production
file : /lib/systemd/system/sidekiq.service

 #
 # systemd unit file for CentOS 7, Ubuntu 15.04
 #
 # Customize this file based on your bundler location, app directory, etc.
 # Put this in /usr/lib/systemd/system (CentOS) or /lib/systemd/system (Ubuntu).
 # Run:
 #   - systemctl enable sidekiq
 #   - systemctl {start,stop,restart} sidekiq
 #
 # This file corresponds to a single Sidekiq process.  Add multiple copies
 # to run multiple processes (sidekiq-1, sidekiq-2, etc).
 #
 # See Inspeqtor's Systemd wiki page for more detail about Systemd:
 # https://github.com/mperham/inspeqtor/wiki/Systemd
 #
 [Unit]
 Description=sidekiq
 # start us only once the network and logging subsystems are available,
 # consider adding redis-server.service if Redis is local and systemd-managed.
 After=syslog.target network.target

 # See these pages for lots of options:
 # http://0pointer.de/public/systemd-man/systemd.service.html
 # http://0pointer.de/public/systemd-man/systemd.exec.html
 [Service]
 Type=simple
 WorkingDirectory=/home/ubuntu/projects/costpe/
 # If you use rbenv:
 # ExecStart=/bin/bash -lc '/home/deploy/.rbenv/shims/bundle exec sidekiq -e production'
 # If you use the system's ruby:
 ExecStart=/home/ubuntu/.rvm/gems/ruby-2.5.1/bin/bundle  exec sidekiq -e production
 #User=deploy
 #Group=deploy
 #UMask=0002

 # Greatly reduce Ruby memory fragmentation and heap usage
 # https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
 Environment=MALLOC_ARENA_MAX=2

 # if we crash, restart
 RestartSec=1
 Restart=on-failure

 # output goes to /var/log/syslog
 StandardOutput=syslog
 StandardError=syslog

 # This will default to "bundler" if we don't specify it
 SyslogIdentifier=sidekiq

 [Install]
 WantedBy=multi-user.target

When i run sudo systemctl status sidekiq.service
I am getting below output

  sidekiq.service - sidekiq
    Loaded: loaded (/lib/systemd/system/sidekiq.service; enabled; vendor preset: enabled)
    Active: failed (Result: start-limit-hit) since Wed 2019-06-05 12:55:14 UTC; 1s ago
   Process: 9510 ExecStart=/home/ubuntu/.rvm/gems/ruby-2.5.1/bin/bundle exec sidekiq -e production code=exited, status=127)
  Main PID: 9510 (code=exited, status=127)

 Jun 05 12:55:13 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=127/n/a
 Jun 05 12:55:13 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
 Jun 05 12:55:13 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
 Jun 05 12:55:14 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
 Jun 05 12:55:14 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
 Jun 05 12:55:14 ip-172-31-16-145 systemd[1]: sidekiq.service: Start request repeated too quickly.
 Jun 05 12:55:14 ip-172-31-16-145 systemd[1]: Failed to start sidekiq.
 Jun 05 12:55:14 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
 Jun 05 12:55:14 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'start-limit-hit'.

I can run sidekiq by manually typing this command /home/ubuntu/.rvm/gems/ruby-2.5.1/bin/bundle exec sidekiq -e production from my app directory.

Can some one help me?

Source

vishaltps

👍1

Most helpful comment

I resolved my issue, i replaced bin with wrappers and its working like charm, i also added User as root
from
ExecStart=/home/ubuntu/.rvm/gems/ruby-2.5.1/bin/bundle exec sidekiq -e production
to
ExecStart=/home/ubuntu/.rvm/gems/ruby-2.5.1/wrappers/bundle exec sidekiq -e production
For checking User
run ps -eo pid,comm,euser,supgrp | grep nginx

vishaltps on 6 Jun 2019

🎉5

All 7 comments

Systemd unit file looks OK. It's not clear the reason of the service failure from systemctl status excerpt. Please, could you share journal entries?

$ journalctl -u sidekiq

Tensho on 5 Jun 2019

@Tensho , Please check below response,

-- Logs begin at Sun 2019-06-02 23:35:01 UTC, end at Thu 2019-06-06 04:15:01 UTC. --
Jun 05 11:48:03 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 05 11:48:03 ip-172-31-16-145 systemd[6004]: sidekiq.service: Failed at step USER spawning /usr/local/bin/bundle: No such process
Jun 05 11:48:03 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER
Jun 05 11:48:03 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 05 11:48:03 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 05 11:48:04 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 05 11:48:04 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 05 11:48:04 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 05 11:48:04 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER
Jun 05 11:48:04 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 05 11:48:04 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 05 11:48:05 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 05 11:48:05 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 05 11:48:05 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 05 11:48:05 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER
Jun 05 11:48:05 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 05 11:48:05 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 05 11:48:06 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 05 11:48:06 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 05 11:48:06 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 05 11:48:06 ip-172-31-16-145 systemd[6013]: sidekiq.service: Failed at step USER spawning /usr/local/bin/bundle: No such process
Jun 05 11:48:06 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER
Jun 05 11:48:06 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 05 11:48:06 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 05 11:48:08 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 05 11:48:08 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 05 11:48:08 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 05 11:48:08 ip-172-31-16-145 systemd[6015]: sidekiq.service: Failed at step USER spawning /usr/local/bin/bundle: No such process
Jun 05 11:48:08 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER
Jun 05 11:48:08 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 05 11:48:08 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 05 11:48:09 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 05 11:48:09 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 05 11:48:09 ip-172-31-16-145 systemd[1]: sidekiq.service: Start request repeated too quickly.
Jun 05 11:48:09 ip-172-31-16-145 systemd[1]: Failed to start sidekiq.
Jun 05 11:48:09 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 05 11:48:09 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'start-limit-hit'.
Jun 05 11:49:01 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 05 11:49:01 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 05 11:49:01 ip-172-31-16-145 systemd[6049]: sidekiq.service: Failed at step USER spawning /usr/local/bin/bundle: No such process
lines 1-41

vishaltps on 6 Jun 2019

Failed at step USER spawning /usr/local/bin/bundle: No such process

Sounds like systemd can't infer correct user to run the process. I think you need to return User=deploy back. Don't forget to restart systemd after unit file editing:

$ sudo systemctl daemon-reload

Tensho on 6 Jun 2019

@Tensho

Check my user name and group

ps -eo pid,comm,euser,supgrp | grep nginx
26906 nginx           root     -
26907 nginx           www-data www-data

I comment out User=root s, getting the same output

-- Logs begin at Thu 2019-06-06 11:13:03 UTC, end at Thu 2019-06-06 11:13:23 UTC. --
Jun 06 11:13:17 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 06 11:13:17 ip-172-31-16-145 sidekiq[25091]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Jun 06 11:13:17 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=127/n/a
Jun 06 11:13:17 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 06 11:13:17 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 06 11:13:18 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 06 11:13:18 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 06 11:13:18 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 06 11:13:18 ip-172-31-16-145 sidekiq[25094]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Jun 06 11:13:18 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=127/n/a
Jun 06 11:13:18 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 06 11:13:18 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 06 11:13:20 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 06 11:13:20 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 06 11:13:20 ip-172-31-16-145 sidekiq[25099]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Jun 06 11:13:20 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 06 11:13:20 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=127/n/a
Jun 06 11:13:20 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 06 11:13:20 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 06 11:13:21 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 06 11:13:21 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 06 11:13:21 ip-172-31-16-145 sidekiq[25102]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Jun 06 11:13:21 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 06 11:13:21 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=127/n/a
Jun 06 11:13:21 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 06 11:13:21 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 06 11:13:22 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 06 11:13:22 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 06 11:13:22 ip-172-31-16-145 systemd[1]: Started sidekiq.
Jun 06 11:13:22 ip-172-31-16-145 sidekiq[25105]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Jun 06 11:13:22 ip-172-31-16-145 systemd[1]: sidekiq.service: Main process exited, code=exited, status=127/n/a
Jun 06 11:13:22 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 06 11:13:22 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Jun 06 11:13:23 ip-172-31-16-145 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.
Jun 06 11:13:23 ip-172-31-16-145 systemd[1]: Stopped sidekiq.
Jun 06 11:13:23 ip-172-31-16-145 systemd[1]: sidekiq.service: Start request repeated too quickly.
Jun 06 11:13:23 ip-172-31-16-145 systemd[1]: Failed to start sidekiq.
Jun 06 11:13:23 ip-172-31-16-145 systemd[1]: sidekiq.service: Unit entered failed state.
Jun 06 11:13:23 ip-172-31-16-145 systemd[1]: sidekiq.service: Failed with result 'start-limit-hit'.

vishaltps on 6 Jun 2019

Feels like you have some discrepancy in gems installation (bundle). Is it possible to starts sidekiq server without systemd? Kind of that:

$ whoami # return logged in user
$ cd /home/ubuntu/projects/costpe
$ /home/ubuntu/.rvm/gems/ruby-2.5.1/bin/bundle exec sidekiq -e production

Tensho on 6 Jun 2019

vishaltps on 6 Jun 2019

🎉5

@vishaltps thanks for sharing! but why wrappers not bin ?

hopewise on 6 Jan 2020

👍2

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Potential bug in superfetch?

michaeldiscala · 4Comments

Getting retry count for current job?

HenleyChiu · 4Comments

Sidekiq-Pro 3.4.2 uninitialized constant Rack::Protection

bartimaeus · 3Comments

How to check the logs in Heroku

rajcybage · 3Comments

PG::UnableToSend: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

igorkasyanchuk · 3Comments