supervisord Starts All Processes at the Same Time

[fredpalmer]

My config:

[program:redis-testapp]
command=/opt/bcs/bin/redis-server /apps/testapp/releases/current/environments/all/redis.conf
stdout_logfile=/var/log/redis_testapp_log
stderr_logfile=/var/log/redis_testapp_log
startsecs=30
priority=1
autostart=true
autorestart=true

[program:celerybeat-testapp]
command=python -O manage.py celerybeat --loglevel=INFO --schedule=/apps/testapp/db/celerybeat_schedule_db
stdout_logfile=/var/log/celerybeat_testapp_log
stderr_logfile=/var/log/celerybeat_testapp_log
priority=999
startsecs=5
autostart=true

[program:celery-testapp]
command=python -O manage.py celeryd --loglevel=INFO --events
stdout_logfile=/var/log/celeryd_testapp_log
stderr_logfile=/var/log/celeryd_testapp_log
priority=100
startsecs=10
autostart=true

[program:gunicorn-testapp]
command=gunicorn_django --workers=10 --log-level info --timeout 500 --bind=127.0.0.1:8004
stdout_logfile=/var/log/gunicorn_testapp_log
stderr_logfile=/var/log/gunicorn_testapp_log
priority=999
startsecs=10
autostart=true

[program:memcached-testapp]
command=/opt/bcs/bin/memcached -m 128 -l 127.0.0.1 -p 11212 -u nobody -P /apps/testapp/run/memcached.pid
stdout_logfile=/var/log/memcached_testapp_log
stderr_logfile=/var/log/memcached_testapp_log
priority=11
autostart=true
autorestart=true

My output =>

2012-05-30 22:37:33,181 INFO daemonizing the supervisord process
2012-05-30 22:37:33,182 INFO supervisord started with pid 16230
2012-05-30 22:37:34,195 INFO spawned: 'redis-testapp' with pid 16232
2012-05-30 22:37:34,206 INFO spawned: 'memcached-testapp' with pid 16233
2012-05-30 22:37:34,214 INFO spawned: 'celery-testapp' with pid 16234
2012-05-30 22:37:34,238 INFO spawned: 'celerybeat-testapp' with pid 16235
2012-05-30 22:37:34,477 INFO spawned: 'gunicorn-testapp' with pid 16241
2012-05-30 22:37:35,240 INFO success: memcached-testapp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-05-30 22:37:39,434 INFO success: celerybeat-testapp entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2012-05-30 22:37:44,434 INFO success: celery-testapp entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2012-05-30 22:37:44,435 INFO success: gunicorn-testapp entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2012-05-30 22:38:04,197 INFO success: redis-testapp entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)

What I expected to happen =>

redis would start and supervisord would wait 30 seconds before starting any lower priority processes.

[disposable-ksa98]

I don't see what's the point of using supervisor if it does not get this right.

@mnaberez If there is an open issue for this, you should post the link so we can find it.

[rca]

I also need the ability to start processes in a particular order.

An event-based approach could work. For example, if I have a [program:agent] and a [program:client] I could subscribe the client to start up when the agent emits a started event. By default, if a program does not subscribe to any event, it will be started when supervisord starts.

[sevastos]

+1

[maximilize]

+1

[tomislacker]

+1

[dmitriy-kiriyenko]

+1, this major issue.

[jefferai]

+1

[hunterloftis]

+1

[orarbel]

+1

[abierbaum]

+1

[gigaroby]

+1

[KenjiTakahashi]

:+1:

[karanlyons]

+1

[voleg]

+1

[toxsick]

+1

[harmy]

must have

[xbeta]

is there a dependency in supervisor? if not, why not?

[analytically]

+1

[devrim]

+1

[alp82]

+1

[mtrienis]

+1

[Pindar]

+1

[miguno]

Not wanting to add to any perceived pressure, but I'd be +1 on the very same feature (process dependencies). :-)

[nand0p]

this would be very useful and mean less ugly hacks like sleep

[jowenn]

+1

[pangon]

+1

[ricardosasilva]

+1. Maybe an option to turn the process loading synchronous.

[brianholcomb]

:+1:

[clintecker]

+1

[kgadek]

:+1:

[vladfr]

+1

[bataras]

+1

[tomislacker]

Anyone got an other implementation thoughts on this? I'm considering making a run at it since no one else seems to care.

[lostsnow]

+1

[kgadek]

@tomislacker well, IMHO the best solution is to make dependency-solver à la Puppet. If I can see well, then simplest topological sort with DFS should be more than enough. Then add the "requiresstarted" field (or "waitforstarted", or sth similar). This issue would be fixed by the way.

(edit) Well, not sure what would happen if while having A -> B -> C (A depends on B, etc), someone would try to stop B. Should A be stopped as well? Well… /depends/…

[sinchb]

:+1:

[swindmill]

[kamilion]

Since everyone seems to need/want to do this a slightly different way -- how about a slightly different tack? Instead of attempting to express dependencies directly in a configuration directive, how about instead adding a directive for a 'helper' script, and a couple example helpers that suit 2-3 general use cases with differing requirements as @kgadek pointed out?

Here's a good example to demonstrate what I mean:
http://www.serfdom.io/docs/recipes/event-handler-router.html

There's some stuff out there for doing this sort of thing easily in bash,
https://github.com/progrium/pluginhook

And you certainly can't discount the nicety of "I felt like writing it in python" (Or any other language of admin's choice)
https://github.com/garethr/serf-master

It seems to me like this would be a reasonable win for everyone without consuming a lot of developer time on supervisor itself.

[vladfr]

Hey,

The serf model looks very promising. Having a collection of scripts instead of just one handler seems appropriate (e.g. checking for 2 deps before starting)

So, you would define a handler to see if process/group:x "can_be_started", and if all scripts return 0, then it's all good. Sounds right?

Imho this makes "priority" obsolete, am I wrong?

[jefferai]

For my needs, simply having a per-process delay from the time the supervisor daemon starts would be enough.

[binhex]

yet another +1 for this

[samet]

+1. Per process delay would be enough tho.

[kamilion]

I've been able to work around this a bit. Quick and dirty.

# Deal with updating our repositories.
supervisorctl start source-code-deploy

# Check for the oneshot process to complete.
while ! supervisorctl status source-code-deploy | grep -q 'EXITED'; do sleep 1; done
# Wait for the while loop to break out signalling success.

# Start the late boot process, now that the deployment is complete.
supervisorctl start system-boot-late

# Check for the oneshot process to complete.
while ! supervisorctl status system-boot-late | grep -q 'EXITED'; do sleep 1; done
# And now we should become EXITED to supervisord and any other tasks relying on the above.

[NicholasTurner]

+1

[msierks]

+1

[vladfr]

Hello,
I'd like to have a go at this in the coming weeks, let me know if anyone else is working on it.

[liutec]

Attempting to start many processes at exactly the same time tends to push the machines to the limit and sometimes even beyond.

Having a parameter like "startdelay" would be very useful for me especially since some processes tend to use a lot of resources when first starting, resources which are released after a short while.

Unfortunately there are quite a few ways to implement such a feature and finding the best one could take some time.

@vladfr Perhaps we could collaborate on this.

My first "dirty" crack at it:
https://github.com/liutec/supervisor/commit/eab7cc1e04ad49768593183e8134298604459827
(I really don't like having to use sleep this way.)

[vladfr]

Hi @liutec, look above at what @kamilion is saying about a hook model - I think this is useful to implement custom checks, and could be used to just sleep for a simple case.
You could run the hook scripts in subprocesses and they can sleep all they want.

[liutec]

In my situation, the goal is to be able to have supervisor manage around 240 distinct programs some of which may require more than one instance.

The start delay is only useful for the first time a program is started, simply to avoid the otherwise unlikely situation when all programs start at the exact same time.

Most of the programs are consumers and will only run for a short amount of time before they shutdown and are restarted by the supervisor -- in this case having a start delay will not do any good.

I've considered both @kamilion 's solution as well as the solution given by @mnaberez (autostart autorestart) but unfortunately none actually produce an easily configurable "startdelay".

[ripasapa]

+1 if supervisord had "sequence control" that would be great, now Kafka depends on Zookeeper etc. and supervisor cannot used like we would like to.

[jackwilsdon]

+1

[phf]

I am using supervisord to start a bunch of exotic qemu instances (for a compilers course) and some of these boot up quickly by themselves but take quite a long time if started all at once. A simple "startdelay" thing would work just fine: Sleep for the given number of seconds after starting this process before moving on to the next one in the list. I don't know exactly how you decide to order these (the priority keyword doesn't seem to order strictly as far as I can tell from the created pids) but whatever it is, being able to delay for a few seconds after each process would help a whole bunch in my case. So :+1: for sure. Twice, actually. :-)

[skafandri]

I think a startdelay option would be very usefull. I have a supervisor to manage an AMQP server and some consumers. Some consumers implement a sleep before starting, some loop over a try to connect / error / sleep sequence (different implementations). I know the maximum time my AMQP server needs to start, so adding a startdelay option to the consumers would avoid those ugly hacks.

I think this may be very helpful in many situations.

After checking the code, I don't see any BC issues, also the supeervisor behaviour is the same when using the default value.
I can't wait for this to be merged.

[kforner]

+1 for dependency support

[sukrit007]

+1

[danros]

+1

[maparent]

+1

[5d]

+1

[karl-forner-quartz-bio]

+10 ;)

[Bessonov]

+1.

[msabramo]

This issue needs to be renamed, because I don't think that startsecs is relevant here. I feel like the OP might've misinterpreted what startsecs is, thinking that it's a delay to wait before starting the process.

startsecs

The total number of seconds which the program needs to stay running after a startup to consider
the start successful. If the program does not stay up for this many seconds after it has
started, even if it exits with an “expected” exit code (see exitcodes), the startup will be >
considered a failure. Set to 0 to indicate that the program needn’t stay running for any
particular amount of time.

[arinto]

+1

[mcneilcode]

+1

[anduslim]

important feature
+1 as well

[seanmcl]

+1

[ahosie]

+1

[piotr-piatkowski]

+1

[maxexcloo]

+1

[arturhoo]

+1

[debdude]

Simple workaround I use:

[program:uwsgi]
command=bash -c 'sleep 5 && uwsgi /etc/uwsgi.ini'

[ReSTARTR]

+1

[MorganAntonsson]

+1

[tomislacker]

Anyone have thoughts on how we may be able to solve this issue in a more responsible fashion? What if there were to be a command that could be invoked to validate the _"online"_ state of a process? Such as:

[program:myapp]
command=/usr/local/bin/myapp-microservice
checkcommand=curl -s http://localhost:54321/v1/_ping
checkfreq=1
checktimeout=3
startsecs=5

The above example would imply:

1. The command would be run as normal
2. The checkcommand would be executed every checkfreq seconds after the above invocation until startsecs, possibly in conjunction with checktimeout, causes an abort or the command returns 0.

_Example: If each call to checkcommand takes <=3 seconds, checkfreq=1, checktimeout=3, and startsecs=5; checkcommand gets run +1.0s, and if it failed then, +5.0s, and discontinues marks the serviced failed on the conclusion of the second checkcommand invocation._

1. If any invocation of checkcommand returns an exit status of 0, then service is considered online.
2. If checkcommand does not return an exit status of 0 after startsecs, or the last possible invocation of checkcommand hangs for >=checktimeout seconds, the service start is considered as a failure & _"normal"_ logic is assumed that is already in placed. _(Mark the service as a failure, output the same log content, etc...)_

It just seems to me that we want the problem solved but we haven't described it well enough in this issue. The low-hanging-fruit answer is certainly make sure that services with different priorities are not invoked until the _max(startsecs)_ of all higher priority services. But aren't we only asking that in hopes of being able to loosely choreograph our intended result instead of being able to validate it on the way there?

[bazilio91]

:+1:

[jefferai]

@tomislacker I think for many uses loose choreography is an acceptable 90% case even if full validation is the 100% case.

Once you get into true dependency management you start looking a lot more like a real init and it starts becoming a much more complicated problem. But there are definitely low-hanging-fruit scenarios.

[vincent-io]

:+1:

[ghost]

+1 I need this

[vladfr]

@tomislacker this is a good idea, and easy to use because you just need the current config file.
But how do you handle an actual dependency? If my_program depends on Postgres, I check for it in checkcommand for a couple of times and hope it will start?

To express a dependency, I think you want to run checkcommand before the actual command.

[klahnakoski]

@tomislacker I would prefer there be simple dependencies between the programs

[program:A]
command=/usr/local/bin/A

[program:C]
command=/usr/local/bin/C
dependson=A

and let the user decide if the programs are going to check for more complicated conditions. For example: We can insert [program:B] that will perform a sophisticated check to ensure A is running properly; and fail if not. So, A starts. B depends on A so will only start when Supervisor considers A is RUNNING. When B starts, it will perform its analysis and fail if A is not running properly. C only starts when B ran (or is running successfully).

My suggestion does not cover the extra features you suggest, like number of checks, the timing of those checks, and when to give up; but I also advocate giving Supervisor some Cron-like features so programs like B are easy to define:

[program:A]
command=/usr/local/bin/A

[program:B]
command=curl -s http://localhost:54321/v1/_ping
dependson=A
startretries=3
restartintervalsec=5

[program:C]
command=/usr/local/bin/C
dependson=B

[mnaberez]

I also advocate giving Supervisor some Cron-like features

Please see response in #635. This has been considered at length by the Supervisor developers in the past and it was decided that cron-like functionality is out of scope for this project. Sorry.

[ain]

:+1: for a way to deliver sequential execution in a feasible fashion.

Using something like startuppriority would be a good robust way to do this IMO.

[SimaWB]

+1

[vladfr]

@klahnakoski in your first example, C depending on A means that supervisor needs to know A is ready. Right now, it only knows it's started.

[klahnakoski]

@vladfr yes, your are correct. That is why I also suggested B, which is a more sophisticated check to determine if A is ready. Ideally B is triggered periodically, and will fail if A is not ready.

[TalonOne]

+1

[zjjott]

+1

[carlopires]

+1 ?? This is from May, 2012, I think that supervisor will never have dependencies.

Someone could suggest alternatives with dependencies management?

[aisengard]

+1, this would make things much more elegant

[naktinis]

+1

[gastounage]

+1

[cocasema]

+1

[Esya]

+1, it's sad that this is not doable after 3 years, using systemd units for now

[citizen-stig]

 .----------------.  .----------------. 
| .--------------. || .--------------. |
| |      _       | || |     __       | |
| |     | |      | || |    /  |      | |
| |  ___| |___   | || |    `| |      | |
| | |___   ___|  | || |     | |      | |
| |     | |      | || |    _| |_     | |
| |     |_|      | || |   |_____|    | |
| |              | || |              | |
| '--------------' || '--------------' |
 '----------------'  '----------------' 

Or please, add it.

[vellamike]

+1

[cutewalker]

+1

[wojss]

+1

[macayaven]

+1

[ddzialak]

+1

[chenjie4255]

+10086

[benjaminjones]

+1

[ravihuang]

+1

[pmec]

+1

[wicksy]

+1

[feisuzhu]

+1

[vishwaraja]

+1

[benma]

+1

[weissjeffm]

+1

[fxstein]

+1

[m30m]

+1

[maguowei]

+1

[tobixx]

+1

[rpzatkoff]

+1

[jorgegarciadev]

+1

[boyvinall]

Wow, really? How come no-one cares about this? The "dependson" sounds like a good idea here. The workaround I've heard mentioned was to launch a script as the program, where the script checks for dependencies and otherwise bombs out. By virtue of bombing out, the script will be re-launched later.

[GonZo]

+1

[mtokumaru]

+1

[vac]

+1

[josehenriqueventura]

+1

[zhuangsirui]

+1. 2016 right now...

[Pithikos]

+1

[ppmathis]

+1

[aplantenga]

+1

[dotpmrcunha]

+1

[ain]

Guys, please use the Like button on the original topic instead of the old-fashioned +1 comment :) This is what Github made it for.

Thanks!

[vincent-io]

Guys, please use the Like button on the original topic instead of the old-fashioned +1 comment :) This is what Github made it for.

+1 :trollface:

[jheiss]

My vote would be for a simple dependson style system where you express dependencies and supervisord waits for them to start before starting their dependents. It would be great if supervisord would also stop them in the correct order when it was shutting down. However, if I manually stop a single service I don't think it is necessary for supervisord to try to stop dependents in that scenario. If someone wanted to get fancy they could add a flag to stop that shut down dependents.

I need supervisord to be smart about ordering in situations where it is unattended, e.g. startup and shutdown. I don't need it to be smart in response to specific commands I trigger, e.g. stop and start.

[orrery]

Hi @mnaberez, I was wondering what the status of process dependency feature. You mentioned in 2012 it was on the to-do list? Is this still coming?

[mnaberez]

I was wondering what the status of process dependency feature. You mentioned in 2012 it was on the to-do list? Is this still coming?

It is on the to-do list (TODO.txt) and this issue is still open. I don't think any developers are currently working on it.

[confessin]

+1

[orrery]

Is this really difficult to implement in supervisor? I was wondering if there are other similar tools to supervisor that provide this feature?

[AJNOURI]

+1

[khotkevych]

+1

[maxivak]

+1

[adamgoose]

+1

[madhusudhanane]

+1

Any updates? when we will have this feature, it's a huge win for supervisord for our applicaiton.

[c0b]

I think supervisord is mostly used as lightweight init manager, instead of heavy lifting systemd (upstart, sysv-init, ...) solution, but where systemd is managing service start dependencies really well and supervisord is still lacking such feature since this issue reported in 2012

Why can't some developer here implement a simple dependency like ? is this project in out of maintainer

https://fedoramagazine.org/systemd-unit-dependencies-and-order/

Wants=sshd-keygen.service
After=network.target sshd-keygen.service

:+1:

[vladfr]

Hello all!

I finally had some time and gave this a try. This implements "dependson" as a config option for both [program] and [group]. You can specify other programs and groups as dependencies, and supervisor will ensure they will be started before your program.

Example config:

[program:a]
command=a

[program:monitord]
command = ./monitord.sh
dependson=a
startsecs=5

[group:monitoring]
programs=c1,c2
dependson=monitord,a

[program:c1]
command = bash c1.sh
dependson=anotherprogram ; this doesn't matter because c1 is part of a group
autorestart=true

[program:c2]
command = bash c2.sh
autorestart=true

This config will ensure the following start order:

1. a
2. monitord
3. monitoring group, which will start c1 and c2

Notice that program:c1 dependson=anotherprogram
Since c1 is part of a group, its dependson value will be overriden by the group value. In my view, members of a group should depend on the same stuff.
Any thoughts here?

Please see the changes, and feel free to checkout the branch "dependson_122" to play around. There's still stuff to do (circular deps, finish tests, etc.) but first, I want some feedback!

https://github.com/Supervisor/supervisor/compare/master...vladfr:dependson_122?expand=1

Thank you in advance.

[c0b]

sorry for the mistakenly created #776

[psigen]

Just to summarize the systemd semantics referenced in https://github.com/Supervisor/supervisor/issues/122#issuecomment-227241447:

• Requires= This directive lists any units upon which this unit essentially depends. If the current unit is activated, the units listed here must successfully activate as well, else this unit will fail. These units are started in parallel with the current unit by default.
• Wants= This directive is similar to Requires=, but less strict. Systemd will attempt to start any units listed here when this unit is activated. If these units are not found or fail to start, the current unit will continue to function. This is the recommended way to configure most dependency relationships. Again, this implies a parallel activation unless modified by other directives.
• BindsTo= This directive is similar to Requires=, but also causes the current unit to stop when the associated unit terminates.
• Before= The units listed in this directive will not be started until the current unit is marked as started if they are activated at the same time. This does not imply a dependency relationship and must be used in conjunction with one of the above directives if this is desired.
• After= The units listed in this directive will be started before starting the current unit. This does not imply a dependency relationship and one must be established through the above directives if this is required.
• Conflicts= This can be used to list units that cannot be run at the same time as the current unit. Starting a unit with this relationship will cause the other units to be stopped.

From https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#unit-section-directives

This subset of commands is extremely powerful, and I strongly suspect that the well-formed dependency-tree + event-generation you would need to implement any one of these directives makes the rest of them quite straightforward.

[psigen]

@vladfr (https://github.com/Supervisor/supervisor/issues/122#issuecomment-228416456)

Since c1 is part of a group, its dependson value will be overriden by the group value. In my view, members of a group should depend on the same stuff.
Any thoughts here?

I think a clearer semantic is to union the dependencies together.

This is useful in cases where a group as a whole depends on a few processes, but there is a member that might depend on an additional process: e.g. a web application where one component is interacting with a database server, but the other components are just serving static files.

In this case, many members of the group could be started earlier than the one member dependent on an extra process.

[krisdigitx]

+1

[danielzheng]

+1, old-fashioned way :smile:

[marc2982]

+1 plz

[sangechen]

check this:
https://mmonit.com/monit/documentation/monit.html#SERVICE-DEPENDENCIES

[rrei]

Is there any logic reason why this is not being merged?! It's been 4 years and dozens of people requesting this feature... I'm one of them. This is frustrating.

[vladfr]

Hi! wow, people are really interested in this.

@psigen sorry for the delay on this. Merging deps of a group makes the config clearer, but it's not better functionally. Due to the way supervisor groups are implemented, you need to start all-or-nothing.

In other words, this won't happen: "In this case, many members of the group could be started earlier than the one member dependent on an extra process." - b/c I can't start just some members of a group.

However, I'm not opposed to merging, because it makes it super-clear from the config which service has the actual dependency.

I'll spend some time next week to implement this, add tests and rewrite docs. The monit page looks really good, I'll do a review and see if there are any low-hanging fruit.

Any other comments on the general approach? @rrei I know there are many people that could use this today, so I'd really like to know if this can be used as-is, before submitting the actual PR.

Thank you!

[psigen]

@vladfr I see, that's alright, it's just an optimization after all. Thanks for the feedback.

I do think that union-ing dependencies makes it clearer to trace dependencies and avoid duplicating information at the group level (and means you don't have to re-evaluate group dependencies every time a member is added or removed).

---

Overall, if you have only a single DEPENDS directive, the monit-style semantics seem most appropriate (with the appropriate translation to supervisord run states, and allowing the directive to refer to both members and groups as dependencies):

https://mmonit.com/monit/documentation/monit.html#SERVICE-DEPENDENCIES

The syntax for the depend statement is simply:

DEPENDS on service[, service [,...]]
Where service is a check service entry name used in your .monitrc file, for instance apache or datafs.

You may add more than one service name of any type or use more than one depend statement in an entry.

Services specified in a depend statement will be checked during stop/start/monitor/unmonitor operations.

If a service is stopped or unmonitored it will stop/unmonitor any services that depends on itself.

If the service is started, all services which this service depends on will be started before starting this service. if start of some service failed, the service with prerequisites will NOT be started and the, but will remember that it should start and will retry next cycle.

If a service is restarted, it will first stop any active services that depend on it and after it is started, start all depending services that were active before the restart again.

The more complex systemd semantics cover an even wider set of use cases, and would probably be desirable at some point, but are obviously more intricate and may not fit the internal state that supervisor is using right now.

[vladfr]

Ok, thanks @psigen! I'll finish the simple approach of "dependson".
I added circular deps detection and docs. I need to finish tests and that's it.
This feature works as expected with group deps, [fcgi-process], and supervisorctl.

It's been suggested here that Stop/restart should be manual, i.e. dependson applies as a start-only restriction. So this is what's happening now: after a program with dependson runs, it's independent: if A depends on B, A is not stopped when B terminates. This is mentioned in the docs.

Here are a few more points of discussion, feedback welcomed.

1. supervisorctl: start will respect deps. Do you feel there is a need to add an override for RPC calls? process.spawn could receive an argument to skip deps checking. This can be a config in the [supervisorctl] section. We can even include this in the [supervisord] section.
2. Do we want to trigger an event whenever a program can't start because of an unmet dep?
3. Here is an example of the WARN message for an unmet dep; is this ok as a WARN? I can see it as a DEBUG, because it's kinda spammy, and this message is expected.

2016-08-08 19:21:00,419 WARN process 'tom' cannot start - group 'cats' depends on processes which are not started yet: ['mouse', 'night']

So @mnaberez how is all of this looking to you so far?

[mnaberez]

So @mnaberez how is all of this looking to you so far?

I haven't looked at any of this. My initial feedback is that if some sort of dependency mechanism was to be added, it would need to have good user feedback in order to not become a support burden. If it were to block a command like supervisorctl start, I think it has to surface that information to RPC consumers so supervisorctl can print the appropriate messages. That may imply new process states or RPC response codes. My experience answering support tickets is that most users are not reading the main log as much as they should. If the dependency code blocks supervisorctl commands, that information needs to be visible in supervisorctl.

[vladfr]

@mnaberez Thank you, I'll give that some thought! Right now, the process goes straight to an error state (abnormal termination).

For the record, this has been a personal thing for some 18 months now, so if it's considered for merge, it's a bonus. In any case, it sounds like a candidate for 4.0.

[jneu]

+1

[donreilly]

+1

[alexzam]

+1

[FAKERINHEART]

@vladfr I think that in your supervisord branch, setting the 'dependson' to be a member of the class GroupProcessConfig would be a better choice. For it is easier to iteratively call 'removeGroup()' in the 'rpcinterface.py'.

[FAKERINHEART]

So enthusiastic~

[vladfr]

Yeah I think I had it like that at some point but changed it to allow for
merging deps at group-level. I haven't looked at rpc yet but I have to,
thank you for the suggestion!

On Sun, Aug 28, 2016, 15:19 FakerInHeart [email protected] wrote:

So enthusiastic~

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Supervisor/supervisor/issues/122#issuecomment-242971739,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAM-h4JlTLaoxoMLEJRkNLBr_ZoOOmZzks5qkXy0gaJpZM4AAzaa
.

[FAKERINHEART]

@vladfr The commands you input in the supervisorctl commandline will be transferred to 'function calls' to the class functions in the rpc. And rpc will finally call the member functions of the class supervisor in 'supervisord.py'.

[codeskyblue]

too many users, too few developers . orz

[codeskyblue]

@FAKERINHEART This is the most amazing I have seen. This situation make me to think is this feature request really needed. Or in another way is that the source code is too hard to modify. Haha, really superize me, after all this is an two years issue.

[codeskyblue]

@FAKERINHEART I did try to write another supervisor using golang, because I just need part of the supervisor function, especially the supervisor web manager page. Still in progress https://github.com/codeskyblue/gosuv

[artinnok]

+1

[treevesvarndell]

+1

[nicobouliane]

+1

[strarsis]

Edit: I am currently using chaperone which supports service ordering/dependencies.

[FAKERINHEART]

I've tried customizing the 'supervisor' to satisfy the dependencies among programs, which is partially based on codes provided by vladfr. However, when I read the source code of the origin version of the 'supervisor', I think that there is something not excellent for the architecture of the 'supervisor'. I found that the basic unit of subprogram config in the 'supervisor' would be transferred to 'config group'(processGroupconfig), no matter whether its config-key is 'program'(homogeneous group) or is merged into a 'group'(heterogeneous group) in the config file.
And for the 'supervisord' class object and in its transition loop, its is based on 'the array of subprocess group'(process_groups) that is generated from the 'config group'(processGroupconfig) I mentioned above. I think that the 'config group' unit is so big that is not advantage for 'superviosr' to manage them. Furthermore, it's more difficult to realize the dependencies among the subprograms, if this function will be in your plan in the future.
I think that it would be better if the basic unit of subprogram is 'config program'(may be called as 'processProgramconfig'), and 'config group'(processGroupconfig) is only used to put them together.

[gaojice]

+100

[jasoncorbett]

I really needed this, and an event listener didn't sound too hard so I wrote one. It won't solve everyone's problem, it doesn't do dependencies, it's just ordered startup:
https://pypi.python.org/pypi/ordered-startup-supervisord/

[petrem]

+1 ...

[vladfr]

@FAKERINHEART apologies for the late reply. I think you are spot on - re: what you described in your last comment

the basic unit of subprogram config in the 'supervisor' would be transferred to 'config group'(processGroupconfig), no matter whether its config-key is 'program'(homogeneous group) or is merged into a 'group'(heterogeneous group) in the config file.

That is exactly the issue, and why this task is not straightforward.
Did you find an improvement or another solution here? I'd love to chat more about it, hit me up.

[Gmanweb]

+1

[joy4eg]

+1

[jerome-plumecoq]

+1

[ZzAntares]

+1

[rastislavs]

+1

[rdrsss]

+1

[jacohend]

+1

[bipeens]

+1

[goddanao]

+1

[julien6387]

For those who are interested, I have addressed this feature in https://github.com/julien6387/supvisors through the 'start_sequence' process' rule of the deployment file.

Supvisors is designed for distributed applications but "if you can move mountains you can move molehills".
It works with a single Supervisor instance.

[GraphicsEmpire]

+1

[afonsosilva91]

+1

[davidraleigh]

+1

[livelace]

+1

[keyndark]

+1

[alexhudici]

+1

[FAKERINHEART]

https://github.com/FAKERINHEART/supervisor
See the Introduction.txt and use the branch dev-3.3.1-sr1.
I have had this branch with dependency features among groups(programs) for supervisor, which is partly based on the codes suggested by @vladfr .
If you find any bugs, please tell me in time.

[prapdm]

+1

[Clamb94]

+1

[orrery]

This issue has been open since 2012, does anyone have ownership over it? It's clearly important to many users; is there any consensus amongst supervisor contributors?

[swapdavv]

+1

[Constantin07]

+1

[bogdanm]

+1

[orrery]

someone on the supervisor project needs to take ownership of this, or close the ticket with a reasonable reason for why this won't be addressed e.g. perhaps it doesn't need to be addressed because supervisor can re-attempt a restart on an error condition (a solution that has worked for me).

This issue has been open since 2012 and it's getting kind of ridiculous that a ticket would remain open for this length of time.

[greenaussie]

+1...

[ecebuzz]

+1

[pickledgator]

+1

[chezbut]

+1

[alecbz]

FYI: replying to an issue just to say '+1' sends a notification to everyone subscribed to the issue. You can just react to the issue's initial comment with the 👍 emoji instead.

[AndrewSav]

@AlecBenzer

FYI: replying to an issue just to say '+1' sends a notification to everyone subscribed to the issue.

But that's kind of the point, is not it? I'm subscribed to this issue, and I'm very interested to see that other people are interested is well. This issue is more than 5 years old. It is a good thing to actually ping the developers to gently remind them that people are still interested. It is also good for those who subscribed, to confirm that there are also others who care, not just that person ;)

FYI: There is an "Unsubscribe" button to the right of the thread, if you do not like to receive the notification from this thread, this is perfectly fine, you can use this button to opt-out.

[LMCom]

I can confirm, that starting processes in a specific order is an important requirement. Sorry, I was not able to confirm it earlier. Will it be implemented now?

[jkemp101]

Sharing my solution to this problem, your mileage may vary. While waiting for this to be fixed I transitioned most of the pieces of our infrastructure relying on this feature to Docker/Kubernetes. The various Docker orchestration frameworks (docker-compose, swarm, Kubernetes, etc) typically have mechanisms to control startup ordering (often with their own set of challenges). So if you are waiting for this to be fixed maybe use it as an excuse to play around with a simple docker-compose project to see if it provides an alternative approach to this problem. I recognize this may only apply to a small fraction of the users waiting for this to be fixed.

[bendikro]

Hi everyone

I've implemented an event plugin with support for starting services in order specified by dependent services.
Take a look at this example.

https://github.com/bendikro/supervisord-dependent-startup

[CoryPulm]

An alternative to @bendikro's awesome plugin could also be the Docker community's wait-for-it.sh script. It's a bit easier to set up if you have a complex automation system managing it, though it isn't as pretty as a native plugin:
https://github.com/vishnubob/wait-for-it

[mtdeguzis]

Also, you can do a simple wait for certain processes like this:

# Wait until PHP FPM is up to start, since supervisord like to start everything at once...
# See
while [[ $(ps -aux | grep "[p]hp-fpm: master process") == "" ]];
do
    echo "Waiting for PHP FPM to come up..."
    sleep 2s
    if [[ $(ps -aux | grep "[p]hp-fpm: master process") != "" ]]; then
        echo "PHP FPM looks to be started, continuing with Icinga daemon initialization"
        ps -aux | grep "[p]hp-fpm: master process"
    fi
done

I only have a few, so I settled for this method

[DenLilleMand]

Something i just realized is that, let's say i use supervisor to host my database and my server. The server service has autorestart=true so as long as it fails connecting to the database it is just going to retry right. So it works out.

[CBEPX]

+1

[zgdgod]

+1

[eugene-kulak]

+1

[mtdeguzis]

Please click the thumbs up on the original post instead of just commenting "+1". Gets my hopes up when I see a notification :D

[AdiBrucker]

On option to solve this problem is to define in the supervisor.conf that the program will get autostart value of false for example:

[program:test]
Autostart:” false”
Command=“/home/user/test”

Second step is to use bash script to start the supervisorctl example:

Gnome-terminal -e “supervisord”
Gnome-terminal -e “supervisorctl”
Sleep 10
Gnome-terminal -e “supervisorctl start test”

*Thanks to Avner gidron for giving me the idea

[kumarshorav11]

Hi,
I have below script in the supervisor.cnf file :

[program:zookeeper]
startsecs=60
directory= /app
command=/bin/bash -c "java -jar zoo.jar"
priority=1
autostart = true
autorestart = true

[program:kafka]
startsecs=60
directory= /app
command=/bin/bash -c "java -jar kaf.jar"
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
priority=999
autostart = true
autorestart = true

I want to start first Zookeeper and then Kafka when Zookeeper is up. This script is some times not working as expected.
What would be the way to handle this through supervisor.
Please suggest.

[mtdeguzis]

@kumarshorav11 it's not pretty, but what I do is monitor for the process to come up first (whatever needs to wait)/. This is an example only.

httpd.conf

[program:httpd]
command=/opt/supervisor/httpd_supervisor
autorestart=true
startretries=3
# Start only after PHP FPM
priority=2
# redirect output to stdout/stderr and do not use a regular logfile
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

/opt/supervisor/httpd_supervisor:

#!/bin/bash

# Wait until PHP FPM is up to start, since supervisord likes to start everything at once...
# See https://github.com/Supervisor/supervisor/issues/122

echo -e "\n==> Starting httpd"
# PHP FPM
while [[ $(ps -aux | grep "[p]hp-fpm: master process") == "" ]];
do
    echo "Waiting for PHP FPM to come up to start httpd..."
    sleep 2s
    if [[ $(ps -aux | grep "[p]hp-fpm: master process") != "" ]]; then
        echo "PHP FPM looks to be started, continuing with Icinga daemon initialization"
        ps -aux | grep "[p]hp-fpm: master process"
    fi
done

# Another project had this, but we won't be using systemd
# service httpd start
/usr/sbin/httpd

# Allow any signal which would kill a process to stop server
trap "service httpd stop" HUP INT QUIT ABRT ALRM TERM TSTP

while pgrep -u apache httpd > /dev/null; do sleep 5; done

[danielmotaleite]

Another team with this problem, we need to start a apps only after dnsmasq correctly started, if not the app will resolve and cache the wrong DNS

So there is already the https://github.com/jasoncorbett/ordered-startup-supervisord, why this is not merged in the main supervisor, is there any reason for this not to be fixed?

edit: another similar fork https://github.com/bendikro/supervisord-dependent-startup

[OrangePJ]

Any elegant solution?

[mrkeyiano]

i'm all for this feature

[benma]

It has been fun tracking this issue for all these years, but I finally unsubscribed :joy:

[feistiny]

My case is that I just want to control the stopping order when manually not in supervisord control period, A > B,C ,
Simple solution: in a shell script, write them one by one,
supervisorctl stop A:*
supervisorctl stop B:* C:*
supervisorctl start A:* B:* C:*
Or if there are other than B,C, do a grep like this
supervisorctl status | grep -v '^A:*' | grep -v '^B:*' | grep -v '^C:*' | awk -F':' '{print $1":*"}' | xargs | supervisorctl restart

[msangel]