Supervisor: Option to create directory of specified log files

[felipemachado-sambatech]

I tried to set the logfile to a file in a directory that doesn't exist. Shouldn't there be an option to automatically create these directories?

[mnaberez]

I'd rather not add this option, sorry. The config file is already quite verbose and this option may not provide much benefit over mkdir. It would probably be less effort to just run mkdirthan to shut down supervisord, edit the configuration file to enable the automagic directory option, and start supervisord again.

[sherbert]

For the record, we would also appreciate this feature. We keep our log files in a path that is periodically purged of files and directories that haven't been touched in some amount of time (and there's nothing I can do about this). This means that if we shut down a process for a month, its log directory will be gone when we try to restart it, and so it would be nice for that directory to be automatically recreated.

[RobWin]

+1 for this option

[jaanauati]

+1

[felipou]

As a related note, this diminishes the reliability of supervisor. I don't know if that's been solved somehow in the master branch, but in version 3.1.1, someone accidentally removed a log directory in our server, and supervisor simply stopped working, taking all other programs with it, not just the one with the log directory missing.

I think the scalability of supervisor is also compromised because of this, since one problem in one of the programs can end up taking all others down.

I'd be happy to contribute with a pull request, I just need some pointers as to where I should start in the code.

Also, I don't know if I should create another issue for this...

[coderlol]

+1...very reasonable request, why not honoring it?

[rb2nem]

Missing this too....

[paambaati]

Is there a workaround that will at least let us create the directory from within the config file? Maybe run mkdir inside?

[kujiy]

+1

We docker users should do this

1. ENTRYPOINT["sh", "/entrypoint.sh"] in Dockerfile
2. In /entrypoint.sh, create a log directory and log files before calling supervisord

This is a shell script for /entrypoint.sh to create them from supervisord.conf automatically.

# Create log dirs and files
mkdir -p $( dirname $(cat /etc/supervisord.conf  | grep logfile= | grep "\.log" | sed s/.*logfile=// ) )
touch $( cat /etc/supervisord.conf  | grep logfile= | grep "\.log" | sed s/.*logfile=// )

# Then run supervisord
/usr/bin/supervisord

Is there any good idea?

[timmyomahony]

+1

supervisord doesn't start on boot for Arch Linux as the /var/log/supervisord.log doesn't exist when booting.

[rb2nem]

Here are more details about why this would be useful in my case.
I use supervisord in a docker to run some software for which I want to persist the logs, so people are mounting a volume. But with supervisord not creating the needed subdirectories, it requires users to make the directories on the host and set owner and permission themselves. And it is not possible to put it in the command directive, as supervisord complains and stops even before looking at the command.

@mnaberez why sticking to your refusal? I have seen more solid arguments in favor of this than in your rejection.... (but I may be missing something, so more explanations welcome ;-)

[aujang]

This continues to be a problem, specifically for users deploying supervisor within Docker in an ElasticBeanstalk environment. If you specify logging in Dockerrun, the service wipes the specified directories during deployment, deleting anything created while Docker is spinning up. This in turn breaks supervisor, because it expects files where none exist. If supervisor simply created directories and logs when none existed, it would completely solve the issue. As it stands, I had to disable discrete log files (and use syslog instead) in order to get my app to deploy.

[Niyojan]

same here, I used the following workaround :
docker run -v /mnt/my_logs/celery:/var/log/celery -v /mnt/my_logs/supervisor:/var/log/supervisor -v /mnt/my_logs/supervisor:/var/log/supervisor -v /mnt/my_logs/nginx:/var/log/nginx -v /mnt/my_media:/home/project/media -p 80:80 -p 5555:5555 -d my_project

[adamreisnz]

I am also running into this now. I have a Raspberry Pi which has /var/log mounted on tmpfs to prevent writes to the SD card. After a reboot of the device, the /var/log/supervisor folder is gone and supervisor won't start. Auto creating this directory would be a much welcome feature.

@mnaberez would you mind reconsidering? There have been offers of PR's for this feature as well.

For context, I've spent about 4 hours debugging this issue today, and could not find any useful information or error messages informing me that a missing log directory causes supervisor to not start at all. That should really not be happening, and I don't know of any other software that doesn't know how to create it's own log folders, let alone refuses to run if they don't exist 😢

[corck]

:+1: I just ran into this as well.

[adamatan]

+1 👍 Would be very helpful for me as well.

[Nidhoggur1993]

A kindly remind here.
If supervisor was installed by default and do not change it's log path out of /tmp/supervisord.log, in certain system like CentOS 7.3 or higher, the systemd-tmpfiles-clean.timer will do a cron to delete everything under /tmp everyday.
We have suffer this a lot...

[mavci]

I just edited /etc/supervisor/supervisord.conf file and changed these lines;

logfile=/var/log/supervisor/supervisord.log to logfile=/var/log/supervisord.log
childlogdir=/var/log/supervisor to childlogdir=/var/log/

and fixed for me. And I was the excatly the same situation with @adamreisnz :)

[adililhan]

docker wasn't around when this issue created. so, we use supervisord in docker nowadays. thanks it is nice tool.

however, if we want to redirect the outputs then we have to add new line (and it creates new layer) to dockerfile such as mkdir /var/log/supervisord/"

if supervisord creates the directory then it would be good. thanks anyway.

[Destroy666x]

I don't know of any other software that doesn't know how to create it's own log folders, let alone refuses to run if they don't exist

Bump

[derekargueta]

+1

[mikelibg]

@mnaberez First of all, supervisor is a great tool. Due to the fact that supervisor is the first one to run (before the log directory can be created by the program it is starting) it is intuitive that it can create the log directory by itself, if it does not exists yet. Is there any other logic explanation not to add this option besides the comment "...the config file is already quite verbose..."?

[mnaberez]

Due to the fact that supervisor is the first one to run (before the log directory can be created by the program it is starting)

If you are starting myprogram directly:

command = myprogram

You could instead start it with something like:

command = bash -c 'mkdir /foo && exec myprogram'

[injaon]

:+1: +1 to this request

[FAKERINHEART]

1063

[await]

+1 would like to to use program name in the log path. it's not completely obvious why supervisor doesn't start when this occurs

[Dayjo]

Also getting this same issue recently. Was quite suprised it didn't handle this internally.

[rb2nem]

@mnaberez You suggest to put it in command, but this does not work for supervisord's logs, as I put it in a comment in this thread:

And it is not possible to put it in the command directive, as supervisord complains and 
stops even before looking at the command

[mnaberez]

Can you create the directories before starting supervisord, e.g. in the init script or equivalent that launches supervisord? Why not?

[adamreisnz]

From the general sentiment in this issue, it would appear to me that most people expect a piece of software that requires certain directories to work, to be able to create those directories for itself.

It is also quite apparent that the majority of software out there does in fact do that, so it's fair that people have come to expect that.

Moreover, from reading the feedback in this thread, it appears that the way this error is handled is unclear and confusing to people, and that a lot of them don't understand why supervisor won't work.

I therefore think it's not a question of whether we can or can't have our initialisation scripts create the directory -- we can, and in our case we have -- but a question of why Supervisor can't simply do it itself.

There are 26 participants in this thread alone, that means there's at least 10x as many people who don't bother to comment here that are feeling the same way about this issue. If there's such a strong desire for what is at first glance a pretty trivial feature to ask for, then why not just implement it or have someone submit a PR?

[adamreisnz]

I would also like to add that for a problem this big, e.g. the whole application not starting if a directory that needs to be manually created doesn't exist, there's surprising little documentation around this fact.

http://supervisord.org/logging.html
http://supervisord.org/faq.html
http://supervisord.org/installing.html

Neither of these pages mention anything about needing to create log directories manually.

[mnaberez]

From the general sentiment in this issue, it would appear to me that most people expect a piece of software that requires certain directories to work, to be able to create those directories for itself.

It is also quite apparent that the majority of software out there does in fact do that, so it's fair that people have come to expect that.

I can't speak for the other maintainers, but I don't have experience with programs that do this. It might be helpful if someone named specific daemons in common use that have this behavior. This may be why I don't immediately recognize why this is needed or desirable. For all the other daemons that I run, I create the needed directories before I start them. You have to do work before starting the daemon anyway, e.g. you have to make a config file. Why can't the directory be created then?

Moreover, from reading the feedback in this thread, it appears that the way this error is handled is unclear and confusing to people, and that a lot of them don't understand why supervisor won't work.

This is the actual error message:

Error: The directory named as part of the path /nonexistant/cat/stdout.log does not exist in section 'program:cat' (file: 'supervisord.conf')
For help, use /usr/local/bin/supervisord -h

If this is unclear, how can it be reworded?

[mnaberez]

If there's such a strong desire for what is at first glance a pretty trivial feature to ask for, then why not just implement it or have someone submit a PR?

• The current behavior prevents configuration mistakes. Currently, if the system has a directory /var/log/ but you accidentally enter /var/logs/foo.txt in the config file, supervisord will halt with the error above. I think that's better than ending up with both /var/log/ and /var/logs/ and maybe not noticing that for a long time.

• What does automatically creating the directory mean, exactly? Does it mean mkdir or mkdir -p? What should the chown and chmod of the automatically created directories be? Currently, the user sets up all these things outside of supervisord with the shell. Does the config file need to have a bunch of options to specify all of them now? If other programs have this behavior, can someone please give specific examples?

• Is it just logs or should this apply to other paths as well, e.g. pidfiles? Why or why not? If not, wouldn't it be better to be consistent and not create any directory than to create some and not others?

[mnaberez]

I would also like to add that for a problem this big, e.g. the whole application not starting if a directory that needs to be manually created doesn't exist, there's surprising little documentation around this fact.

Fail-fast is a common approach when daemons and other server software are started with a bad configuration. There's are no documentation stating which configuration options are allowed to be invalid and which aren't because none of them are.

In this case, the user has specified an optional logfile but supervisord can't create that logfile because the directory specified by the user doesn't exist. The existing implementation wasn't written to automatically create directories so supervisord outputs a descriptive error message and refuses to start until the configuration is fixed. It does that not only with this condition but with as many configuration issues as possible.

The alternative would be to somehow enable supervisord to partially start with a known broken configuration, perhaps trying to ignore only those options that are invalid. This would make its operation unpredictable and delay fixing the problems.

[rb2nem]

@mnaberez My primary reason to ask this feature was that I used supervisord in docker. Docker containers don't have an init system and are designed to run only one command. When people need to run multiple processes in one container, suprevisord is a great fit, except for this problem.
When users want to persist logs of the commands started by supervisord, they mount a volume where supervisord saves the app's logs. Due to this problem, they have to know the hierarchy of directories to create.

Of course there's alway a justification possible to refuse to implement it ("if docker's for one command, you don't use it as meant if you need supervisord", or "people should know what happens in the image") , but why not just ease the life of your users by integrating it?

I have tried to find a reasonable solution to create directories where supervisord sends the logs, but none was working (add creation of the logfile in command="mkdir blah && command") and easy and maintainable (parse the log file to create the directories before starting supervisord).

[mnaberez]

Of course there's alway a justification possible to refuse to implement it ("if docker's for one command, you don't use it as meant if you need supervisord", or "people should know what happens in the image") , but why not just ease the life of your users by integrating it?

I've already reopened this issue after it being closed for years, which should be an obvious sign that we're openly reconsidering it.

I wrote detailed responses above (here and here). Those need to be answered to make progress on this issue. Those questions are open to anyone asking for this change.

With regards to "why not ease the life of your users?" and similar comments, I obviously spent a lot of time replying to comments here and trying to understand this issue better. I do care about users which is why I have spent nearly ten years maintaining this software, fixing issues, and responding to the mailing list. When you send comments like that to open source maintainers, it makes it an already thankless task miserable and leads to burnout.

Edit: Reworded; I confused similar posts from two different users.

[adamreisnz]

In this case, the user has specified an optional logfile but supervisord can't create that logfile because the directory specified by the user doesn't exist

So that's the thing, in my case, supervisord is installed by another piece of software (Dataplicity), to monitor/start their application. I don't configure supervisord myself, so I don't specify where it should log or if it should log at all.

All I know is that it gave me a lot of grief because of a missing folder, and I had to modify our startup scripts to create the folder manually after every reboot (because it's on tmpfs).

So who's responsible in this case? You say supervisord doesn't log by default, so Dataplicity uses supervisord and (presumably) configures it to use a log file. But then Dataplicity doesn't know anything about the state of the machine it's running on, so they can't really know what log folders to create either. The user is stuck in the middle because he doesn't know he needs to create a log folder until after he encounters the error after a reboot. So it's a bit of a chicken and egg situation.

For all the other daemons that I run, I create the needed directories before I start them. You have to do work before starting the daemon anyway, e.g. you have to make a config file. Why can't the directory be created then?

I get your point and now that I understand that the log file is not a default setting, I think I understand why you don't bother creating the folder. After all if the user is specifying the folder, then they should make sure it exists. Completely agree.

So I think the problem might come in part from the fact that other software uses supervisord to start and manage their apps, and they maybe make assumptions about the log configuration and folders without being clear about it to their end users.

I still feel the simplest solution would be if supervisord would just create the folder, using sensible defaults and best practices for how this is done. I don't think the perceived issue of configuration error and then ending up with two different log folders is a biggie. That sounds like a very odd scenario with minimal impact if it does eventuate, and would be a clear user error.

[felipou]

@mnaberez Thanks a lot for re-considering this and for all the replies. I've been reviewing my original comments (from so many years ago) to better understand why I created this issue in the first place.

I believe my original wish was that supervisor should be able to create the directory of the log files of the programs it manages, not the directory of its own logs. And this is still something that I have to do some workarounds nowadays. Perhaps because my use of supervisor is somewhat different from expected: I update it constantly, adding and removing programs multiple times per minute.

I share many of your concerns regarding the creation of supervisor's own log directories. And come to think of it, I solved both problems the same way: for supervisor, I create the log directories in the init file (systemd nowadays), and for the programs, I use a kind of "init script" (bash) for then, and this script is responsible for creating the log directories if they don't exist (and "execs" the program after).

All things considered, I'm not really sure if this is a feature that is really required, but perhaps we should just add a little more documentation about this, since it seems to be a pain for many people.

[Dayjo]

In response to everyone's comments here. I would say additionally that absolutely if this is something that is difficult to achieve, simply improving the error messages and actually documenting this in various places and in the setup process, it will simplify the process for users.

The main reason I came here was that I spent a good hour re-installing, changing settings, re-reading documentation to try and work out what the problem was and why it didn't 'just work' after following the install instructions. If in those instructions it had said "If you want to customise the path of the logs, you will need to make sure that the directory first exists and has appropriate permissions", then I wouldn't have gotten stumped at all.

Folder permissions is a common issue so I don't think having a bit of documentation on this as part of the setup would negatively impact that process.

[rb2nem]

With regards to "why not ease the life of your users?" and similar comments, I obviously spent a lot of time replying to comments here and trying to understand this issue better. I do care about users which is why I have spent nearly ten years maintaining this software, fixing issues, and responding to the mailing list. When you send comments like that to open source maintainers, it makes it an already thankless task miserable and leads to burnout.

@mnaberez I'm happy you reconsider this request, and I appreciate your dedication to your project. My comments were always meant to be respectful and I feel my question is justified for a request with users' recurring interest since 2012.

Regarding the request itself, the problem I experienced is that supervisord complains about the inexisting log dir before starting the command. The easiest would be to let the user create the directories either by combining it in the command ( like mkdir /var/log/supervisord && mycommand), but that might have some impact on the internals, or by having a PreCommand directive for which the presence of the log dir is not required.

[mnaberez]

My comments were always meant to be respectful and I feel my question is justified for a request with users' recurring interest since 2012.

Negative comments like your "why not just ease the life of your users?" are never respectful or justified.

No maintainer of this project is intentionally trying to make life difficult for users. We are users of the software ourselves. There is no incentive to make life more difficult for users. It only creates more support requests for us to deal with. There is almost always another reason why a particular thing doesn't get done in a particular way other than the maintainers don't care about users. There were sound reasons for the existing behavior, which were explained in the thread already.

I'm not sure what should be done on this issue, but I agree with @adamreisnz and @Dayjo that updating the documentation to clarify the current behavior is a good first step.

[rb2nem]

Negative comments like your "why not just ease the life of your users?" are never respectful or justified.

As my last comment on this issue, I encourage you to read http://www.dealingwithdisrespect.com . I feel my comment is acceptable, and far from disagreeable.

[mnaberez]

So that's the thing, in my case, supervisord is installed by another piece of software (Dataplicity), to monitor/start their application. I don't configure supervisord myself, so I don't specify where it should log or if it should log at all.

All I know is that it gave me a lot of grief because of a missing folder, and I had to modify our startup scripts to create the folder manually after every reboot (because it's on tmpfs).

So who's responsible in this case?

I'm not familiar with Dataplicity but you may want to report this issue to Dataplicity as well.

Supervisor itself does not handle installation. It's just a Python package. It does not provide official init scripts or even a config file, it only provides a command echo_supervisord_conf that prints a sample config file to stdout. The person repackaging or installing Supervisor is responsible for creating a configuration compatible with the target system.

If I understand correctly, Dataplicity installed Supervisor for you but the way it was installed would not survive a reboot on your system. The log directories were created initially by Dataplicity but did not persist after reboot because they were on tmpfs. If Dataplicity is creating the init script that starts Supervisor, they may be able to change the init script so it checks for the directories first and creates them if needed (the same solution you did). They will also probably want to track this ticket because even if a change is made to Supervisor or a new feature is added, Dataplicity may not pick up the new release automatically if they have pinned to a specific version.

[Radoslaw-K]

@adamreisnz @mnaberez

Our Dataplicity installation agent downloads and installs supervisor. When it is downloaded it has all the settings as configured upstream, we don't change anything. The default directory that supervisor uses for logging can be seen in the file below. This is what you would probably want to adjust if you'd like your logs to be stored elsewhere.

cat /etc/supervisor/supervisord.conf

; supervisor config file

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0700                       ; sockef file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)

--- ... shortened output ... ---

As for Dataplicity specific logs, below is our unit file for supervisor that specifies where we keep them.

cat /etc/supervisor/conf.d/tuxtunnel.conf

--- ... shortened output ... ---

stdout_logfile=/var/log/dataplicity.log
stderr_logfile=/var/log/dataplicity.log

Alternatively if you'd like to keep the log files where they are and just make sure supervisor works with tmpfs, here's something you can add to your /etc/rc.local file:

if [ ! -d /var/log/supervisor ]; then
        mkdir /var/log/supervisor;
        touch /var/log/supervisor/supervisord.log
        chown -R dataplicity:dataplicity /var/log/supervisor;
        service supervisor restart
fi

Hope this helps,
Radoslav from Dataplicity

[adamreisnz]

@mnaberez @Radoslaw-K

Our Dataplicity installation agent downloads and installs supervisor. When it is downloaded it has all the settings as configured upstream, we don't change anything. The default directory that supervisor uses for logging can be seen in the file below

So these are the default settings of supervisor? From what I understood from what @mnaberez wrote;

In this case, the user has specified an optional logfile but supervisord can't create that logfile because the directory specified by the user doesn't exist.

I thought Supervisor doesn't create any logs or specify any log directories by default.

So either Dataplicity must be specifying those directories, or supervisor must have them as the default. If it's the former, it probably shouldn't be done. If it's the latter, then I think the defaults might need to be changed to not have any logging at all, or to auto create the default directory.

It doesn't make much sense imo to have a default directory but then be unable to start the application because the directory doesn't exist.

[Radoslaw-K]

@adamreisnz

These are the default settings of supervisor, correct.

You are right that:

_"It doesn't make much sense imo to have a default directory but then be unable to start the application because the directory doesn't exist."_

The emonpi guys who back in the past were installing dataplicity on their system had the same problem and I think it was reported to the developers of Supervisor but their response was that it's user's responsibility to create that directory if it doesn't exist.

Best,
Radoslav from Dataplicity

[mnaberez]

These are the default settings of supervisor, correct.

This is not correct because this project (Supervisor) does not supply init scripts or config files at all (see more above). We only provide a command, echo_supervisord_conf, that writes a sample config to stdout. I looked at your configuration above, and it does not match what is output by echo_supervisord_conf (e.g. the directories /var/run and /var/log do not appear anywhere in that sample output). You are using a configuration that was created and written to disk by someone else.

Our Dataplicity installation agent downloads and installs supervisor. When it is downloaded it has all the settings as configured upstream, we don't change anything.

The Supervisor project only provides Python packages that are published to PyPI. Our packages ares usually installed with pip. Our packages do not contain init scripts or config files. It sounds like you are installing the package from another source, e.g. maybe you installed it with a command like apt or rpm. Those are third party packages that are created by others who aren't involved with the main Supervisor project. We don't know much about these packages and we can't make changes to them. Any init scripts or config files come from them. I suggest contacting the maintainers of the third party package you are installing. They are the only ones that would be able to modify their default init script to create the directories.

[barfle]

Hi there,
I'm just writing in to provide a little backfill here and to clarify one or two things since Dataplicity has been referenced more than once here :-)

First, thanks everyone for the renewed interest in this issue, and thanks to the supervisor devs in particular for devoting so much time and effort to the supervisor daemon that we rely on to run Dataplicity Agent on devices.

The precise issue for us is already pretty well described above. The behaviour in respect of creating the log directory if it doesn’t exist is not without precedent. In particular, nginx has similar behaviour (not creating the directory), as do a number of other mainstream services. As it happens, @adamreisnz has produced a pretty good list in the emonpi repos where every service that has this issue has had to be fudged in precisely the same way.

If I were to make a pretty wide observation, what is new now as opposed to five years ago is that logging to volatile storage is becoming increasingly common. More precisely, with the en masse arrival of IoT devices (such as Raspberry Pi) and systems such as Docker and Amazon EBS, people are increasingly relying on tmpfs for a fast logging system that doesn’t wear out flash storage. This is not a distribution level issue - Ubuntu, Arch and Slackware are all similarly affected.

In the medium term, I take the view that the use of volatile storage for logging has become sufficiently common that treating its use as a downstream, deployment related issue is no longer appropriate. If I am right, this will require a small change in every service that presently logs directly to a subdirectory of the system log directory.

In the case of Dataplicity, it affects us because SBCs including Raspberry Pi commonly use tmpfs to store logs to prevent flash wear. And on most of these systems that means /var/log is a fresh file system with no directories in it the moment it boots up. Which then means that supervisor breaks at startup because it can't access a directory that it's expecting.

For clarity:
1) dataplicity agent is the device-side daemon that's running on user devices, commonly a Raspberry Pi, to connect the device to the dataplicity service.
2) The agent uses supervisor to ensure we're running as a non-privileged user on the system, and to handle things like auto-restart of agents etc.
3) The Dataplicity installer uses apt-get on systems where it is available, which means yes, the installation process for supervisor derives from a combination of the software source (here) and the precise installation instructions baked into the apt package by the distribution people at Debian/Ubuntu etc.

There are three places we (Dataplicity) could put something in to create this directory on boot, but the only one (for us) without significant drawbacks is if supervisor was to create the expected directory itself on startup.

Tier 3 (at the dataplicity installer/agent level)

• We can put a fudge in the dataplicity installer to create the directory. This won't work because the directory is tmpfs and won't survive reboot.
• We can put a different fudge in the dataplicity installer to modify rc.local or in the supervisor start script to create this directory at boot every time. In our testing, platforms use of rc.local is inconsistent, meaning that we cannot rely on this for every distribution including those where people roll their own builds (pretty common for our users). The latter is plausible, but also ends up platform dependent and clashes with the distribution standard packages so distribution updates etc will be constantly fighting with our little fudge. And Dataplicity aside, this could hardly be expected behaviour for _every_ installation in the same situation.
• We can modify the default supervisor config. However given that many things can use supervisor on the same system, assuming dataplicity is the only thing that “operates” supervisor on a machine is a pretty mighty assumption: people could get stroppy with us if we inadvertently clobber other services running on their machine.

Bottom line is that it ends up a bit of a fudge that after we do "apt-get install supervisor" we then have to do "mkdir /var/log/supervisor/" just to make it work.

Tier 2 - Distribution

• We can ask every distributor to change the installation process to ensure that particular directories are created every time a system boots. As it happens, the apt-get package does already create the initial directory because it matches the template config files they supply. But it doesn't account for the fact that so many systems operate on tmpfs, which means it doesn't survive the first reboot.

Tier 1 - Supervisor itself.

• I understand that this does potentially add another layer here. But other than creating a directory I don't see any downside (of course I might be missing something).
• The benefit is that this behaviour will trickle down and mean that every distribution supporting a temporary file system (or equivalent), will eventually get this behaviour, which means we don't have to persuade every distribution to put this fudge in place.

I hope that helps.

Elliot (Dataplicity)

[mnaberez]

Bottom line is that it ends up a bit of a fudge that after we do "apt-get install supervisor" we then have to do "mkdir /var/log/supervisor/" just to make it work.

Please report these same details to the maintainers of the package that you are installing. They may be able to help.

[barfle]

Hi there

We have a few options, but this particular one is likely to take me into retirement: making this change for all the platforms we support amounts to every distribution from red hat to arch via Ubuntu and Debian in between. And it also misses the point that out of the box supervisor doesn’t work on systems with a volatile log store.

Is there something I am missing here that presents either a particular challenge or an argument against doing so? Is it likely to cause maintenance or deployment issues of which I am unaware?

Best
Elliot

[mnaberez]

Is there something I am missing here

Yes: The third party packages that you are installing often lag considerably behind the main Supervisor version, often by years. Even if a change is made to Supervisor (which is being considered here, this issue is still open), it may not help you for a very long time if you wait for those third parties to package the new version. If you report the issue to the maintainers of the package you are installing, they may be willing to patch the init scripts of their current packages.

[barfle]

Now that I can totally understand.

My concern, however, is always for what is the best solution, followed somewhat thereafter by what is the quickest.

So what I really see here is two problems. One is resolving this in the best way possible, for which my personal suggestion is to add this in all the upstream packages (I am still open to better solutions too).

The second is to then push that fix downstream. But that is a much easier issue than trying to hack in a fudge for every distribution. And it is something that we would be keen to support through PRs to the main distributors if and when that was required.

[mnaberez]

One is resolving this in the best way possible, for which my personal suggestion is to add this in all the upstream packages

We'd be interested in your (or anyone's) feedback on the questions above. There was a pull request that silently and automatically created any missing directory. I don't think that's the right thing to do for the reason mentioned above, but I'm not sure what a new option should look like.

[barfle]

@mnaberez I see your question.

The engineering side of me says everything should be absolutely consistent. But the practical side of me says that this is an issue related primarily to /var/log existing on tmpfs and I haven’t heard complaints elsewhere. On balance, I would suggest the solution could therefore be kept narrow in focus until such time as it is deemed appropriate to widen the scope.

The narrowest option that I can think of is indeed an implicit mkdir. My reasoning is that the issues we are seeing relate to the lack of /var/log/supervisor/ on boot rather than a lack of /var/log, and that in practice I have rather rarely come across a system that does not have a top level logging directory dictated by the distributor.

I will give a more thought out response tomorrow.

[willmcgugan]

Rather than implicitly creating directories, could there be an explicit section in the conf for ensuring directories exist prior to forking the process?

Something along these lines:

[directory:log]
path=/foo/bar/baz
owner=www-data
create=true
recursive=true

If you just specify a path, it will fail if that path doesn't exist (which could be useful in itself). But create=true declares you want to create it (if it doesn't exist), and recurse=true declares you want to create any parent directories.

I do agree that implicitly creating directories is probably inviting errors, and changing the behaviour now is probably going to annoy as many people as it pleases. But the above wouldn't change the current behaviour, while still giving developers the option of auto-creating log directories (for example).

[rzaharenkov]

In my opinion supervisor daemon itself should be super reliable and start even if there is any issue with configuration of particular programs.

In my case I am deploying my application in 2 steps:

1. Bootstrap server using chef (install packages and create configuration files).
2. Deploy application using capistrano (fetches application from github and restart particular services using supervisorctl restart my_group:*.

I run 1st step first time when I am bootstrapping new machine or when I do any changes to software or system configuration. 2nd step is running in order to release new version of application code.

So after very first running of chef I am ending up with broken supervisor (service is dead) because it can't find directory. Then run capistrano in order to deploy application 1st time (it creates all the required directories) and tries to restart services using supervisorctl but fails. So I need to run chef 1 more time which is inconvenience.

I do believe if something is wrong with single program configuration then supervisor daemon itself should start but program should be in failed state.

[shankar-moeng]

+1

[ghost]

If I need some wrapper scripts for the supervisor itself, it kind of defeats the very purpose of the program. I think it should automatically create the whole directory branch by default. There could be an option to _disable_ this behavior, not the other way around.

[ghost]

There are actually two major problems with the behavior of the supervisor right now:

1. Not creating the directory branch automatically.
2. Shutting down completely just because some log files could not have been created. This is even worse than the first problem. (This is addressed in #1143, which I think is _not_ a duplicate.)

[ghost]

@mnaberez wrote:

The current behavior prevents configuration mistakes. Currently, if the system has a directory /var/log/ but you accidentally enter /var/logs/foo.txt in the config file, supervisord will halt with the error above. I think that's better than ending up with both /var/log/ and /var/logs/ and maybe not noticing that for a long time.

The _hypothetical_ problem of accidentally creating log files in a slightly wrong location seems like a much lesser trouble to me than the very real and frequent (it should be) problem of bringing down the entire server just because the administrator forgot to create the folder for a log file.

And what if you mix up the name of the file and not the folder? What if you create a log in a wrong location that nevertheless exists? What if you specify a completely nonsensical name in an existing folder? After all, you know, garbage in, garbage out.

Another aspect of this question is that one can conceptually see full file paths as fully qualified file names. That is, one can think of the file system as of a collection of files with names that happen to have slashes in them and not as of a tree of folders. Yet this collection can also be viewed as a tree of folders in another representation. Which brings us to:

What does automatically creating the directory mean, exactly? Does it mean mkdir or mkdir -p?

mkdir -p, of course. Purely from a practical standpoint of the user not having to deal with some wrapper scripts or dependent command sequences.

What should the chown and chmod of the automatically created directories be?

How about they would correspond to the chown and chmod of the created log file? You create the log file itself, don't you? Why not also create the directory?

Currently, the user sets up all these things outside of supervisord with the shell.

...which they don't do at first, but they find out about this problem after their server is brought down unexpectedly.

If supervisord requires a wrapper script or dependent command sequences to deal with these things, it greatly reduces its usefulness as an application, as what the supervisor does is not particularly rocket science, so when you have to fix these kinds of problems yourself, you start thinking about the necessity of using the supervisord at all. Also makes you think about the general approach of the authors and what other traps they might have prepared for you.

Is it just logs or should this apply to other paths as well, e.g. pidfiles? Why or why not?

Well, this sounds like it is out of scope for the current issue.

If not, wouldn't it be better to be consistent and not create any directory than to create some and not others?

No. Different situations require different handling. Logs are different from e.g. critical files whose consistency is vital to the system. Logs are important, but aren't critical or vital. ~If some logs cannot be created, you could issue a warning, and the user would later fix them. It is much better than to shutdown the entire system just because one log file was not created.~ Edit: I lost my chain of thought here. The argument was about creating the folder structure for log files automatically, so the problem from which we are protecting ourselves here is that the name of the folder for some log file may be wrong. Thus, given that log files and their locations are usually not critical or vital, I would gladly exchange the risk of placing a log file in a wrong folder with the risk of unexpected outages of the server. For the (supposedly rare) folk for whom the location of the log files is mission-critical I would make an opt-in possibility to ensure that the folder for a log file exists.

[mnaberez]

Logs are different from e.g. critical files whose consistency is vital to the system. Logs are important, but aren't critical or vital. If some logs cannot be created, you could issue a warning, and the user would later fix them. It is much better than to shutdown the entire system just because one log file was not created.

This opinion is application-specific. We will not guess about how important log files might be for someone else's application. If the user provides a supervisord.conf file that the supervisord daemon cannot fulfill, it will fail-fast so the problem is corrected sooner than later.

[ghost]

@mnaberez

Logs are different from e.g. critical files whose consistency is vital to the system. Logs are important, but aren't critical or vital. ...

This opinion is application-specific.

Well, sure, I am certain there are people out there who have designed a system in which log files and their locations are vital for the functioning of that system, and they would prefer an outage of the entire server to a possibly wrong name of a folder for a single log file. But are those the majority of use cases? How typical is that situation? Do you generally prefer to optimize for the most common use case or the most uncommon use case?

We will not guess about how important log files might be for someone else's application.

But you did guess that an outage of the entire system is less critical than a possibility that the name of the folder for a single log file might be wrong. Essentially, if we have to choose between these two evils (service outage vs. possibly wrong folder name for a log file), some guessing seems unavoidable.

[mnaberez]

But you did guess that an outage of the entire system is less critical than a possibility that the name of the folder for a single log file might be wrong. Essentially, if we have to choose between these two evils (service outage vs. possibly wrong folder name for a log file), some guessing seems unavoidable.

The supervisord daemon doesn't accept an invalid configuration file. If you change the configuration file to one it cannot fulfill (including log files it can't write), it will refuse to (re)start and will give you an error message why. People who configure other daemons regularly should find this behavior typical.

While it's entirely possible that some option to create log directories could be added at some point as discussed here, it's unlikely that the fail-fast behavior will change. It is unlikely that we are going to add special cases to ignore only some configuration mistakes as you are requesting. It would mean that users could no longer have confidence that a running supervisord instance is in compliance with its configuration file.

[aujang]

As I mentioned in my original comment, there are circumstances where the directory literally cannot pre-exist (namely, docker within Elastic Beanstalk) because the deploying service wipes the master log directory during any new deployment. This behavior causes supervisord to fail every time, without any fault on the user's behalf. I don't find the counter-arguments against automatic directory creation terribly compelling. For example:

The current behavior prevents configuration mistakes. Currently, if the system has a directory /var/log/ but you accidentally enter /var/logs/foo.txt in the config file, supervisord will halt with the error above. I think that's better than ending up with both /var/log/ and /var/logs/ and maybe not noticing that for a long time.

OK, let's assume there is a config mistake. What is the ultimate consequence of said mistake? Completely valid logs are generated and stored in the wrong place. When someone goes looking for the logs, they don't find them where they expected, so they go to the config file, see that the log destination has been incorrectly configured... and correct it. Meanwhile, they have all of the previous logs, and nothing has broken in the interim. If something did break (let's say there was an aggregator running and it wasn't collecting the logs because they weren't in the right place), then only one of two things could occur:

1. The aggregating service fails with an error that the expected logs were not found (which would prompt an investigation that would lead back to the config file), or:
2. The aggregating service does not include the supervisord logs, in which case anyone examining the aggregator notes the lack of logs, and then goes to find them manually, discovering the discrepancy. Meanwhile, all the services continue to run, and logs have been generated, so they can be manually uploaded into aggregator without any real consequence.

I don't see the pidfiles argument as analogous because logs aren't mission-critical, but PIDs are. You can't run processes at all without generating a PID, but you can run any application without generating logs.

I don't think anyone here is arguing against the general fail-fast behavior here: instead, we're pointing out that the current fail-fast behavior for log files is a bad design pattern. But it strikes me that there's a simple solution: include a configuration parameter called "Automatically create log directory if absent?" and, when set to true, have it execute mkdir -p (as @dmitry-akimov-dXgjvQjNiMuVYecJ suggests) with the same permissions/ownership as the created logfile, and move on. You can have the config param default to "false" so that the error procs the first time a non-existent log directory is passed, so the user is forced to take a closer look at log configuration before making a decision.

[mnaberez]

I don't find the counter-arguments against automatic directory creation terribly compelling.

Most of the discussion of the last couple of days, including my own replies, has been a rehash of things already stated in this thread. I don't think it's worth going around again because I do not think new points are being made.

But it strikes me that there's a simple solution: include a configuration parameter called "Automatically create log directory if absent?"

This issue is currently open. I reopened it because I think it is worth reconsidering. It is possible that we may add some sort of optional feature to create directories. I stated this above also. I am currently working on other issues and I am not able to work on this particular issue right now. The other committers on this project may be willing to work on it, or it can stay open.

[sdanbury]

Haven't seen this for a couple of years, but it has randomly started appearing again.

$ sudo supervisord -c /etc/supervisor/supervisord.conf
Error: The directory named as part of the path /var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log) does not exist
For help, use /usr/local/bin/supervisord -h

I have made sure the /var/log/supervisor/supervisord.log file exists and have even given permissions of 777 to test, and it is still happening. Any ideas?

[mnaberez]

@sdanbury This ticket is a request to add an option to create the directory of log files. Your issue is a different one since you already created the directory. I've opened #1229 for it.

[danmihai89]

+1

[bbankowski]

+1

[renaudcerrato]

Such a stupid bug and yet unfixed. Our /var/log directory is a tmpfs, so totally wiped between reboots.

[beasteers]

I really don't understand why there is so much pushback on this idea. Tbh I think this behavior should be available and enabled by default for logs (but I understand the concern about changing default behavior). I don't see any argument about having it available and disabled by default however.

If I ever get a minute I might look into submitting a pull request, but in case someone else has a desire to figure it out:

Here's where they throw an error if the directory doesn't exist
https://github.com/Supervisor/supervisor/blob/b6b762d809e8c5966208b0836a9403294c3294c4/supervisor/datatypes.py#L94-L105
https://github.com/Supervisor/supervisor/blob/b6b762d809e8c5966208b0836a9403294c3294c4/supervisor/datatypes.py#L342-L351

So a start of a fix could be:

# https://github.com/Supervisor/supervisor/blob/3e7c082a71fef3860ecf06727edd091ddb3cc282/supervisor/options.py#L635
self.create_log_directories = boolean(
    get('supervisord', 'create_log_directories', 'true')) # +++
...
section.logfile = existing_dirpath(
    get('logfile', 'supervisord.log'), 
    create=self.create_log_directories) # +-

# https://github.com/Supervisor/supervisor/blob/b6b762d809e8c5966208b0836a9403294c3294c4/supervisor/datatypes.py#L342
def existing_dirpath(v, create=False):
    nv = os.path.expanduser(v)
    dir = os.path.dirname(nv)
    if not dir:
        # relative pathname with no directory component
        return nv
    if create: # +++
        os.makedirs(dir, exist_ok=True)
    if os.path.isdir(dir):
        return nv
    raise ValueError('The directory named as part of the path %s '
                     'does not exist' % v)

# https://github.com/Supervisor/supervisor/blob/b6b762d809e8c5966208b0836a9403294c3294c4/supervisor/datatypes.py#L94
def logfile_name(val, create=False):
    coerced = val.lower() if hasattr(val, 'lower') else val

    if coerced in LOGFILE_NONES:
        return None
    elif coerced in LOGFILE_AUTOS:
        return Automatic
    return existing_dirpath(val, create=create) # +-

# https://github.com/Supervisor/supervisor/blob/3e7c082a71fef3860ecf06727edd091ddb3cc282/supervisor/options.py#L970
lf_val = logfile_name(lf_val, create=self.create_log_directories) # +-

# https://github.com/Supervisor/supervisor/blob/3e7c082a71fef3860ecf06727edd091ddb3cc282/supervisor/options.py#L426
self.add("logfile", "supervisord.logfile", "l:", "logfile=",
         lambda v: existing_dirpath(v, create=self.create_log_directories),
         default="supervisord.log")

[xiandong79]

I have created the log path myself
but supervisord do not creat the log file

~~ help

[mnaberez]

Tbh I think this behavior should be available and enabled by default for logs (but I understand the concern about changing default behavior). I don't see any argument about having it available and disabled by default however.

We probably would not change the default (at least not without bumping the major version). This issue has been open for a long time and there have been a lot of comments asking for it, so at this point we would consider adding it as an option (this was stated above) but nobody is currently working on it. On this issue tracker, issues that are rejected are closed. Issues that are still open are considered unresolved.

[bob-smetana]

This thread has been silent for a minute, but as the issue is still open I assume comments are still welcome.

I agree with @mnaberez and others who have said that changing default behavior is not the thing to do. And the fail-fast approach is a good one - highly unambiguous - but situations demonstrably exist where it is a problem for non-existent directories specifically, as there is occasionally no way around it. Docker environments using supervisor have been mentioned as an example, and that situation is what brought me here today.

The suggestion @willmcgugan made is a good one in my opinion; it doesn't change any existing or default behaviors so is compatible with any config files that may exist today. Not sure whether I think there needs to be a recursive option or if it should simply assume mkdir -p if it's creating directories. I'm leaning toward the latter.
Ownership could be set to the user:group the logging process runs as, whether that be supervisor itself or a child. I think documenting that created directories will have 775 or 755 permissions would be sufficient.

As for questions about whether supervisor should also then create directories for pids and the like - I think it could be thought of as a separate issue, although whatever decision is made here certainly could be applied elsewhere. Perhaps even it simply becomes a global option; will supervisor be creating directories where needed, or not?

All that to say - another late vote in favor of this feature.