Postgresapp: Postgres 9.5.1 Failed startup on OSX 10.11.3

[jonathansimmons]

I recently received a new work issued machine and while setting it up I noticed the postgres server wouldn't start.

The server log showed

FATAL:  postmaster became multithreaded during startup
HINT:  Set the LC_ALL environment variable to a valid locale.

Running locale in terminal I got this:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

Which for what it's worth is exactly what my old machine has as well. Regardless I tried setting the LC_ALL in my .bash_profile like so:

export LC_ALL="en_US.UTF-8"

after doing this my locale now looked like:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

This however did not fix the issue.

For the time being I've resolved the issue by grabbing Postgres 9.4.5.0 off my old machine and it works fine.

Let me know if you need any further information to track this down.

[cdonadeo]

After upgrading from OSX 10.10 to 10.11 I'm having the same issue. My locale looks like this:

➜  ~ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

So I export LC_ALL when starting new shells now:

➜  ~ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

But I don't think Postgres.app is starting a shell to get that environment variable so that's pretty pointless.

[tbussmann]

ok, here we see the error level of that specific message was actually changed between 9.4 and 9.5. It was newly introduced in Jan 2015 and backpatched up to 9.0 here after discussion and explanation as a safety net against a unhealthy condition here
Unfortunately, I don't yet fully understand the problem with the --enable-nls compile and which conditions need to be given for the error to occur (and thus could be changed to get a workaround). It seems the locale cannot be determined at startup, a method is called to determine this from the OS which starts a thread which in turn is unhealty for the postmaster startup.
Obviously this affects not all OSX users. So, to narrow it down, can you please give details on your OS version and the Language & Region settings in System Preferences?

[cdonadeo]

OS X version 10.11.3

➜  ~ uname -a
Darwin chris-work-laptop.local 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64

Attaching screenshot for my Region and Language settings - they should pretty standard for en-US.

Let me know if you need any additional information.

[jonathansimmons]

Mine are the exact same as @cdonadeo above.

[tbussmann]

this indeed looks rather straight forward - having an empty LC_ALL seem to be normal
I'm out of ideas here. Blindly experimented a bit with these functions in https://gist.github.com/tbussmann/a8bba87c2643b57ad29ba907ea1d3a55 which you could compile with gcc locale-test.c -o locale-test to me the output looks like

Environment LANG = "de_CH.UTF-8"
Environment LC_ALL = "(null)"
Request LC_ALL = "C"
Set LC_ALL to default = "de_CH.UTF-8"
Request LC_ALL = "de_CH.UTF-8"

To my understanding that sequence shows the problem would be triggered on my system as well, but this isn't the case.

Moreover you could check if en_US.UTF-8 is a valid locale at your system:

locale -a | grep en_US.UTF-8
ls -lah /usr/share/locale/en_US.UTF-8

but I doubt this all will lead to anything, unfortunately. Maybe the message' hint is misleading in respect to the locale at all and there is any other reason the postmaster became multithreaded (pthread_is_threaded_np was set) in your cases.

You mention PostgreSQL 9.4 is working well on your machine. Do you see the mentioned FATAL as a LOG message in the logs under 9.4?

Maybe @jakob has any further idea on this?

[cdonadeo]

Out of curiosity, I tried installing and using Postgres 9.5.2 from Homebrew instead, and ran into the same issue:

➜  Postgres postgres -D /usr/local/var/postgres
FATAL:  postmaster became multithreaded during startup
HINT:  Set the LC_ALL environment variable to a valid locale.

So it seems the issue isn't with anything specific to Postgres.app. Still no idea what the problem might be.

[davedash]

So I tried both hombrew and the postgres.app. I wonder if the LC_ALL hint is a red herring.

Since @jonathansimmons mentioned work computer and my coworkers had similar issues, I uninstalled the anti-virus software we use, Cylance and the problem went away.

[jonathansimmons]

@davedash did you by any chance try re-installing Cylance to see if it began failing again?

[davedash]

@jonathansimmons yes it did begin failing in the same manner, and everything was back to normal when Cylance was removed a second time.

[jakob]

Thanks for the update! Maybe I should update the docs to add a hint about disabling anti-virus and firewalls when issues come up. The most hard to debug issues are often related to security software that interferes.

[jonathansimmons]

I think we should mention it as a possible proponent.

I'm going to check with my office today about removing then readding Cylance to see if it resolves my issue as well.

It would be nice to isolate the root cause and see if a setting can be adjusted to avoid the need for complete removal.

I couldn't care less about antivirus but my employer has an auto deploy system so I don't have much say.

[jonathansimmons]

So I got around to reviewing this with my office. Un-installing Cylance did allow 9.5 to begin running.

Our IT department had this to say:

"The odd thing is that usually when Cylance stop a process from running we can see that in a list for the stopped processes for the machine. Even though Cylance apparently prevents Postgres from starting on your machine the Postgress process is not listed as a something that Cylance stopped."

I don't know if that should imply there is still an issue for Postgres app to fix or not but it was interesting none the less.

I'm functional at this point but I'll leave this open for @jakob to close once he has seen the update.

[davedash]

Our it team was able to configure cylance to leave postgres alone. Not sure
what they had to do.
On Fri, May 20, 2016 at 7:00 AM Jonathan Simmons [email protected]
wrote:

So I got around to reviewing this with my office. Un-installing Cylance
did allow 9.5 to begin running.

Our IT department had this to say:

"The odd thing is that usually when Cylance stop a process from running we
can see that in a list for the stopped processes for the machine. Even
though Cylance apparently prevents Postgres from starting on your machine
the Postgress process is not listed as a something that Cylance stopped."

I don't know if that should imply there is still an issue for Postgres app
to fix or not but it was interesting none the less.

I'm functional at this point but I'll leave this open for @jakob
https://github.com/jakob to close once he has seen the update.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/PostgresApp/PostgresApp/issues/317#issuecomment-220613379

[jonathansimmons]

@davedash would you be willing to ask them what they did?

[davedash]

Hi, so this is for the brew version of postgres:

He upgraded me to the latest Cylance Agent and Cylance is currently excluding the paths below:

/usr/local/Cellar/postgresql/9.5.3/bin/postgres 
/usr/local/Cellar/postgresql/9.5.3/lib/postgresql/ 
/Library/Caches/Homebrew/postgresql-9.5.3.el_capitan.bottle.tar.gz 
/usr/local/bin/postgres 
/usr/sbin/ 

I imagine you could exclude similar paths that the .app provides. Hope this helps.

[jakob]

OK, I'm closing this since it seems you have resolved the issue. Thanks for all the help everyone!

[odupuy]

I had the same issue on my with Mac with Sierra (10.12) for both 9.5 (port 5431). and 9.6 (5433).
I have not tried to run Postgres on a previous version.
My locale variables (>locale) are identical to the ones of Jonathan upper. My Mac is set to English as the preferred language in the system preferences.

My installer is the first one on the PosgreSQL download page aka the EntrepriseDB one. I Idid nothing fancy in the installation or later.
My /Library/PostgreSQL/9.5-6/data/pg_log folders were full of log files with

FATAL: postmaster became multithreaded during startup
HINT: Set the LC_ALL environment variable to a valid locale.

and the servers could not be connected by telnet.
Running >pg_ctl start -D /Library/PostgreSQL/9.5/data -l logfile.txt I had no success at starting the server.

In the Mac activity monitor, I could see a few processes for the user 'postgres' such as lsd, trustd.. but not the postgres process itself.
The PostgreSQL related processes start at boot time due to the entries in /Library/LaunchDaemons such as com.edb.launchd.postgresql-9.5.plist. It seems that there is no way to set an environment variable in the same file in a simple way.

After a couple of unsuccessful attempts including setting 'export LC_ALL=en_US.UTF-8' in my personal .bash_profile I have finally found the solution.
1) identify the home folder of the user postgres (~postgres), in my case /Library/PostgreSQL/9.5
2) create a .profile and .bash_profile there. I have not tested more to see which one is used.

$ cat .profile
LC_ALL=en_US.UTF-8
export LC_ALL
$ cat .bash_profile
export LC_ALL=en_US.UTF-8

3) chmod 755 .profile .bash_profile
4) delete all the errors in /Library/PostgreSQL/9.5-6/data/pg_log
5) restart the Mac (or restart these daemons (not tested))
6) check in /Library/PostgreSQL/9.5-6/data/pg_log for a log file containing something like

LOG: database system was shut down at 2016-10-05 14:12:05 EDT
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: invalid length of startup packet

7) check for many 'postgres' process names in the activity monitor
8) connect with telnet, pgAdmin and other and enjoy

May the luck be with you.

[jnicho02]

I added to the .plist in /Cellar/osgeo-postgresql (so that it doesn't get removed on restart)

<key>EnvironmentVariables</key>
<dict>
  <key>LC_ALL</key>
  <string>en_GB.UTF-8</string>
</dict>