Yugabyte-db: [YSQL] invalid locale name "en_US.UTF-8"

[aegershman]

[EDIT] See @mbautin 's update here: https://github.com/YugaByte/yugabyte-db/issues/3732#issuecomment-653261887

Root Cause: << When the installation directory is a symlink, post_install.sh currently is skipping a major part of what it is supposed to be doing (replacing substrings in files to point to the actual installation location). >>

---

Manually setting up YB 2.0.11 tservers on Ubuntu 16.04 on AWS and getting the following errors:

In YugaByte DB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/YugaByte/yugabyte-db/issues/1557initdb: invalid locale name "en_US.UTF-8"

I've followed the advice and troubleshooting from the following links and github issues:

• https://github.com/yugabyte/yugabyte-db/issues/2000#issuecomment-520945181
• https://github.com/yugabyte/yugabyte-db/issues/1557
• https://askubuntu.com/questions/76013/how-do-i-add-locale-to-ubuntu-server
• https://www.digitalocean.com/community/questions/language-problem-on-ubuntu-14-04
• https://stackoverflow.com/questions/17712700/postgres-locale-error
• https://www.postgresql.org/docs/9.4/locale.html
• etc.

I've also tried performing variations on the following to ensure the locale of en_US.UTF-8 is installed on the machines:

locale-gen en_US.UTF-8
dpkg-reconfigure locales --frontend=noninteractive
update-locale LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8"

and unfortunately still getting the invalid locale name error.

Here are the results of locale -a:

C
C.UTF-8
en_US.utf8
POSIX

and of locale:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

if we cat out /etc/locale.gen we'll see en_US.UTF-8 UTF-8 is uncommented and should be available for when we run locale-gen en_US.UTF-8:

cat /etc/locale.gen
...
# en_SG.UTF-8 UTF-8
# en_US ISO-8859-1
# en_US.ISO-8859-15 ISO-8859-15
en_US.UTF-8 UTF-8
# en_ZA ISO-8859-1
# en_ZA.UTF-8 UTF-8
...

locale-gen en_US.UTF-8
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.

and when checking etc/default/locale, I see the following:

cat /etc/default/locale 
LANG=en_US.UTF-8

Finally, when cating out the /i18n/supported locales, we see en_US.UTF-8

cat /usr/share/i18n/SUPPORTED | grep "en_"
...
en_SG.UTF-8 UTF-8
en_SG ISO-8859-1
en_US.UTF-8 UTF-8
en_US ISO-8859-1
...

I've tested running the initdb process similarly to how the pg_wrapper calling initdb would run the initdb script and the only success I've had has been if I run with --locale C, --locale POSIX, or --no-locale (same as --locale C):

reminder, results of `locale -a`:
C
C.UTF-8
en_US.utf8
POSIX

# these options work fine
./yugabyte/postgres/bin/initdb --locale C -D /tmp/ysqltest
./yugabyte/postgres/bin/initdb --locale POSIX -D /tmp/ysqltest
./yugabyte/postgres/bin/initdb --no-locale -D /tmp/ysqltest

## all result in...

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

# fails
./yugabyte/postgres/bin/initdb --locale en_US.UTF-8 -D /tmp/ysqltest
./yugabyte/postgres/bin/initdb --locale en_US.utf8 -D /tmp/ysqltest
./yugabyte/postgres/bin/initdb --locale C.UTF-8.utf8 -D /tmp/ysqltest

## all result in...

initdb: invalid locale name "C.UTF-8" # or some variant

I'm assuming that this section of the initdb operations configuring locale is going to force locale to be en_US.UTF-8 by default, however I could really use some advice to figure out why en_US.UTF-8 is not being configured and made available on Ubuntu 16.04?

Thanks in advice 👍

[aegershman]

Some more notes based off of https://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue

• performing apt-get install language-pack-en-base -y did not help, but it did populate other en_X.utf8 variants:

running locale -a before apt-get install language-pack-en-base -y:

C
C.UTF-8
en_US.utf8
POSIX

running locale -a after apt-get install language-pack-en-base -y:

C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
POSIX

• populating /etc/environment did not help unfortunately:

echo 'LC_ALL=en_US.UTF-8' >>/etc/environment
echo 'LANG=en_US.UTF-8' >>/etc/environment
source /etc/environment

followed by logging out / logging in again, still results in initdb: invalid locale name "en_US.UTF-8"

[d-uspenskiy]

@aegershman It is not a problem that there is no en_US.utf-8 in the output of locale -a have you tried to relogin after fixing your locale?

[aegershman]

yes I have, I've also tried doing a full restart as well as a recreate of the VM after setting locale to make sure the settings were loaded from boot.

If the output of locale -a doesn't matter, from where / how is initdb loading or checking if the locale is acceptable?

[d-uspenskiy]

@aegershman The following commands works on newly created Ubuntu 16 instance in AWS

    sudo apt-get install wget python
    wget https://downloads.yugabyte.com/yugabyte-2.1.0.0-linux.tar.gz
    tar xzfv yugabyte-2.1.0.0-linux.tar.gz 
    cd yugabyte-2.1.0.0/
    ./bin/post_install.sh 
    ./bin/yb-ctl create

Could you please try them?

[aegershman]

I'm using yugabyte-2.0.11 to test this, but here's my results with ./bin/yb-ctl create --v=4 --verbose --data_dir=/yb-ctl-test

2020-02-26 20:21:40,411 INFO: is_ysql_enabled returning True
2020-02-26 20:21:40,411 INFO: Running initdb to initialize YSQL metadata in the YugaByte DB cluster.
2020-02-26 20:21:41,413 INFO: initdb log file:
2020-02-26 20:21:41,414 INFO: The files belonging to this database system will be owned by user "root".
2020-02-26 20:21:41,414 INFO: This user must also own the server process.
2020-02-26 20:21:41,414 INFO: 
2020-02-26 20:21:41,414 INFO: In YugaByte DB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/YugaByte/yugabyte-db/issues/1557initdb: invalid locale name "en_US.UTF-8"
2020-02-26 20:21:41,414 INFO: Successfully ran initdb to initialize YSQL data in the YugaByte cluster
2020-02-26 20:21:41,414 INFO: Number of servers for daemon type master determined as 1 based on replication factor
2020-02-26 20:21:41,414 INFO: Number of servers for daemon type tserver determined as 1 based on replication factor
2020-02-26 20:21:41,414 INFO: is_ysql_enabled returning True
...
cat initdb.log 
In YugaByte DB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/YugaByte/yugabyte-db/issues/1557initdb: invalid locale name "en_US.UTF-8"

So even though the yb-ctl create process says it successfully started, and the master is up, the tserver didn't come up, and it's still throwing invalid locale name "en_US.UTF-8"... Hm.

./bin/yb-ctl status shows only the master is running, not tserver:

----------------------------------------------------------------------------------------------------
| Node 1: yb-tserver (Stopped), yb-master (pid 12983)                                              |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/postgres                                  |
| YSQL Shell          : bin/ysqlsh                                                                 |
| YCQL Shell          : bin/cqlsh                                                                  |
| YEDIS Shell         : bin/redis-cli                                                              |
| data-dir[0]         : /yb-ctl-test/node-1/disk-1/yb-data                          |
| yb-tserver Logs     : /yb-ctl-test/node-1/disk-1/yb-data/tserver/logs             |
| yb-master Logs      : /yb-ctl-test/node-1/disk-1/yb-data/master/logs              |
----------------------------------------------------------------------------------------------------

and here's the tserver ERROR logs:

tail -n 300 -f /yb-ctl-test/node-1/disk-1/yb-data/tserver/logs/yb-tserver.0ce0673f-2c3a-4594-846b-34c1715a2805.root.log.ERROR.20200226-202133.12986
Log file created at: 2020/02/26 20:21:33
Running on machine: 0ce0673f-2c3a-4594-846b-34c1715a2805
Application fingerprint: version 2.0.11.0 build 23 revision 2d52e1dfef65294a0ddf7c6377e3da2b8ed5a6b0 build_type RELEASE built at 24 Jan 2020 17:54:53 UTC
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0226 20:21:33.825103 12986 main_util.cc:22] Runtime error (yb/yql/pgwrapper/pg_wrapper.cc:331): /yugabyte/postgres/bin/initdb failed with exit code 256
E0226 20:21:48.835755 12986 io_thread_pool.cc:63] Io service failed to stop

Is there anything else I can do to cross-check exactly what it's looking for from the locale? Or perhaps the error message is a red herring? I tried running postgres-11.4 on the same server type using:

./postgres/bin/initdb -E utf8 --locale en_US.UTF-8 -D /testdir

and it succeeded and spat out a postgres.conf. I know the initdb.c files are different between postgres and yb, but just figured I'd point it out if it helps

I can also test with yb 2.1.0?

[aegershman]

Hm. I think this error message was a red-herring. I _think_ the real underlying problem is a filesystem permissions issue (trying to run the yb-tserver and master processes from a read-only volume mount, and it appears to be trying to write things to that read-only dir where the main yb binaries are stored), but the error message is manifesting with "unknown locale" for some reason. Just a theory.

[xyloman]

This leads to a question is yb-tservers + ysql can run as unprivileged (e.g. not root user). I believe what we both noticed was the yb-tserver would be installed by a root user but then attempt to run the pg process as postgres and it wouldn't be able to read and write files because it didn't own the directory. Maybe talking out of turn here.

[d-uspenskiy]

@xyloman YB cluster doesn't require root privileges for installation and running.

[aegershman]

Will post more on this later, but I think what we’re realizing is the “locale” error is a red herring, and that the issues we’re experiencing are with user, filesystem, and/or process permissions & symlinking issues, but were seeing the locale errors and focusing on that. Can post more details later, just wanted to provide and update before the weekend.

[srtripathy]

For what it's worth, identical problem exists on Redhat Centos Linux 7.

[kmuthukk]

Thx @aegershman -- if you have additional findings please share.

[m-iancu]

@aegershman @srtripathy
To confirm it is not a file permission issue you could double-check the permissions of the yugabyte data dir folder, and make sure the yb-tserver (and yb-master) processes are started with the appropriate user.

For the initdb locale issue, the simplest way to debug is to manually run the command that the yb-tserver process runs internally and confirm if you get the same error or not.
The command should look something like this:

YB_PG_LOCAL_NODE_INITDB=1 <path-to-yugabyte-installation>/postgres/bin/initdb -D <path-to-yugabyte-data-dir>/pg_data/ -U postgres

Now, if you get a locale error for the above command, you can try to set the LC_* environment variables before running (you might have to clean up the pg_data folder before (re)running):
For instance:

LC_ALL=en_US.UTF-8 YB_PG_LOCAL_NODE_INITDB=1 <path-to-yugabyte-installation>/postgres/bin/initdb -D <path-to-yugabyte-data-dir>/pg_data/ -U postgres

Or you can try setting it to C if that does not work.
If this succeeds then it should be just a matter of setting the appropriate vars in the script or service starting the yb-tserver process.
Otherwise, please let us know the error initdb reports and we can investigate further.

_Edit: added YB_PG_LOCAL_NODE_INITDB=1 flag to the commands (see comments below)._

[chuckwhite279]

Setting LC_ALL on command line seems to work when you test with initdb directly. However, the tserver process doesn't seem to pass that along.

$ LC_ALL=C /opt/yugabyte/bin/yb-tserver --flagfile /etc/yugabyte/tserver.conf
The files belonging to this database system will be owned by user "yugabyte".
This user must also own the server process.

In YugabyteDB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/yugabyte/yugabyte-db/issues/1557initdb: invalid locale name "en_US.UTF-8"

[m-iancu]

@chuckwhite279 You are correct, I should have added one thing. When the tserver runs it it adds an additional flag YB_PG_LOCAL_NODE_INITDB=1.
So the standalone command should be:

LC_ALL=en_US.UTF-8 YB_PG_LOCAL_NODE_INITDB=1 <path-to-yugabyte-installation>/postgres/bin/initdb -D <path-to-yugabyte-data-dir>/pg_data/ -U postgres

The extra flag does assume the en_US.UTF-8 is installed and configured locally since it will be used for the default databases. If that is not installed it will report that error invalid locale name "en_US.UTF-8"
In that case, does something like locale-gen en_US.UTF-8 work to generate the locale (assuming it's not in the locale -a list.
But all that is indeed system-specific so we have to do some more digging to provide the exact instructions.

[aegershman]

To confirm it is not a file permission issue you could double-check the permissions of the yugabyte data dir folder, and make sure the yb-tserver (and yb-master) processes are started with the appropriate user.

I can confirm the processes are started with the appropriate user (yugabyte) and the yugabyte user has read/write permissions to the directories specified by --fs_data_dirs=/mnt/store/yb/.... Just to clarify, this is what you mean by the yugabyte data dir folder, right?

For the initdb locale issue, the simplest way to debug is to manually run the command that the yb-tserver process runs internally and confirm if you get the same error or not.

so with en_US.UTF-8, there's still a locale error:

LC_ALL=en_US.UTF-8 YB_PG_LOCAL_NODE_INITDB=1 /mnt/bin/yugabyte/postgres/bin/initdb -D /mnt/store/yb/pg_data -U postgres
invalid locale name: "en_US.UTF-8"

same with en_US.utf8:

LC_ALL=en_US.utf8 YB_PG_LOCAL_NODE_INITDB=1 /mnt/bin/yugabyte/postgres/bin/initdb -D /mnt/store/yb/pg_data -U postgres
invalid locale name: "en_US.utf8"

but with C, we're okay, and the postgres process does init (and if you perform initdb before the yugabyte process starts, you can get the master/tserver processes running. This is because yugabyte sees that the pg_data/ dir exists, so it starts up):

LC_ALL=C YB_PG_LOCAL_NODE_INITDB=1 /mnt/bin/yugabyte/postgres/bin/initdb -D /mnt/store/yb/pg_data -U postgres
...

going further with LC_ALL=C

Out of curiosity I kept going further to see what would happen. If you do what's listed above and manually override --locale=C (LC_ALL='C') the yugabyte processes will start; but when you try to connect via ysqlsh (or use a sample app, etc.), you'll see the following:

/mnt/bin/yugabyte/bin/ysqlsh -p 5433 -h 10.156.89.41
ysqlsh: FATAL:  database locale is incompatible with operating system
DETAIL:  The database was initialized with LC_CTYPE "en_US.UTF-8",  which is not recognized by setlocale().
HINT:  Recreate the database with another locale or install the missing locale.

which I'm not even sure how that'd happen since if you check mnt/store/yugabyte/pg_data/postgres.conf there's no mention of en_US.UTF-8:

...
#------------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
#------------------------------------------------------------------------------
...
# These settings are initialized by initdb, but they can be changed.
lc_messages = 'C'                       # locale for system error message
                                        # strings
lc_monetary = 'C'                       # locale for monetary formatting
lc_numeric = 'C'                        # locale for number formatting
lc_time = 'C'                           # locale for time formatting
...
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------

# Add settings for extensions here
... # (empty)
...

or in ysql_pg.conf (which appears the same as postgres.conf):

...
#------------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
#------------------------------------------------------------------------------
...
# These settings are initialized by initdb, but they can be changed.
lc_messages = 'C'                       # locale for system error message
                                        # strings
lc_monetary = 'C'                       # locale for monetary formatting
lc_numeric = 'C'                        # locale for number formatting
lc_time = 'C'                           # locale for time formatting
...
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------

# Add settings for extensions here
... # (empty)
...

or in the postmaster.opts:

cat postmaster.opts
/mnt/bin/yugabyte/postgres/bin/postgres "-D" "/mnt/store/yb/pg_data" "-p" "5433" "-h" "10.156.89.41" "-k" "" "-c" "logging_collector=on" "-c" "log_directory=/mnt/log/yb/yb-tserver" "-c" "shared_preload_libraries=pg_stat_statements,yb_pg_metrics" "-c" "yb_pg_metrics.node_name=DEFAULT_NODE_NAME" "-c" "yb_pg_metrics.port=13000" "-c" "config_file=/mnt/store/yb/pg_data/ysql_pg.conf" "-c" "hba_file=/mnt/store/yb/pg_data/ysql_hba.conf"

My guess(?) would be because it's being initialized with the ../share/initial_sys_catalog_snapshot/.. directory, which perhaps is initialized with en_US.UTF-8. But regardless, that doesn't matter too terribly much, since it seems the root of the problem is that the initdb process.

starting pure pg_ctl

Using the LC_ALL=C process above, if instead of running the yugabyte, we just start the "plain" postgres process, we can connect:

/mnt/bin/yugabyte/postgres/bin/pg_ctl -D /mnt/store/yb/pg_data start
/mnt/bin/yugabyte/postgres/bin/psql -h localhost -p 5432 -U postgres
postgres=#

and if we list all the tables, we see the following:

postgres=# \l
                             List of databases
   Name    |  Owner   | Encoding | Collate | Ctype |   Access privileges
-----------+----------+----------+---------+-------+-----------------------
 postgres  | postgres | UTF8     | C       | C     |
 template0 | postgres | UTF8     | C       | C     | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
 template1 | postgres | UTF8     | C       | C     | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
(3 rows)

and finally, here are the results of querying the collation table to see what options are available:

postgres-# select * FROM pg_collation;

 collname  | collnamespace | collowner | collprovider | collencoding | collcollate | collctype | collversion
-----------+---------------+-----------+--------------+--------------+-------------+-----------+-------------
 default   |            11 |        10 | d            |           -1 |             |           |
 C         |            11 |        10 | c            |           -1 | C           | C         |
 POSIX     |            11 |        10 | c            |           -1 | POSIX       | POSIX     |
 ucs_basic |            11 |        10 | c            |            6 | C           | C         |
(4 rows)

Alright so here's why I bring this up-- I also tested loading postgres-11.6 onto this same Ubuntu 16.04 instance, with the exact same system config (as far as I'm aware of), and ran the following:

/mnt/bin/postgres/bin/initdb -E utf8 --locale en_US.UTF-8 -D /mnt/store/postgres/pg_data

and it initialized. Then if we check the OSS postgres, we see the defaults are initialized with en_US.UTF-8:

postgres-# \l
                              List of databases
   Name    | Owner     | Encoding |   Collate   |    Ctype    | Access privileges
-----------+-----------+----------+-------------+-------------+-------------------
 postgres  | yugabyte  | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | yugabyte  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/yugabyte          +
           |           |          |             |             | yugabyte=CTc/yugabyte
 template1 | yugabyte  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/yugabyte          +
           |           |          |             |             | yugabyte=CTc/yugabyte
(6 rows)

and when we check pg_collation, we see en_US.utf8 as an option...

postgres-# select * FROM pg_collation;
  collname  | collnamespace | collowner | collprovider | collencoding | collcollate | collctype  | collversion
------------+---------------+-----------+--------------+--------------+-------------+------------+-------------
 default    |            11 |        10 | d            |           -1 |             |            |
 C          |            11 |        10 | c            |           -1 | C           | C          |
 POSIX      |            11 |        10 | c            |           -1 | POSIX       | POSIX      |
 ucs_basic  |            11 |        10 | c            |            6 | C           | C          |
 C.UTF-8    |            11 |        10 | c            |            6 | C.UTF-8     | C.UTF-8    |
 en_US.utf8 |            11 |        10 | c            |            6 | en_US.utf8  | en_US.utf8 |
 en_US      |            11 |        10 | c            |            6 | en_US.utf8  | en_US.utf8 |
(7 rows)

... which is the same locale option when displaying locale -a:

C
C.UTF-8
en_US.utf8
POSIX

en_US.utf8 v. en_US.UTF-8

So on the Ubuntu 16.04 instance that's being used for all this, it looks like en_US.utf8 and en_US.UTF-8 are treated effectively the same. I've tried doing all the steps like running localedef -i en_US -f UTF-8 en_US.UTF-8 and locale-gen en_US.UTF-8 and all that, but it still ultimately produces the same thing by default which is en_US.UTF-8 and en_US.utf8 kind of being treated the same. Here's the results:

locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

locale -a
C
C.UTF-8
en_US.utf8
POSIX

localectl status
   System Locale: LANG=en_US.UTF-8
       VC Keymap: n/a
      X11 Layout: us
       X11 Model: pc105

cat /etc/default/locale
LANG=en_US.UTF-8

This makes me wonder, is it possible there's something different with how initdb does things in upstream postgres vs. yugabyte?

Thanks again for all your time I appreciate it. Let me know if there's any other logs that I can provide, etc.

[d-uspenskiy]

@aegershman Thanks for detailed description. Could you please clarify what version of YB 2.0.11 do you use? Was the archive downloaded from the following link
https://downloads.yugabyte.com/yugabyte-2.1.0.0-linux.tar.gz
or it was built from source by your own?

[aegershman]

yes it's from the precompiled archive, downloaded and extracted from (now running 2.1.2.0) https://downloads.yugabyte.com/yugabyte-2.1.2.0-linux.tar.gz 👍

[mbautin]

Hi @aegershman, we found another occurrence of this issue where the YugabyteDB installation directory was a symlink. In your installation, was the installation directory (/mnt/bin/yugabyte as far as I can see from the above), a symlink by any chance?

[mbautin]

Some more details on the post_install.sh issue related to this. When the installation directory is a symlink, post_install.sh currently is skipping a major part of what it is supposed to be doing (replacing substrings in files to point to the actual installation location).

Steps for a manual workaround:

• Remove .post_install.sh.completed from the installation directory, if you've already run post_install.sh.
• Modify the bin/post_install.sh script as follows.
  After the that sets distribution_dir=..., insert the following:

if [[ -L $distribution_dir ]]; then
  distribution_dir=$( realpath "$distribution_dir" )
fi

For context, if $distribution_dir is a symlink, the following command at the end of post_install.sh that was replacing some strings in binary files to point to the correct locations based on the installation directory, does not actually do anything because it does not follow symlinks:

find "$distribution_dir" \( \
   -type f -and \
   -not -path "$distribution_dir/var/*" -and \
   -not -name "post_install.sh" \
\) -exec sed -i --binary "s%$ORIG_BREW_HOME%$BREW_HOME%g" {} \;

(and I would like it to avoid following symlinks just in case there is a symlink inside the distribution tree that points somewhere else.)

[aegershman]

Yes, in my case the installation directory is a symlink (by the way it looks like the doc suggests symlinking the installation directories as well? just a heads up)

The yugabyte executable / bits for a particular version are stored and mounted in a read-only location on disk, and then that entire location is symlink'd in-full to a directory for the runtime user with full permissions to those symlinked executables. So if any parts of the yugabyte code follow symlink'd file paths back to the original source on the filesystem rather than operate on the symlink'd files and/or relative folder paths of those symlink'd files, they'll be working on files and in a filesystem location which is read-only and doesn't belong to that runtime user. Let me try this out.

[kmuthukk]

@aegershman --

The docs link you shared has a subtle difference. The post_install.sh step was done in a non-symlinked directory:

"cd ~/yb-software/yugabyte-${YB_VERSION}; \
        ./bin/post_install.sh"; \

and then, it suggests linking master/yb-tserver software directories as symlink to the installed software.

 "ln -s ~/yb-software/yugabyte-${YB_VERSION} ~/master";
 "ln -s ~/yb-software/yugabyte-${YB_VERSION} ~/tserver";

My understanding is that this issue happens if the post_install is run on an installation which lives in a symlinked directory. And @mbautin has a fix in review for the same.

[aegershman]

I'll post on this again once I definitely confirm it, but I think that fix might have squared things away 👍 I'll keep posted and update asap.

[aegershman]

Ayyyy I think we're in good shape. The only problem we're seeing now is that upgrading universes in-place which had --enable_ysql=false switched over to --enable_ysql=true appear to not have the postgres-related system tables be created, I'm assuming because --enable_ysql=false was passed at creation time in order to at get ycql/yedis rolling. The universes can be re-created from scratch or can be setup with an export/import of existing data, so it's not the end of planet earth.

Any thoughts on when this change might be incorporated into a release? In the mean time we'll make the modifications manually 👍

[mbautin]

@aegershman That's right, when we enable YSQL at universe creation time, we load the initial system catalog metadata from a pre-packaged snapshot, which already contains all YSQL system tables. If we need to do that on an existing universe, we will need to run a special form of the initdb command to create all of those tables (it takes a couple of minutes to run, that's why we don't do it this way during universe creation). I will share that command after I verify it internally.

[mbautin]

Hi @aegershman, so here is the command to initialize YSQL metadata (just tested locally). Substitute yb-master IPs for ip1, ip2, ip3.

YB_ENABLED_IN_POSTGRES=1 FLAGS_pggate_master_addresses=ip1:7100,ip2:7100,ip3:7100 ~/tserver/postgres/bin/initdb -D /tmp/yb_pg_initdb_tmp_data_dir -U postgres

[aegershman]

Right on, thanks @mbautin -- does the location of the data dir passed in initdb -D /tmp/yb_pg_initdb_tmp_data_dir matter? Does it need to be a /tmp directory, or should it be the same file location directory of the fs_data_dir?

[aegershman]

I just left it as the tmp directory since that's how it initializes and then it just continues on from that point on in the fs_data_dir, so it appears placing it in the tmp directory is the way to go.

Alright, right on, I think we can close this out. Thanks a lot, all. Very much appreciate it. Will keep an eye out for upstream post_install.sh changes merged into master branch 👍

[mbautin]

@aegershman The location of the data dir does not matter in this case -- it just creates some files there as a side effect, but it makes all its important changes in the YugabyteDB cluster's system catalog tablet by sending RPCs to the master leader. Glad it worked! I will add a link to the fix here soon when I commit the fix.

[aegershman]

👍 @mbautin thanks again for your help and time. I did end up having problems when trying to activate YSQL on an existing universe with 3 masters / 3 tservers and convince them to initialize themselves and to re-init with the initial snapshot and create the appropriate postgres tables. --enable_ysql was turned on for all masters/tservers, and they all started okay, but none of the pg tables loaded, so nothing with ysqlsh could be done. And re-running initdb didn't seem to help-- it _did_ work on 1-master 1-tserver, but barfed on me with 3-master, 3-tserver. It was probably just a fluke or I screwed something up, and of I didn't keep any logs 🤷 I just really didn't want to fight it anymore, and it didn't really matter because we ended up able to recreate the universes and reload the app data from scratch without fuss anyway. So it all worked out. Now that we're able to keep YSQL on by default it's allll good 👌

[mbautin]

@aegershman I'm glad you found a way to enable YSQL by default!
BTW I've merged the post_install.sh script fix in commit 18bd3da6af62e3cb3d8fce4d2c2392e9bfd3db5b.
We should be able to include it in the next patch release.