Spksrc: DSM6: adduser command not working [$100]

[Dr-Bean]

Turns out that the reason users aren't created on DSM6 is due to the fact that adduser can't be found. It seems that the correct PATH isn't set, so the file(s) can't be found.

We can solve this in a number of ways:

• Set PATH before calling adduser and similar commands. E.g. PATH=${PATH} adduser [..]. Yeah...probably not ;)
• Untested, but I see no reason why it wouldn't work: Export PATH in postinst and preuninst (some kind of generic setpath call maybe? Could tie in with some type of common.sh approach)
• Use the full path to the binaries, e.g. set ADDUSER = "${INSTALL_DIR}/bin/adduser".
  Somewhat of a pain because we have multiple packages that don't bundle their own busybox, but use e.g. Python's (which keeps those packages noarch), but otherwise it should work.

This applies to adduser, deluser, delgroup. There may be other commands that I didn't think of.

@SynoCommunity/developers?

There is a $100 open bounty on this issue. Add to the bounty at Bountysource.

[Dr-Bean]

Annoyingly, this issue disappears after you run adduser once successfully as root. Fix it once setting PATH (I did it via a edited package installer), and every subsequent package installation works fine, without any changes.

I don't know what's up with this. It almost looks like something got corrupted during the migration. Some package users have vanished without a trace. start-stop-daemon fails repeatedly, but similar to adduser: after you get it running once, all issues with it are gone and packages start working again.

[GoodOmens83]

I was looking at the "privilege" file (/conf) for the DSM6 Download Station package. Not sure if this was how the DSM5.x package was but I see things like:

{
"defaults":{
"run-as": "package"
},
"username": "DownloadStation",
"groupname": "DownloadStation",

See full file: https://gist.github.com/GoodOmens83/2d465ef7adbfce045820

The resource file (in the same folder) seems to have a section for creating a database:

"pgsql-db": {
"create-user": true,
"create-database": [{
"db-name": "download",
"db-owner": "package",
"db-encoding": "SQL_ASCII",
"db-collision": "skip",
"db-enable-lang-c": true
}]
},

[Dr-Bean]

More info:
Running adduser doesn't create an entry in /etc/shadow (not on my device anyway), which ultimately causes the permission denied errors etc when attempting to start packages.

On top of that, DSM runs /usr/syno/cfgen/s30_synocheckuser on startup which compares entries in both /etc/passwd and /etc/shadow. Entries found only in one of the two files are removed...which explains the disappearing of package users after reboots.
I believe it's also responsible for (re)setting the shell to sbin/nologin for users created via the GUI after rebooting, which makes that it useless for most, if not all, package users as well.

There are Synology packages that create their own user, e.g DownloadStation, but of course Synology has packed whatever it is into a binary... The source may be available, but I haven't spent much time in searching.
Probably best to see if Synology is willing to work with us here, this is getting ridiculous...

[GoodOmens83]

There are Synology packages that create their own user, e.g DownloadStation, but of course Synology has packed whatever it is into a binary... The source may be available, but I haven't spent much time in searching.

That makes sense - I looked all through the packages for any sort of reference to creating a user and that code I pulled out was as close as I could find.

I also noticed Synology spcifically calls the /bin/sh shell when starting things - e.g., su $PACKAGE_USER -s /bin/sh -c "$1" whereas we do something like su - ${USER} -c "PATH=${PATH} ${PYTHON} ${COUCHPOTATOSERVER} --daemon --pid_file ${PID_FILE} --config_file ${CFG_FILE}"

[Diaoul]

A little bit of email history.
9/2014: I officially asked Synology to provide packages user management tools so that we don't rely on our workaround (adduser from busybox) to manage our system users. It would also have the side effect of fixing our other permission workaround (using DSM groups to manage permissions due to DSM 5 ACLs). I'm against running packages as root for obvious security reasons.

I got the response that the request was forwarded to the engineering team.
Then I asked again (11/2014) and again (12/2014) and again (3/2015) and again (4/2015) and one last time as DSM 6 beta was announced (11/2015) without any other response that this was under development.
I asked for DSM 6 documentation multiple times (1/2016).

Now DSM 6 has landed and I asked again (3/2016) for documentation about DSM 6 user management changes. Hopefully this time my request will be answered and we will have a proper solution for system users.

[GoodOmens83]

wow - run everything as root :-1: :)

[Dr-Bean]

And I also sent in a request just after my last comment...

@GoodOmens83 Passing on a valid shell to su is a workaround for accounts without a valid shell (/sbin/nologin) There's a reason we prefer not to use accounts created via synouser/GUI, but depending on the response of Synology, I imagine it's our last resort..

[Diaoul]

So I've got an answer, @GoodOmens83 was on the good track: there is a configuration file that tells DSM to run that package as a dedicated user with the same name of the package itself.

1. By default, user name is the same as "package" field in INFO. Or he can specify one in the Json config.
2. Not editable from DSM UI, but allow to change its permission via Control Panel > Shared Folder or ACL editor.
3. The user won't be removed when package is uninstalled.
4. It's still possible to delete user using command line tools.

[Dr-Bean]

Ok, that actually works. After a bit of trial and error, a user was created during package installation. Not sure, but it seems there could be some limitations for the account name (max length or maybe certain characters are not accepted, had some trouble getting it going at first).
Along with that, the installation seems to error out if the user already exists.

The user shows up as a system internal user in the Shared Folders editor, so that works too.
Haven't attempted to start a package though, so not done yet.

[Dr-Bean]

Something to consider if we get this to work: prepend the user with sc- (or something like it, it depends on acceptable chars).
Heaps of people seem to think it's a good idea to manually create users, and if the installer craps out if a user already exists, we need to be able to avoid that. Still waiting for documentation from Synology on this, it's apparently on its way.

[simenon]

Reading through the DSM Developer Manual i really don't get wiser at all. They suggest using synouserto create users. However these show up in the user list.I guess preference is that they are not there, but on the system internal list. However if the userid is below 1024 they do not show up there.

The synouser command line tool can help you create user accounts. This account is visible in DSM,
and end users are able to edit the account settings in Control Panel. However, the accounts will not be
editable in DSM settings if they are created via methods other than synouser and with UID less than
1024.

Checking /etc/passwd i only see users with uid higher then some high value showing up on the system internal list. I would guess, if we add an user it would be to the system internal list, or does it even matter.
CloudSync:x:177120:177120::/var/packages/CloudSync/target:/sbin/nologin
PhotoStation:x:138862:138862::/var/packages/PhotoStation/target:/sbin/nologin
StorageAnalyzer:x:276949:276949::/var/packages/StorageAnalyzer/target:/sbin/nologin

I guess some logic of user creation also applies to group creation... however this looks even more mysterious. Some groups show up, some don't

[GoodOmens83]

editable in DSM settings if they are created via methods other than synouser

What are those other methods? Why does Synology have to be so mysterious about something so simple.

[Dr-Bean]

The available developer guide is outdated, and doesn't contain anything related to new features of DSM6. Quote from Synology rep: As DSM6.0 has been recently released, you will have access to an updated document soon..

My interpretation of the cryptic sentence wrt the other methods is that it's probably a reference to the new DSM6 method to create system users (which presumably was being developed at the time), or less likely, to our use of busybox's adduser.

As for the mysteriousness, learn to love it ;)

[GoodOmens83]

Maybe reach out to @kevinfang over here: https://github.com/SynologyOpenSource/pkgscripts-ng

Seems to be putting up DSM6 related things to making packages. Poked around the various things on https://github.com/SynologyOpenSource but didn't find anything groundbreaking. But I'm guessing they are tied to Synology, Inc. Or at least all of their copyrights mention Synology, Inc. as the owner....

[Dr-Bean]

I've made enough progress to know how to create system users 'the DSM6 way': without documentation, it's just tedious and it takes longer. The main issue is that the approach Synology has taken is quite error-prone, and there's no leeway.

For example, anything incorrect in the JSON and the package install will fail. The end result could be that the previous package version is uninstalled, but the updated package isn't installed...which means data loss.
Another outcome is that you end up in a state where you're not able to recover via Package Center, but have to manually intervene.

On top of that, we'll have to remove our old system users during preupgrade (or something), seeing as system users created via busybox aren't recognized as valid...which means another install failure, but with a cryptic message this time.
That, in turn, means we have to have the DSM6 toolchains added to spksrc first, because it allows us to create DSM6-only packages (via TC_FIRMWARE), and we need that ability to handle a migration.

And the list goes on...
All in all, it'd be a whole lot easier if Synology had implemented creation of system users via command line.

[b-sarrazin]

Hello! Sorry I don't have your developer skills so I maybe saying something stupid or that you have already tried but don't you have access to the synology commands like synouser --add or synogroup --add ? I am scripting the default configuration of my NAS this way.

[philwolstenholme]

I've added to the bounty on Bountysource

[shiv81]

I've added to the bounty on Bountysource

[GoodOmens83]

@Dr-Bean @Diaoul : have you heard any more / is there any more direction on this? What is the JSON method for user creation?

[Dr-Bean]

I haven't had time to look at it since my last comment, so use the following as a starting point to conduct your own tests. Don't quote me on the exact syntax for any command here, this is off the top of my head.

The gist of it is that we use the privilege file to create the user (similar to the DownloadStation one you linked to), and make additional changes in the installer.

In the privilege file:

• run-as should be set to package;
• use root for all the actions in ctrl-script;
• Remove the groupname section, because we'll (have to) continue to use synogroup for creating the sc-xyz groups for backwards compatibility. We have no particular use for a group per package at this point, and if we do, we should use synogroup as it provides more control;
• For username, use the following format: sc-${package}. Several reasons: deluser behaves wonkily on DSM6. It is also not able to remove users that are added via synouser/GUI.
  On top of that, we can't remove users via synouser, because those aren't 'our' users. In other words, this is necessary to make sure that the user creation and subsequently, the package installation doesn't fail because a user already happens to exist.)

In installer:

• In preupgrade, remove group- and user-relation with delgroup ${user} ${group}, deluser ${user}, similar to a package uninstall. We may want to force a synouser --rebuild all and synogroup --rebuild all, so the synology userdb and groupdb are updated.
• In postupgrade, set a description for the newly created user with synouser -add {user} {description}. Add the new user to an sc-${group} if applicable (via a syno* command).
  For package upgrades, we may want to continue adding the user to the users group. For new installs, we might not want to do that anymore, and only use sc-groups.
• Make sure USER is set based on buildnumber, so we maintain backwards compatibility (ties in to the sc- prefix we add for DSM6)

Open items:

• Make sure that permissions are still valid one way or the other. If we can't get this working right, we essentially will have to force clean installs for all the affected packages...we don't want that.
• Move generic functions to a common script, so we only have to write them once. There's an open issue on that somewhere.
• Testing, testing, testing

[Dr-Bean]

Stuff related to dsmcontrol:

• Set a shell when running su. The following syntax works for both DSM5 and DSM6: su ${USER} -s /bin/sh -c ${COMMAND}. We'll need to verify if ${COMMAND} actually needs to be wrapped in quotes or not: I believe I had noticed some issues when spaces are used, but I can't recall specifics.
• start-stop-daemon is a different matter. Not sure there's a way to get the same command to work for both DSM5 and DSM6, so we might need to split those commands up :/ DSM6 will probably force us to wrap that inside its own su call.

[loli71]

Just to help regarding su command, after looking at download station official package, the best way could be to common script definition (synology use pkg_utils file inside scripts directory) with a function (can be depending of DSM version if needed) like:
RunAsPackage() { su $PACKAGE_USER -s /bin/sh -c "$1" }

After, using this function in every needed scripts (even for start-stop-daemon) by

!/bin/sh

. /var/packages/DownloadStation/scripts/pkg_utils
.....
RunAsPackage "system command to run"

[Dr-Bean]

That's the idea, yes, but there's no particular reason to move a start command to common if its as generic as that, right?
Also, we'd probably want to use a format like

. `dirname $0`/common

for the inclusion, that way we don't have to make it package-specific.

[Martijn085]

With some help from another Synology user we got the start-stop-status script working with DSM 5.2 and 6.

kill find /proc -maxdepth 1 -user ${RUNAS} -exec /usr/bin/basename {} \;

if [ find /proc -maxdepth 1 -user ${RUNAS} -exec /usr/bin/basename {} \; | wc -l -gt 0 ]

[Dr-Bean]

@Martijn085 I'm not sure what you're referring to here? We don't have issues with start-stop-status, apart from the changes I mentioned above.

[MrLurch81]

@Dr-Bean the statements of @Martijn085 can be used if the application does not create a pid file. Previously, a ps statement was used to determine if the application was running, but we couldn't create a ps statement that was compatible with both DSM5 and DSM6. Therefore the status of the package (running/stopped) could not be determined correctly in the DSM Package Center.
The new statements avoid ps, which makes them compatible with both DSM5 and 6. However, if a pid file is created, that would still be the easiest way to monitor and kill an application.

[Dr-Bean]

Sure, I understand what the code does. But as you say, it's unnecessary when using a PID file. We use those for all of our packages. Hence the question.

[Martijn085]

i didn't know that al packages used a pid file. A few of my own packages did not, so this was a solutions to make it work. I thought I'd share it here in case the packages over here had the same problem.

[Dr-Bean]

Right. For packages that don't implement a daemon, we use start-stop-daemon. You can include it in your package like this: https://github.com/SynoCommunity/spksrc/blob/master/spk/python/Makefile#L40-L41.
Alternatively, you can depend on another SPK which includes start-stop-daemon. For example, add Python to SPK_DEPENDS, and put the bindir on $PATH in dsmcontrol.sh. This allows you to keep a package noarch (provided you don't need to cross-compile).

If you have other questions, let's continue that in a new issue :)

[Dr-Bean]

Opened #2345 as a first attempt to get DSM6 support to our packages. Feedback there is welcome.

[JoopNL]

Anyone interested in a solution, please review and comment on #2345 that @Dr-Bean has created. This seems to be the best way to move forward.

[throttlemeister]

What I have found when struggling with packages like Sonarr and CouchPotato (DSM6) is:
1 - Users for the packages appear to be created with UID 100 by the package installer
2 - Any subsequent install of an official package that creates a user will see the UID for above be changed to an available number above 1000
3 - The above packages are now broken and will no longer start and are unable to read their configuration files
4 - Changing the UID for affected user back to 100 will fix the problem until 2) happens
5 - Changing ownership of the package directories to reflect new UID will also fix the problems, this time permanently.

It appears that the UNIX useradd command will start numbering from 100 for regular users (standard Linux behavior), while the DSM usermanager will start numbering from 1000. It also appears that DSM does not like UID's < 100 for regular users, considering these restricted and actively changes them to be in-line with DSM user management, ie >1000.

[BenjV]

At last Synology has produced a manual how to build packages for DSM 6.
It is to be found here:
https://usdl.synology.com/download/Document/DeveloperGuide/DSM_Developer_Guide.pdf

In there you can find a link to a GitHub with tools and examples.

I don't have the knowledge nor the environment to build packages myself but I hope this information helps others who can.

[piejanssens]

Section of interest: https://usdl.synology.com/download/Document/DeveloperGuide/DSM_Developer_Guide.pdf#page=99&zoom=auto,-159,818

[tofuSCHNITZEL]

What I have observed:
When first installing e.g. sonarr the user is created like so:
"nzbdrone: x:101: 100:Sonarr User:/usr/local/nzbdrone/var:/bin/sh"
but after adding a user via dsm manager or restarting dsm the user entry disappears from /etc/passwd - I think because dsm is rebuilding passwd from its internal database.

[Dr-Bean]

A new branch, dsm6, has been created which will serve as a starting point for DSM6 supported packages.

Currently, it contains the latest available DSM6 toolchains (6.0.2), as well as updates for nzbget-testing and sabnzbd-testing packages to make them work on DSM6. I'll work on adding packages as and when I have time, PR's using the same approach are welcome.

Packages will not be published on the repository yet, seeing as the repository backend software needs some changes to be able to work with the DSM6 changes as well.

[fonix232]

@Dr-Bean would it be possible to release separate, manual-install SPKs of the packages related to this issue? I believe there are many of us who want to use e.g. Transmission but can't due to the last released SPK being still buggy.

[Dr-Bean]

@fonix232 If I gave the impression that the repository backend is the cause of delays: not really.

Currently, there are two packages available on the dsm6 branch. There's no point in asking for other DSM6 packages, they quite literally do not exist. I'll give prio to the most-used packages. No ETA's, it's done when it's done.
Once packages are updated on the dsm6 branch, but not yet available on the repository, then my answer would be that people can build those packages themselves.

[fonix232]

Ah, sorry, my fault.

It is quite frustrating though that Synology introduced such a breaking change without an easy solution for 3rd party package developers to easily adapt their packages to the new format... I was contemplating on buying a new NAS as my current one isn't exactly serving my needs any more, but this move makes me question if Synology keeps the needs of their users in their interest.

I'll check if I can contribute in any way to the fixing of the packages I use most (Transmission, Sickbeard, Sonarr, etc.).

[cytec]

@Dr-Bean a little bit OT but... have you been able to get a DSM6 VM up and running?

[Dr-Bean]

@cytec Haven't gotten around to it yet...I could sure use one though ;)

[BuSHari]

@Dr-Bean @cytec Can you please tell me how can i install 'Plexpy' on my DSM6? the package exists on the repository but cant install.

[cytec]

@BuSHari any error message? also: keep in Mind that we currently don't support DSM 6

[BuSHari]

After install it fail to run, but there is no error message in the log, or any message.
Do you want the message I get when trying to run the package?

[cytec]

well then you actually CAN install it... it just doesn't run which is exactly why this issue exists... as i said before: our packages currently don't support DSM 6 becuase of the changes synology did to user creation. Keep an eye on this issue, if there is a solution found, it'll likely be posted here

[BuSHari]

OK I thought there is a solution that I can do manually, thank you I'all keep an eye on this issue

[attaxia]

@cytec, You're probably not the right person to address, but I'm wondering why it isn't mentioned on either https://synocommunity.com/ nor the README for this repo that DSM6 is not supported? This seems like a pretty big deal.

[Argon-]

It isn't? I didn't even notice this since packages I installed with DSM5 are still running just fine on DSM6 for a long time now (e.g. ZNC). Does this mean I couldn't reinstall them?

[attaxia]

That is my experience. Users already created under DSM 5.0 will stay functional but adding new packages that require a user to be created will not work. I ran in to this, thought my DSM 6.0 install was corrupted so I did a reinstall. Then I couldn't install any of my packages anymore. I then had to go through the painful process of downgrading to DSM 5.

There are some workarounds like adding the user manually to the passwd file but these break as soon as you use the DSM built in user management to add a user there, breaking your installed packages. It only keeps working reliably if the package was installed on DSM 5

[rcork]

I've given up on SynoCommunity packages due to the lack of compatibility with DSM6 and moved to using Docker containers. It has a bit of a learning curve but there's pretty much a docker container for everything.

[attaxia]

Unfortunately my older NAS can't handle Docker.

[BenjV]

@Dr-Bean
I was looking at the script installer.sh in your DSM 6 fork and wanted to share my findings.

You used the command addgroup to add the user to the sc-..... group.
On my test NAS (DS116) with DSM6 that command does not exist.
To add users to a group I have to use:
synogroup --member
The problem with that command is, that the list of users must include al the users that are already in that group.
So you have to get them fist with the following command:
synogroup --get
Which produces a list of all the users in that group and add those users to the "synogroup --member" command.

[Dr-Bean]

adduser and addgroup are added via busybox. That weird approach synogroup uses is exactly the reason why I'm not using it (of course it's possible...but I'm not going to do that if I don't have to)

[BenjV]

Ok, thanks for the info.

[madcowGit]

I have no experience with programming, but am an enthousiat user of Synocommunity.

Is it not possible to make a new package that reinstates the missing commands? Much like the Nano package does (https://synocommunity.com/package/nano) ?

[Dr-Bean]

@madcowGit It's not a question of reinstating. Synology has made changes that prevent us from using anything else than their commands.

[Intrapixel]

It seems that the only way is via SSH with the command synouser and editing the file /etc/passwd, I do manually to be able to install any server. Maybe you can make an installation script to run as root.

[cytec]

@Intrapixel there is already an official solution and @Dr-Bean is working hard on it to move most of our Packages to the DSM 6 compatible solution (see dsm6 branch) so we already know how to do it... but still have to do it for every package and if i remember corretly there still is some server side stuff to handle afterwards. so it might be a while till DSM6 Packages will be available.

[zebulon501]

Concerning the issue with the /etc/shadow file, it might be a good idea to enable the feature CONFIG_FEATURE_SHADOWPASSWDS in cross/busybox/configs/usrmng.

I didn't tried it, but it seems to do the trick by looking at the code of busybox-1.26.2/loginutils/adduser.c.

[Dr-Bean]

@zebulon501 The issue isn't adding entries into /etc/shadow, it's keeping them in there ;) DSM6 removes entries that don't fit certain criteria.

[ghost]

is this related to deluge showing 'stoped' state in package manger? But yet it still runs and I can access it just fine via web ui and daemon

[gdubicki]

How can I help with moving the dsm6 branch forward, @Dr-Bean @cytec ? I can compile and test something on my Syno.

[BenjV]

I don't know if it is any help but I just post it here.

I have created a package that will install and run under DSM 6 and that will add the user that is created by DSM via the privilege file to the group "sc-media".
It does not need busybox, just scripting
I put this in the postinst

Here is the script:

#!/bin/sh

SC_GROUP="sc-media"
SC_GROUP_DESC="Media related group"
USER="PackageUserName"

    # Create syno group with the user in it
synogroup --add ${SC_GROUP} ${USER} > /dev/null
if [ $? -eq 0 ];
    then
        synogroup --descset ${SC_GROUP} "${SC_GROUP_DESC}" ;
    else
        MEMBERS="$(synogroup --get ${SC_GROUP} | grep '^[0-9]' | sed 's/.*\[\([^]]*\)].*/\1/' | tr '\n' ' ')";
            # The member command needs al users on the command line so we add the user to the list we just found
        if [[ ${MEMBERS} != *${USER}* ]] ;
            then
                synogroup --member sc-media ${MEMBERS}${USER};
        synogroup --rebuild all;
        fi  
fi
synogroup --rebuild all
exit 0

[ymartin59]

@BenjV Thanks for searching. The main issue is to gain control on user creation and remove to get packages able to upgrade from 5.2 to 6.0.

I am looking at the way MariaDB package declares its service user mysql. Content of /var/packages/MariaDB is very interesting but found no clue about how mysql user is created (or removed).

[BenjV]

Creating of users is done via the privilege file in the config folder:
It looks like this:

{
    "defaults":{
        "run-as": "package"
    },
    "username": "<PackageUsername>",
    "ctrl-script": [{
        "action": "postinst",
        "run-as": "root"
    }, {
        "action": "preuninst",
        "run-as": "root"
    }, {
        "action": "postuninst",
        "run-as": "root"
    }, {
        "action": "preupgrade",
        "run-as": "root"
    }, {
        "action": "start",
        "run-as": "root"
    }, {
        "action": "stop",
        "run-as": "root"
    }]
}

Adding extra users can be done with the same script as I posted before by replacing the "synogroup" command with the "synosuser" command

[ymartin59]

@BenjV Sure but there are still some troubles to get any package upgraded properly: a DSM 5.2 package may have created "user" at installation and after upgrade to DSM 6, package upgrade with privilege may create a "userXYZ" instead of reusing "user". As far as I understood that is the reason why @Dr-Bean looks for a way to manage user by script.

Package MariaDB does not have "privilege" file (or I have not found it) and its service user is "mysql", so not the same as package name. And I found no place where synouser or synogroup is used. So remains the question how this "mysql" service user is created.

[BenjV]

Just replace the "PackageUserName" with "mysql" .
Removing old users could be done via the "synouser" command.

If you don't understand how a package for DSMS 6 must look like I suggest to read this.
https://usdl.synology.com/download/Document/DeveloperGuide/DSM_Developer_Guide.pdf

[olskar]

Any news on this @Dr-Bean ? :)

[ymartin59]

I consider it as closed: #2949 #2904

[gerwitz]

I would like to thank the longevity of this issue for motivating me to switch to Docker.