DietPi-Drive_Manager | NTFS as userdata location, permissions issues with applications.

[Random90]

I have some trouble installing ownCloud using dietPi-software and configuring it to use external hdd.
I have used dietPi driver manager to mount external hdd with ntfs partition after installing ownCloud. Then i set User Data Location to that external drive. But owncloud web interface said something about wrong permissions in data folder.
Problem is, that driver manager forgot to add permissions attribute to /etc/fstab.
But now i have wrong permissions set, and owncloud is still now working properly. Tried to move back user data location to /mnt/dietpi_userdata. There was an error about low space (i've already copied my data to hdd), but dietpi config says that user data is located in /mnt/dietpi_userdata anyway But, moving back user data to external hdd is now not working. That is a big mess.

But most important thing is, that DietPi Driver Manager should mount ntfs partitions with permissions flag to avoid further problems.

[Random90]

Did clean install, set everything, but after reboot system forgets permissions for mounted ntfs drive.
So, setting ntfs partition as user data storage should automaticly modify fstab with gid, umask etc for user dietpi.
Edit: after changing premissions via fstab to dietpi with umask 007, i get this error when trying to access web interface:
Exception occurred while logging exception: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory
It seems, that issue is with /var/lib/mysql permissions - everithing is owned my dietpi (should it be mysql:mysql? Anyway I cannot change that).

edit2
Tried to run mysql service:
`root@DietPi:~# systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql; generated; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-09-26 22:42:22 BST; 20s ago
Docs: man:systemd-sysv-generator(8)
Process: 2179 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Sep 26 22:41:54 DietPi mysqld[2366]: 2018-09-26 22:41:54 1995644928 [Note] Plugin 'FEEDBACK' is disabled.
Sep 26 22:41:54 DietPi mysqld[2366]: 2018-09-26 22:41:54 1995644928 [ERROR] Could not open mysql.plugin table. Some plugins may be not
loaded
Sep 26 22:41:54 DietPi mysqld[2366]: 2018-09-26 22:41:54 1995644928 [ERROR] Unknown/unsupported storage engine: InnoDB
Sep 26 22:41:54 DietPi mysqld[2366]: 2018-09-26 22:41:54 1995644928 [ERROR] Aborting
Sep 26 22:41:54 DietPi mysqld[2366]:
Sep 26 22:42:22 DietPi mysql[2179]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed
!
Sep 26 22:42:22 DietPi systemd[1]: mysql.service: Control process exited, code=exited status=1
Sep 26 22:42:22 DietPi systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Sep 26 22:42:22 DietPi systemd[1]: mysql.service: Unit entered failed state.
Sep 26 22:42:22 DietPi systemd[1]: mysql.service: Failed with result 'exit-code'.`

Anyway, there is problem with external drive mounting AND database for owncloud.

[Fourdee]

@Random90

Many thanks for the report 👍

We'll look into this as soon as we can.

[Fourdee]

Additional logs:

Log file contents:
ownCloud is in maintenance mode - no app have been loaded

Your Data directory is readable by other users
Please change the permissions to 0770 so that the directory cannot be listed by other users.

An unhandled exception has been thrown:
Exception: Environment not properly prepared. in /var/www/owncloud/lib/private/Console/Application.php:135
Stack trace:
#0 /var/www/owncloud/console.php(105): OC\Console\Application->loadCommands(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#1 /var/www/owncloud/occ(11): require_once('/var/www/ownclo...')
#2 {main}

/DietPi/dietpi/func/dietpi-set_software setpermissions not being applied.

---

Yep, same issue as Samba mounts, we need to define the fixed permissions during mount.

[Random90]

As I was trying to fix it,i found out that defining fixed premissions for NTFS during mount of whole drive could lead to access denial to user data for other software. Maybe the solution is mounting specific folders with custom premission (i.e mysql:mysql for owncloud mysql data) after whole drive mount?
Or init script with chmod and chown after mounting ntfs with premission flag?

[Fourdee]

@Random90

Or init script with chmod and chown after mounting ntfs with premission flag?

Appears NTFS does not support setting any permissions on files, must be done during mount:

root@DietPi-ASUSTB:~# chmod 700 /mnt/ntfs/test
root@DietPi-ASUSTB:~# ls -lha /mnt/ntfs
total 13K
drwxrwxrwx 1 root root 4.0K Sep 28 19:25 .
drwxr-xr-x 8 root root 4.0K Sep 28 19:25 ..
drwxrwxrwx 1 root root 4.0K Sep 28 18:57 dietpi_userdata
-rwxrwxrwx 1 root root    2 Sep 28 19:08 test

defining fixed premissions for NTFS during mount of whole drive could lead to access denial to user data for other software.

Yep, however we did the same for Samba, we'll do the same here with mounting permissions, then run some tests against software.

[MichaIng]

We actually do permissions check before userdata transfer: https://github.com/Fourdee/DietPi/blob/master/dietpi/func/dietpi-set_userdata#L125

@Random90
Was transferring the userdata via dietpi-drive_manager not aborted? Then we would need to investigate our check. Impossible to resolve all needed different permissions for our offered software with only the possibility to set these globally for the whole drive on mount.

@Fourdee
For my impression we simply should stay with not allowing to move userdata to non-permissions drives/file systems, including NTFS and Samba. Best we can do is mounting the drive with dietpi:dietpi 775 (or 770) permissions to allow our software accessing and writing downloads/media and such there with cross access.

[Fourdee]

@MichaIng

For my impression we simply should stay with not allowing to move userdata to non-permissions drives/file systems, including NTFS and Samba. Best we can do is mounting the drive with dietpi:dietpi 775 permissions to allow our software accessing and writing downloads/media and such there with cross access.

Yep, its looking that way.

different file/dir mode options for ntfs?

mount -o uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770 /dev/sda1 /mnt/ntfs

root@DietPi-ASUSTB:~# ls -lha /mnt/ntfs
total 8.5K
drwxrwxrwx 1 dietpi dietpi 4.0K Sep 28 19:40 .
drwxr-xr-x 8 root   root   4.0K Sep 28 19:41 ..
drwxrwxrwx 1 dietpi dietpi  448 Sep 28 18:57 dietpi_userdata
-rwxrwxrwx 1 dietpi dietpi    2 Sep 28 19:08 test

[Fourdee]

@MichaIng

Was transferring the userdata via dietpi-drive_manager not aborted?

Nope its not, because chown returns a 0

root@DietPi-ASUSTB:~# chown -R root:root /mnt/ntfs/test; echo $?
0

[Random90]

Yep, however we did the same for Samba, we'll do the same here with mounting permissions, then run some tests against software.

In working external ext4 hdd, installation of owncloud creates two folders: mysql and owncloud_data:
root@DietPi:~# ls -la /mnt/ext_hdd/dietpi_userdata/ total 44 drwxrwxr-x 11 dietpi dietpi 4096 Sep 27 18:37 . drwxr-xr-x 5 root root 4096 Sep 27 22:03 .. drwxrwx--- 5 mysql mysql 4096 Sep 27 22:07 mysql drwxrwx--- 6 www-data www-data 4096 Sep 27 19:54 owncloud_data drwxrwxr-x 6 ubooquity dietpi 4096 Sep 27 22:03 ubooquity
With set premissions during mount mysql, owncloud_data and ubooquity would have the same premissions, making folder inaccessible by software that uses them.

@MichaIng

Was transferring the userdata via dietpi-drive_manager not aborted?

No, transfer to hdd was successful.

dietpi:dietpi 775 permissions to allow our software accessing and writing downloads/media and such there with cross access.

I suppose this is the best solution. Dietpi-drive_manager could also inform user of NTFS limits and possible solutions, i.e formatting to ext4.

Another solution is to shrink ntfs partition to minimum size and if left free space is equal or bigger than shrinked ntfs partition, create ext4 next to ntfs, copy all the data from NTFS to ext4, delete NTFS and resize ext4 to full disk xD But thats overkill

[Random90]

@MichaIng @Fourdee
Lets suppose we mounted NTFS drive with uid and gid 'dietpi'. Adding mysql, www-data and others to dietpi group. If we mount this NTFS with write access to groups, i.e 0770, this could work. Maybe even owncloud won't cry with

Please change the permissions to 0770 so that the directory cannot be listed by other users.

[MichaIng]

Okay so we need to check for permissions support the traditional way:

• Create test file
• Get current owner
• If current owner is root (should be), chown dietpi, otherwise try to chown root.
• Verify that owner changed accordingly.

I guess there is no separate check needed for chmod, right?

Of course we can also exclude some file systems directly, where we know that no permissions are supposed:

• fat, exfat, NTFS so far?

[MichaIng]

@Random90
Adding all possible software users to dietpi group and letting all the group access all those files is quite dangerous. There are many software's directories where you definitely only want the software's own user access. This is at least the case for mysql and www-data (webroot)! Other software with write access can not only damage e.g. the database unrecoverable but also change owner. The latter might not block access for the owning software but many will throw warnings complaining wrong permissions.

[Random90]

@MichaIng You are right, that is not good idea.

I've looked into ntfs-3g man page and there is one option:

permissions

  | Set standard permissions on created files and use standard access control. This option is set by default when a user mapping file is present.

It seems that permissions can be set with this option present while mounting.
From the same manual:

By default, files and directories are owned by the effective user and group of the mounting process and everybody has full read, write, execution and directory browsing permissions.

So the solution is probably mouting whole drive as a root with 'permission' option and then setting individual permissions for each installed software that uses userdata located on that drive on each boot.

I've tested this solution and it seemns to work

root@DietPi:~# mount -t ntfs-3g /dev/sdb1 /mnt/ntfs/ -o permissions,file_mode=0770
Using default user mapping
root@DietPi:~# touch /mnt/ntfs/test.txt
root@DietPi:~# chown dietpi:dietpi /mnt/ntfs/test.txt
root@DietPi:~# chmod 770 /mnt/ntfs/test.txt
root@DietPi:~# ls -la /mnt/ntfs/
total 8
drwxrwxrwx 1 root   root   4096 Sep 28 20:24 .
drwxr-xr-x 7 root   root   4096 Sep 28 20:14 ..
drwx------ 1 root   root      0 Sep 28 20:16 System Volume Information
-rwxrwx--- 1 dietpi dietpi    0 Sep 28 20:24 test.txt

Then the full solution is to create some array of required premissions for each software userdata directories and set permissions on each system boot after mounting NTFS drive using ntfs-3g with permission option.

[MichaIng]

@Random90
Ah good find!

But, by setting the permissions option, you can benefit from the full ownership and permissions features as defined by POSIX. Moreover, by defining a Windows-to-Linux user mapping, the ownerships and permissions are even applied to Windows users and conversely.

This is great. But those permissions do not survive reboot?
If they do, this would be a way we can support NTFS. However the "new" permissions check above would anyway succeed then.

But if this does not survive on reboot, I am not keen to adding this and better unallow moving dietpi_userdata to NTFS directly. Especially on larger cloud directories with many files, in my case large modelica models, tex files and picture galleries, re-applying permissions can take a very long time. Not keen to have the need to do this on every reboot and remount of the drive. Also manual set permissions are lost every time.

• At least the array already exist, as we reapply those permissions on dietpi-software installs. However my idea actually was to not run this on every install but only when moving userdata and on first run setup, instead just change the permissions of the individual software directories that are installed.

I generally see no reason/benefit of having dietpi_userdata on a NTFS drive. Jep you could theoretically unplug it and mount to a Windows system. But this should be better done as NFS share or such, as dietpi_userdata drive is expected to be always plugged in the DietPi system, as databases and software services have there binaries/data there.
Blame Microsoft for never adding ext4 support to windows, while it is done the other way round... off topic 🤣...

[Fourdee]

• DietPi-Software | MPD: Now runs under the group 'dietpi' and user 'root', allowing access to userdata directories when contained on samba networked drives: https://github.com/Fourdee/DietPi/issues/2092

As this also changes the above, our official stance is as follows:

To support moving of userdata, the target location MUST now support native linux filesystem permissions. The updated check for v6.17 will do this automatically.

As Sambado not support this fully (limited to fixed permissions for whole drive during mount options), to achieve userdata location on these devices, it will need to be completed manually by the user, outside of DietPi programs.

@MichaIng

Thoughts?

---

The other option is we allow for an "override" if the permissions check fails (eg: samba users with music), but this will be a nightmare to support.

[Random90]

@MichaIng

This is great. But those permissions do not survive reboot?
If they do, this would be a way we can support NTFS.

Another test. This time i've set only 'permissions' option, without specifying file_mode and others.
Result:

root@DietPi:~# mount -t ntfs-3g /dev/sda1 /mnt/ntfs/ -o permissions
Using default user mapping
root@DietPi:~# ls -la /mnt/ntfs
total 8
drwxrwxrwx 1 root   root   4096 Sep 28 20:24 .
drwxr-xr-x 7 root   root   4096 Sep 28 20:14 ..
drwx------ 1 root   root      0 Sep 28 20:16 System Volume Information
-rwxrwx--- 1 dietpi dietpi    0 Sep 28 20:24 test.txt
root@DietPi:~#

Permissions survived reboot 😃
@Fourdee
So it seems NTFS can be used to store userdata

I generally see no reason/benefit of having dietpi_userdata on a NTFS drive.

I see at least one reason - I have NTFS drive with more that 1TB data, that I hoarded over the years. I want these files to be available online in cloud for some reasons. If i need ext4 partition, the only possibility is to use second HDD with ext4. Simple plug in of origin NTFS to dietpi would be the simplest solution, without buying redundant drive.

Other reasons... Maybe someone uses dietpi to stream videos on local network and have 2 homes while using 1 hdd, moving it as that person travels between houses, and the second house residents use just Windows 😃.

[Fourdee]

@Random90

I have NTFS drive with more that 1TB data, that I hoarded over the years.

Hehe, done the same myself 👍

So it seems NTFS can be used to store userdata

If this works, we can roll it out, it will then pass the permissions test. I'll run some tests.

[Fourdee]

@Random90

Appears to work a charm, great find 👍 💯

root@DietPi-ASUSTB:~# ls -lha /mnt/ntfs/dietpi_userdata/
total 12K
drwxrwxr-x 1 dietpi   dietpi   4.0K Sep 19 14:12 .
drwxrwxrwx 1 root     root     4.0K Sep 28 20:30 ..
drwxrwxr-x 1 dietpi   dietpi      0 Sep 19 14:12 downloads
drwxrwxr-x 1 dietpi   dietpi      0 Sep 19 14:12 Music
drwxrwx--- 1 dietpi   dietpi   4.0K Sep 28 20:12 mysql
drwxrwxr-x 1 www-data www-data    0 Sep 28 18:57 owncloud_data
drwxrwxr-x 1 dietpi   dietpi      0 Sep 19 14:12 Pictures
drwxrwxr-x 1 dietpi   dietpi      0 Sep 19 14:12 Video

[Fourdee]

Re-test:

• Install owncloud
• Move userdata to ntfs drive
• 🈯️ bob's your uncle fanny's your aunt

---

@MichaIng

Hmm, I still think we need an permissions check override option, for those with samba mounts and music.
But we'd need to make the user aware its officially unsupported.

What do you think?

[MichaIng]

@Random90

Permissions survived reboot 😃

Very nice 👍.

@Fourdee

Hmm, I still think we need an permissions check override option, for those with samba mounts and music.
But we'd need to make the user aware its officially unsupported.

We already did the permissions check, thus at least wanted to not support non-permissions file systems. It is simply not possible to run Nextcloud, MySQL and e.g. media players/downloaders with such a userdata directory. So for my impression no override option, as this would only lead to follow up issues.

I would anyway never advice anyone to put whole userdata on a network mount. Incredible disadvantage to have software services to pull their binaries, config and data all through network, having network request for every single database request and such. I think this would even make MariaDB break/complain, dependent on speed/latency/stability of the network and file transfer protocol. Having the network share (server side) on userdata is another thing, since files are locally available for server, but the other way round for my impression is a total misconfiguration.
Of course everyone can manually mount e.g. a Samba music share into the userdata/Music dir.

Actually we already hide moving userdata for network drives:

if (( ! ${aDRIVE_ISNETWORKED[$INDEX_DRIVE_BEING_EDITED]} )); then

    G_WHIP_MENU_ARRAY+=('' '●─ Userdata & Swap options ')

[Fourdee]

@MichaIng

Good points 👍

Actually we already hide moving userdata for network drives:

Yep, but the dietpi-software userdata move offers a "list all mounts".

Regardless, a working permissions test is the way it should of been from the start, now fixed. So lets mark this as completed.