Suitecrm: 403 errors in cache folder

[dagostinelli]

This seems like a bug. I'm getting 403 errors on a fresh install. Here are several use-cases, none of them work:

I'm using nginx. I can get mess with the security privs a whole bunch, open the folder and files wide open on the /cache folder -- (777, remove security bit) and do something in nginx to blind serve the files in there. Now it works. No 403 errors. However, is that really what's intended though? As soon as a cache file is recreated, it will go right back to this broken state.

None of this really helps:

• unzip from the supplied zip file and use the /cache directory as it comes out of the .zip file
• unzip from the supplied zip file and then chmod 755 directories and 644 files -- doesn't work
• delete the whole /cache directory and let the app rebuild it -- doesn't work
• git clone whatever comes from git and let the app rebuild it - nope

Deleting the whole thing and let the app rebuild it seems to get closest. But I still end up with 403 errors on some of the files in there.

The files that throw the 403s are:

• sugar_grp_yui_widgets.js
• sugar_field_grp.js
• style.css
• deprecated.css
• yui.css
  ... and more....

I know that we are not supposed to totally whack the /cache folder in production because it's not really a cache folder -- but I'm still just setting this up and I need to go full 777 and no setguid bit to make this work despite it being automatically created -- that's why I'm filing an issue.

As a work around, can you tell me exactly what permissions are needed in the cache directory?

[pgorod]

@dagostinelli you might want to take a look at this recent issue #1655 I think it is related.

[dagostinelli]

@pgorod That does sound related. Maybe we have it narrowed down. I don't think it's in the installer. I'm thinking it might be in whatever reconstructs the cache when the cache is missing.

Can you verify that for me? Perhaps, if you can verify it, then that will provide a little better info to the dev team.

To verify:

• the contents of /cache folder. Head to browser. The contents of /cache should be rebuilt on first load. But the site won't look correctly and I see 403 errors in the log (if you press F-12 (Firefox) and look at the Console, it also shows the 403s/
• As a variation, try completing deleting the /cache folder all together and see if it comes back.

For me, to fix, I have to head to the cache folder and chmod 777.

Can you verify that you are getting same behavior?

[pgorod]

I'm sorry, I would love to help (really!) but it would involve quite some work for me (because my problem got solved when I fixed my permissions, and it's not showing now... I would have to start a new install on a new VM) and I'm afraid this next week I am extremely busy with a very tough deadline on Friday, on another project. I won't be able to give SuiteCRM the attention it deserves this week... : - (

I agree with you that we need to narrow this down. And the installation use case is a great place to start, it's really a disastrous experience for anybody coming to SuiteCRM for the first time. Even if the problem is not with the installer code, but with the cache rebuilding code.

[samus-aran]

@dagostinelli What is your default permissions on your config.php file?

[dagostinelli]

@samus-aran Here you go:
-rwxr-xr-x. 1 apache apache 9953 Jul 1 23:58 config.php

[pgorod]

@dagostinelli I think she means _inside_ config.php, your settings for "default permissions", something like this:

 'default_permissions' =>
  array (
    'dir_mode' => 1528,
    'file_mode' => 432,
    'user' => '',
    'group' => '',
  ),

[dagostinelli]

Ah ok. Here they are. Anything look off? It would have been generated from the installer.

  'default_permissions' => 
  array ( 
    'dir_mode' => 1528,
    'file_mode' => 493, 
    'user' => '',
    'group' => '',
  ),

[samus-aran]

@dagostinelli Where are you hosted as some hosting providers have different setups.

Have you had a look at the below thread? After Will's post people do run periodic changes if they do somethings in Admin.

https://suitecrm.com/forum/installation-upgrade-help/2712-permissions#10919

[dagostinelli]

We are self-hosted. It's an internal VM.

Was setting those two values mentioned in the installer somewhere? I just set user and group to apache:apache. What about "dir_mode" and "file_mode"? What should they be set to?

[dagostinelli]

As a test, I set it like the following, then I removed the /cache folder and let the app rebuild it on first load. The 403s came back. So that wasn't a complete fix. Do the decimal numbers look OK?

  'default_permissions' => 
  array ( 
    'dir_mode' => 1528,
    'file_mode' => 493, 
    'user' => 'apache',
    'group' => 'apache',
  ),

[dagostinelli]

To make it work, I still had to chmod 777 on the cache folder. It has to be done to the cache dir itself. Just doing it to the contents of cache isn't enough.

chmod -R 777 cache

[pgorod]

The good thing about this issue is that we can keep the focus here on squashing the bug. This is not a forum post where we help the user solve his particular situation. This is the place where the good thing is to have the problem, and have it again, and again, until we can pinpoint the exact line of code that is wrong. And believe me, there is such a line, and the project badly needs it to be fixed.

So, kudos to dagostinelli - if I read you correctly, you now have a reproducible case of the infamous SugarCRM/SuiteCRM permissions degradation problem - where permission are set correctly, something is done on the system, and permissions get degraded to something that doesn't work. Am I right?

And can you debug it, stepping through the code to see exactly where the problem is?

[willrennie]

@dagostinelli As detailed in the forum post, you should set your dir_mode to '1517' and file_mode to '420', run a Quick Repair & Rebuild and then test to see what issues you have and if the 403's still occur.

[dagostinelli]

@willrennie That was the right fix. The final solution is:

  array (
    'dir_mode' => 1517,
    'file_mode' => 420,
    'user' => 'apache',
    'group' => 'apache',
  ),

Now... that said, I think there is still an issue somewhere. When I installed SuiteCRM, I did not touch the config.php file. It was created by the installer alone. Also, I do not recall any message asking the user to go into the config file and set these 4 lines just so. (did I miss it?)

The installer has a pre-flight check where it checks that permissions for certain folders be set just so. It tells the user to go fix them. However, the pre-flight check does not make any mention of the cache folder being set incorrectly in my case (as I recall).

In the end, the installer set dir_mode and file_mode to 1528 and 493 which resulted in a broken install.

I'll have to go through the installer again to verify. At this point, this seems to point back to the installer now.

[willrennie]

@dagostinelli I understand your dilemma, but there are so many flavours of OS and web server set ups that you will most of the time have to manually set the config.php permissions to set values to be correct for your environment. The thing to take away from this is that we should maybe document this clearer in the installation, within the wiki documentation(https://suitecrm.com/wiki), or within the installer scripts themselves to improve permssion checks/setting.

[pgorod]

Since this is the cause of so many troubles for users, I think it would really help to mitigate it somehow.

Maybe a more complex installation/repair procedure is required. Couldn't we set permissions, write a sample file, test file access, and then adjust permissions if necessary? And then do it a second and a third time until it's correct?

Other less ambitious mitigations could be:

• telling the user which user/group their web server is running under (this would save us about 20 posts a week in the forums)
• printing the necessary commands on the screen (chown, chmod), adapted to their installation path
• having a permissions test (much like the one that runs before upgrading SuiteCRM) telling the user if there are problematic files. This would help conclude whether the experienced problem (whatever form it assumes) is likely to be a permissions problem or not. The current verification in the Upgrade wizard doesn't seem to serve this purpose completely, maybe it leaves out some cache directories?

[dagostinelli]

I concur. The installer + documentation need to be fixed / updated. The mitigation steps that @pgorod sound appropriate for this.

To add, maybe the installer should choose better defaults for the config file. The installer is choosing these defaults, not the user and it appears that they aren't reasonable defaults. The user is being left with a broken install right off the bat. How about just making them 1517 and 420 initially?

What's the next step?

[pgorod]

I was installing a fresh new Ubuntu 16.04 VM today, with PHP 7.0.

I unzipped the SuiteCRM files, took the care to set all permissions correctly, then installed successfully. I got a login screen with broken permissions (literally showing lots of LBL_X strings).

I immediately checked permissions on my files and got this:

[drwxrws--- root    ]  ./cache/cache
[drwxrws--- root    ]  ./cache/cache/modules
[drwxrws--- root    ]  ./cache/cache/modules/Schedulers
[drwxrws--- root    ]  ./cache/modules/Administration
[drwxrws--- root    ]  ./cache/modules/Employees
[drwxrws--- root    ]  ./cache/modules/Import_1
[drwxrws--- root    ]  ./cache/modules/Import_2
[drwxrws--- root    ]  ./cache/modules/MergeRecords
[drwxrws--- root    ]  ./cache/modules/Trackers
[drwxrws--- root    ]  ./cache/modules/UserPreferences
[drwxrws--- root    ]  ./cache/modules/Users
[drwxrws--- root    ]  ./cache/Relationships

These are only the directories that got a different owner:group than the correct one, which would be www-data:www-data on my system. So the installer (or the subsequent first run) created most directories correctly, and _a few_ wrong. Notice: not wrong read/write permissions, but wrong owner:group.

This proves that the installer has problems, and I mean fixable problems, not accountable to "there are many different hosting setups out there" excuses. If it can do things right for some files, it can for others, right?

And this turns our attention to user:groups instead of permissions values like 755 and 775. The default_permissions array that we set in include/utils.php leaves those fields undefined, so I would assume that directories and files get created as owned by the user that is running the code. But that would be the php executable, under www-data. Why would root be used instead, what could cause that? What makes the directories on the above list different from the rest of the directories created under cache/?

[chris001]

This long standing permissions ownership bug is fixable in the installer code so that a human system administrator intervening to run commands is not required any more, I had a go at fixing it several years back and solved for apache on debian linux but with a few more lines of code to detect the environment it could easily fix this bug by applying accurate ownership and permissions for any web server on any Linux distribution. I have the code I'll update it on my fork and make a pull request as soon as possible.

[pgorod]

@chris001 that would really be a huge help. I've been working on this also, when I have the time. But I don't have any code or solutions, I'm just reading stuff about permissions, umask, etc., and I started trying a few things on a new VM as I said above.

I've been playing in my mind with a promising new idea, that maybe there are other processes messing up the permissions. I think the installer runs as www-data but the cron jobs run as root. I set up my cron jobs early in the installation process (in that preliminary screen that gives the crontab line), so they should be firing even before the install completes. Maybe that's what's creating directories as root...

I am absolutely determined to fix these permission bugs (I think there's more than one) and I really appreciate your help. Nothing provokes more disappointment with SuiteCRM than this, and it especially affects newbies and people who are just trying out the product for the first time.

[pgorod]

BINGO! It's the cron jobs running as root, during the installation. When nothing is installed yet, they just fail silently. But as soon as the database starts getting created, some jobs get to connect to the database and do at least part of their job, causing subdirectories to start popping up under the cache folder. Sure enough, if the cron job is running as root, these will be owned by root. Later on, when the normal www-data user tries to use them, it fails to gain access.

My tests confirm:

1. Configuring the cron jobs before install will result in a broken installation, and I see the guilty directories (owned by root) appearing halfway through the install process.
2. Without the cron configured beforehand, the directories are created only when the login screen runs, and get created with www-data as owner.
3. With cron.php configured to run as www-data, directories do get created before install finishes, but their owner is www-data so it doesn't break the application.

So, let's examine the factors of this bug one by one:

1- cron.php running as root. I'm not sure this is standard, it could be a stupid thing to do, but the fact is that if you set things up with crontab -e, that's what you get. There seems to be no official documentation on how to set up the cron command, but in many places in the forums this is exactly what is recommended. Maybe it should be running as www-data, something you can achieve in Ubuntu by editing /etc/crontab which gives you a slightly different syntax where you can specify a user name. Maybe other flavours of Linux behave differently.

Notice the command suggested at the bottom of the initial screen:

No mention of the user it will run under, and uses the syntax for crontab -e, not for nano /etc/crontab, i.e., without user name in the line.

2- cron jobs running before installer is finished. This is probably a bad idea anyway. I don't know exactly which jobs are running, or what they're doing. But since that pre-install screen above reminds admins to go and set up schedulers before install even begins, the cron.php code should probably check whether an installation is completed before running anything. You can get all sorts of weird behaviours. What could be a good sign that installation has concluded, so we can test it in cron.php? I see a file called install/status.json that is updated right at the end of the install process, that might do (if it's never touched again later)...

[chris001]

@pgorod
You shouldn't add any line to the crontab file belonging to the user root.
You should add the line in the crontab belonging to the user the web server is running as.
For example, to edit the crontab belonging to the user www-data you do:
crontab -u www-data -e
And add a line, something like this, depending on the path the application:
* * * * * cd /home/server1/domains/mycompany.com/public_html/crm; php -f cron.php > /dev/null 2>&1

[pgorod]

@chris001 yeah, I know that now, my last test (n. 3) was done like that. I did it slightly differently, editing /etc/crontab and adding

* * * * * www-data cd /var/www/html; php -f cron.php > /dev/null 2>&1

That column for the username is only available in the so-called system-wide crontab. But the effect is the same as per your instructions.

But most forum posts out there are not explaining this correctly, the official documentation has nothing about this, and the message on the pre-install screen is at least incomplete, and perhaps even misleading. Lots of people had this problem with broken installations, and now we know why.

I mean to make the instructions more explicit and include checks in the code so that it doesn't happen even to people who misconfigure cron like I did.

And the cron jobs running during install...? Think about it, so many things can go wrong, and none of it was ever tested.... it's the kind of thing that simply will not occur to you when writing a scheduled task, what will happen if it runs with a half-complete installation... with some tables already created, others not... and the same goes for cron jobs running during upgrades!

No wonder SuiteCRM sometimes acts mysteriously and creates incomprehensible and irreproducible permissions problems.

We should really try to do things like:

1. cron.php checks which user it is running under and refuses to run if it is not the Web server user.
2. We need a servicing flag, that is on during installs and upgrades. It could simply be a file created on the filesystem. cron.php exits immediately if the flag is on. Probably it is a good idea to clear this flag in Repair and Rebuild in case some failed upgrade leaves it there.
3. The pre-install screen should detect the web server user and inform the user. The text suggesting the crontab command can be improved to include this.
4. Documentation on the wiki needs to include recommendations for setting up the schedulers properly.

The time spent doing all of the above, and other things to improve the permissions problems, will be _far less_ than the time currently spent by dozens of us on the forums repeatedly saying to users "just chmod the whole thing without ever understanding why it was broken".

These things will make SuiteCRM a lot less treacherous, especially for non-technical users and beginners.

[chris001]

@pgorod You're right on all points. Especially a lock so cron will stop and exit before running when system is undergoing self modification, such as, either installation of the CRM, installation of modules, or upgrade to the CRM itself. Without that lock and checking it and exiting if it exists, all code on the system, which updates database tables, MUST be written to exit gracefully if and when it encounters serious problems, such as missing database tables, because it would be a legitimate scenario that the any table many not have been created yet, and so therefore the code must catch this and NOT commit bad data to other existing tables. This is impossible to guarantee without forcing all data access through a single data access layer, and forbid raw SQL, which is NOT the case, many modules and the CRM system itself run raw SQL without passing it through any data access layer.

[pgorod]

@chris001 I want to thank you for your help diagnosing this issue, I opened a couple of new issues (#2787 and #2788) to systematize and expose this a little better. Please do keep following this discussion there.

[pgorod]

@samus-aran I suggest closing this issue, if @dagostinelli doesn't mind, since it is being handled (more precisely) on those two other issues I created, and there is a PR for one of them already.

(BTW, this discussion was essential to getting me to a reproducible test case of the infamous SuiteCRM permissions degradation problem - thanks to everyone involved, this is now no longer a mistery).

[samus-aran]

No problem - will do.

@dagostinelli if you still have an issue then please feel free to re-open