Silverstripe-framework: htaccess in "resources" folder denies page URL named "resources"

[cwchong]

Affected Version

4.01

Description

After installation, the "resources/.htaccess" file contains rules that rewrites all requests that does not match a file to 404.
This will mean that the website cannot contain a urlsegment like "resources" e.g. http://website/resources

Propose to remove that 404 rewrite and let the system handle 404s instead of via htaccess

Tasks

• [x] Agree on new name
• [ ] Auto rename folder transparently (via vendor-expose? no need to opt-in)
• [x] Investigate feasibility on customising folder name (separate out into new card?)
• [x] Upgrade notes (call out static publishing, caches and any htaccess)
• [x] Rewrite rule for htaccess to redirect if file exists
• [x] Check with infrastructure about SSP compatibility
• [x] Update dev docs

Pull requests

• [x] https://github.com/silverstripe/silverstripe-framework/pull/8364 (SS5 only)
• [x] https://github.com/silverstripe/vendor-plugin/pull/25 (SS5 only)
• [x] https://github.com/silverstripe/vendor-plugin/pull/26
• [x] https://github.com/silverstripe/silverstripe-framework/pull/8519
• [x] https://github.com/silverstripe/silverstripe-installer/pull/242
• [x] https://github.com/silverstripe/recipe-core/pull/33
• [x] https://github.com/silverstripe/vendor-plugin-helper/pull/5
• [x] https://github.com/silverstripe/vendor-plugin/pull/27
• [x] https://github.com/silverstripe/silverstripe-cms/pull/2353
• [x] https://github.com/silverstripe/recipe-cms/pull/22

[tractorcow]

Unfortunately, SiteTree::getExcludedURLSegments() intentionally excludes any real folders in the public webroot. It's not currently possible to have a page named 'resources' at the top level.

You can modify that .htaccess directly (add it to source control) and modify it if you need, but we'll be unlikely to make this a core change.

[tractorcow]

In 5.0 maybe we can rename to something like _resources

[chillu]

Yeah in hindsight that's a pretty silly naming decision. Since it's an automatically created and maintained folder, I don't think this is part of semver and needs to wait until 5.x. A lot of sites will run into this issue because it's such a common name, and "rename your URL paths" is a shit solution. In particular because of SEO - stable URLs are important. Raising to impact/high.

How much effort would it be to auto-rename the folder via vendor-expose?

[chillu]

Also: Ideally that’s configurable, to allow people to mask their install. Any generic name we choose will clash, and non-generic name will be a marker that you’re running an SS install. Might need to be in the projects composer.json?

[tractorcow]

Yes actually that could work. Both vendor module and silverstripe core could use that path in composer.json and it would "just work".

That means "resources" could be the default, but let it be set per-project in 5.0.

[tractorcow]

The plugin should be able to safely rename it if necessary too. That folder isn't normally committed to source control.

[robbieaverill]

Is there any way we can implement this in a backward compatible manner in the 4.x line, while leaving "resources" as the default?

[tractorcow]

Based on the name _resources I've implemented a PoC at https://github.com/silverstripe/silverstripe-framework/pull/8364. Plugin change at https://github.com/silverstripe/vendor-plugin/pull/25.

[tractorcow]

@chillu we can soft-configure this if you want to go the composer extras route.

[jenniferaslan]

Could you call the folder “etc”? That's a common name for that kind of folder in many systems, including Magento and the Mac OS.

[DorsetDigital]

The problem with setting anything arbitrary is that you will always have the chance of a collision with something (ETC is a company name in a number of industries, for example). If it were possible, I'd love to see it as a configurable option, then the problem shouldn't re-occur.

[maxime-rainville]

I had a look at Damian's PRs. I'm thinking let's add a new SS_RESOURCES_DIR flag to the .env config ... I saw @chillu's suggestion above about using composer.json for this, but every other path we track is handled through the env config.

This would involves the following steps.

@tractorcow @chillu Let me know if you think I've missed anything.

silverstripe/framework

• update frameworks's constants.php to define a new SS_RESOURCES_DIR constant that defaults to _resources
• Replace any reference to ManifestFileFinder::RESOURCES_DIR with SS_RESOURCES_DIR
• Update ManifestFileFinder::RESOURCES_DIR to get it's value from SS_RESOURCES_DIR and add a @deprecated note.
• Replace absolute references to resources in Dev/Install/Installer.php, src/Dev/Install/install5.php, tests/php/Control/SimpleResourceURLGeneratorTest.php
• Add entry to change log
• Update .env doc
• Update composer.json reference to vendor-plugin

silverstripe/vendor-plugin

The module can read its value from the .env files like framework. However we have to be careful here, because if we release this change as a minor release, people running pre-4.3 releases will get our new logic ... basically you don't want to end up in a scenario where, framework 4.2 wants to use resources while vendor-plugin wants to use _resources.

So vendor-plugin will need a way to detect if it's running a >=4.3 release or a <=4.2 release and default to a different SS_RESOURCES_DIR. Or we do a major vendor-plugin release.

Special points to consider

• Make sure SS4.2, SS4.1, SS4.0 install still work with the newer release of the vendor-plugin.
• Make sure sites upgrading from 4.2 to 4.3 transparently switch to the new name. Need to test with and without the public web root.
• Find any other module that explicitly reference resources and update them accordingly.
• How to handle transition for one custom name to another.
• Update .gitignore file in installer and recipe-core

[chillu]

So vendor-plugin will need a way to detect if it's running a >=4.3 release or a <=4.2 release and default to a different SS_RESOURCES_DIR. Or we do a major vendor-plugin release.

Right, we can't check for the presence of the SS_RESOURCES_DIR, and fall back to resources/ - because we don't require this constant to be defined in >=4.3. Does a composer create-project generate a .env file that we could add this constant to?

Otherwise, looks good to me, thanks for the great overview!

[ScopeyNZ]

Does a composer create-project generate a .env file that we could add this constant to?

No - You have to implement your own .env file. I assume you can define it and then run vendor-expose again to fix it though? Might want to add that to the "points to consider" - what if we change the resource constant?

[maxime-rainville]

I would expect the vast majority of people to not defined SS_RESOURCES_DIR and run with whatever our default is. So most people shouldn't need to have it in there .env file.

I just thought that I'll also need to update our .gitignore file in installer/recipe-core. People will need to manually update their own file when upgrading to 4.3, I don't think there's any way around it.

@ScopeyNZ raises a good point. I can write some simple rule to remove resources when running a vendor-expose, however if people successively change their SS_RESOURCES_DIR to a different values, there's no way to figure out what the old value was and to delete that folder for them. So they'll need to do that themselves.

[maxime-rainville]

I think I'll need to update https://github.com/silverstripe/vendor-plugin-helper as well :dizzy_face:

[chillu]

Heads up: This is likely missing the 4.3.0 release window (rc1 target date was today), because we've identified that it requires changes in the way we route requests in our own platforms (SilverStripe Platform and Common Web Platform). For people hosting their own environments, it's an opt-in change, so they're not impacted by default. But as a platform provider, we need to assume people will opt to another name, and cater for this.

Since it's a new feature, it might have to wait until 4.4.0, not comfortable pushing this out with a 4.3.1.

[maxime-rainville]

We've decided to remove this from the 4.3 release. That wouldn't allow enough time for SSP/CWP to be updated.

[maxime-rainville]

I've got most of the functionality working. I just need to test that this still works with silverstripe/vendor-plugin-helper.

I think I'll ditch the auto-removal of resources/_resources folder when they are not in use. My guess is that if you want to decide to use something else, a probably reason is that you need to store some other files in those directories. If I auto delete them, you pretty much screwed.

[tractorcow]

I had a look at Damian's PRs. I'm thinking let's add a new SS_RESOURCES_DIR flag to the .env config ... I saw @chillu's suggestion above about using composer.json for this, but every other path we track is handled through the env config.

@maxime-rainville instead of trying to shiv in env file support to composer plugins you should probably shift it to an extra config in composer.json. As far as the plugins are concerned, that is already the source of truth.

SilverStripe can already read config from the composer.json. It's not an environment value (it's not different for different environments) so it doesn't make sense to environmentalise it. It's more an aspect of the project itself, and should be committed to source control as a "fixed" flag.

[maxime-rainville]

Is anyone especially attached to https://github.com/silverstripe/vendor-plugin-helper ?

It's already relying on vendor-plugin for most of its functionality. The bit that it does customise pretty duplicate other logic from vendor-plugin any way. It's referencing a bunch deprecated class in vendor-plugin as well.

There's literally a Command in vendor-plugin that I could add to a Symfony Application executable. People who need the helper could just do composer global require silverstripe/vendor-plugin instead.

[jakedaleweb]

Coming from a platform hosting point of view I want to raise is that it will be possible to work with this and populate relevant configuration provided that the setting is in a predictable place.

I'm foreseeing that we will need to read from the composer.json and format some rules etc on the web boxes based on where the "resources" end up being populated to.

I'm not overly familiar with the extra section in composer.json but something like the following would suffice. It would be nice if this was set as default i.e. populated even when not customised.

    "extra": {
        "resources_dir": "_resources"
    }

[maxime-rainville]

@jakedaleweb Stig and Damian have been expressing similar concerns on the PR https://github.com/silverstripe/vendor-plugin/pull/26#pullrequestreview-169980093. I've already adjusted my approach to use the extra attribute in the composer.json file.

If you got a chance, maybe have a read through the discussion there and add your 2 cents.

[maxime-rainville]

It's ready for review. I don't think we need the SS5 pull requests ... not unless we want to revert to not having a configurable resources-dir in SS5.

[maxime-rainville]

We can't close this until ops has done its bit with CWP/SSP.

The matching JIRA job can be found at https://silverstripe.atlassian.net/browse/AWSMP-1081 (only available to SilverStripe staff)

[chillu]

Indy has tested this, and support for it has rolled out to nearly everyone on SSP now (see comments in https://silverstripe.atlassian.net/browse/AWSMP-1081), so we can close this.