Silverstripe-framework: Run webroot in public subfolder

[chillu]

Overview

Continued from work around moving modules into the vendor folder. Requires use index.php for routing instead of framework/main.php

Acceptance Criteria

• Support operation on various webservers (incl. ones without rewrite rule support)
• Support running through php -S
• The resources/ folder can be placed in the public/ folder
• The installer creates this project structure by default (to promote best practice for new projects)
• The recipe-core module creates the public/ folder
• Upgrading to 4.1 does not change your project structure, you have to opt-in
• Opting out on newly created projects is possible (may require manual effort) but the default is to use public folder.
• Folder is fixed to public folder. If the folder exists it will be used, and if it doesn't legacy non-public project is assumed.
• Works with inlined themes (not composer dependencies) but these need to be manually installed to public folder if not using composer
• Works with themes / non-vendor modules as composer dependencies (see silverstripe/vendor-plugin#4) . May require per-modules changes to support 4.1, but should be trivial to upgrade a module for 4.1 support.
• Documentation for adding / removing a public folder from a website is available.
• Theme usage docs are upgraded
• Server installation docs are upgraded
• There is a mechanism for treating public committed files as a themed resource.

Tasks

• [x] Update recipe-plugin to conditionally move public files into public / root depending on project configuration.
• [x] Ensure that new projects with all recipes (recipe-core, recipe-cms, installer) create public folder by default.
• [x] Shift all public resources for all recipes (including recipe-core/index.php) into the public folder on each repo).
• [x] Update vendor-plugin Auto-move resources/ into public/resources (detect public folder and use if present).
• [x] Update vendor-plugin to expose resources for silverstripe-module and silverstripe-theme types as well.
• [x] Update vendor-plugin to expose project resources not controlled by composer dependencies. E.g. to expose a custom-committed theme folder.
• [x] Update module resource locator to work with files in public (e.g. public folder themes).
• [x] Fix framework tests
• [x] Test installer
• [x] Test and fix IIS operation
• [x] Adjust default gitignore paths
• [x] Server installation docs
• [x] Upgrading docs
• [x] Check that #1384 has been addressed
• [x] Ensure that resource loader / requirements searches public files, before private files. Add a new convention that resource paths exclude the public/ prefix, but is assumed to look there before searching private files.
• [x] Allow $public to be specified as a theme-set that treats the public folder as a theme.

PRs:

• [x] https://github.com/silverstripe-themes/silverstripe-simple/pull/51
• [x] https://github.com/silverstripe/silverstripe-framework/pull/7741
• [x] https://github.com/silverstripe/silverstripe-installer/pull/220
• [x] https://github.com/silverstripe/vendor-plugin/pull/12
• [x] https://github.com/silverstripe/recipe-plugin/pull/4
• [x] https://github.com/silverstripe/recipe-core/pull/13
• [x] https://github.com/silverstripe/recipe-cms/pull/7
• [x] https://github.com/silverstripe/silverstripe-assets/pull/108
• [x] https://github.com/silverstripe/silverstripe-cms/pull/2069
• [x] https://github.com/silverstripe/silverstripe-serve/pull/21

Notes

• Can we introduce this as opt-in in a 4.x release?
• The public/index.php file should be considered user-land (and customiseable), so should contain minimal code (it's going to be hard to upgrade it later)
• Ideally this moves the current index.php from installer to a specialised server.php, and creates a new public/index.php
• I think we should suggest moving mysite/ public assets into public/, rather than trying to symlink them
• Inlined themes (without a composer dependency) need a script to copy them over (can't use composer lifecycle hooks). These themes don't have a composer.json file for expose metadata either. We could add it in the root composer.json? We should reuse the code from https://github.com/silverstripe/vendor-plugin-helper to create a command within framework to symlink/copy.
• Configurable public folder might be tricky, since we're referencing files in Installer.php (which happens before we can rely on constants.php etc)
• Requirements should assume that paths are relative to the public/ folder (unless they are qualified with a module prefix)
• Decision is per project, not per environment (in order to allow for files to exist in public/ without symlinks)

Laravel layout

server.php <- only used for php -S
vendor/
public/
  index.php
  .htaccess

Related

• Support expose in themes

[chillu]

OK, I've got a PoC for this.

Branches:

• https://github.com/open-sausages/silverstripe-installer/tree/pulls/4/request-uri
• https://github.com/open-sausages/silverstripe-framework/tree/pulls/4/request-uri
• https://github.com/open-sausages/silverstripe-assets/tree/pulls/1/remove-base-path-autodetect
• https://github.com/open-sausages/silverstripe-serve/tree/pulls/index.php-support

Notes:

• In order to make this the default behaviour in SS4, we'd need all modules which expose assets to implement the new silverstripe-vendormodule. That might be reasonable? The larger impact will be requiring every dev to change their webhosting structure (incl. SS Ltd platforms). Maybe we'll make it opt in for 4.x, and make it the default for newly installed projects in a later 4.x release once we're confident it works well?
• Removed main.php, making the index.php in installer a larger part of the API surface. This means users should be able to rely on the contained PHP at least for the 4.x release cycle. I believe we've already made that commitment by documenting how to overwrite framework/main.php with mysite/main.php
• Campaigns and assets won't fully work until https://github.com/silverstripe/silverstripe-framework/issues/7405 is merged
• Relies on a themes symlink (until https://github.com/silverstripe/vendor-plugin/issues/4 is implemented)
• Relies on a resources symlink. That'll be hard to fix, since VendorModule.php has a hardcoded path (VendorModule::DEFAULT_TARGET) which can't be overwritten by SS. The command needs to run without executing any PHP code in the project (for security reasons). We could add the ability to read .env files there?
• Try server.php with php -S 127.0.0.1:8181 -t public server.php (needs explicit webroot)

First, test "legacy" operation with webroot == base path on those branches.

git clone -b pulls/4/request-uri https://github.com/open-sausages/silverstripe-installer.git 4-clean
composer config repositories.framework vcs https://github.com/open-sausages/silverstripe-framework.git
composer config repositories.assets vcs https://github.com/open-sausages/silverstripe-assets.git
composer require "silverstripe/framework:dev-pulls/4/request-uri as 4.0.x-dev"
composer require "silverstripe/assets:dev-pulls/1/remove-base-path-autodetect as 1.0.x-dev"

Then move the webroot into public/. First, add SS_PUBLIC_DIR="public" to your .env. Then run the following:

mkdir public/
mv assets public/
cd public
ln -sf ../themes themes
ln -sf ../resources resources
cd ..

cat > public/.htaccess <<EOF
### SILVERSTRIPE START ###

# Route errors to static pages automatically generated by SilverStripe
ErrorDocument 404 /assets/error-404.html
ErrorDocument 500 /assets/error-500.html

<IfModule mod_rewrite.c>

    # Turn off index.php handling requests to the homepage fixes issue in apache >=2.4
    <IfModule mod_dir.c>
        DirectoryIndex disabled
        DirectorySlash Off
    </IfModule>

    SetEnv HTTP_MOD_REWRITE On
    RewriteEngine On

    # Process through SilverStripe if no file with the requested name exists.
    RewriteCond %{REQUEST_URI} ^(.*)$
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule .* index.php [QSA]

</IfModule>
### SILVERSTRIPE END ###
EOF

cat > public/index.php <<EOF
<?php

use SilverStripe\Control\HTTPApplication;
use SilverStripe\Control\HTTPRequestBuilder;
use SilverStripe\Core\CoreKernel;
use SilverStripe\Core\Startup\ErrorControlChainMiddleware;

require __DIR__ . '/../vendor/autoload.php';

// Build request and detect flush
\$request = HTTPRequestBuilder::createFromEnvironment();

// Default application
\$kernel = new CoreKernel(BASE_PATH);
\$app = new HTTPApplication(\$kernel);
\$app->addMiddleware(new ErrorControlChainMiddleware(\$app));
\$response = \$app->handle(\$request);
\$response->output();
EOF

[sminnee]

Is that mod_rewrite bit still needed now that you're using index.php regardless?

[tractorcow]

Relies on a resources symlink. That'll be hard to fix, since VendorModule.php has a hardcoded path (VendorModule::DEFAULT_TARGET) which can't be overwritten by SS. The command needs to run without executing any PHP code in the project (for security reasons). We could add the ability to read .env files there?

We can't rely on symlinks always being available. In this case is a copy sufficient?

[sminnee]

If I understand correctly, we need a way of setting target to "public/resources" without making reference to the SilverStripe codebase. Possibly via a SS_SOMETHING env var that can optionally be set in .env.

I'd probably make the dotenv package a suggest rather than require of the vendor module, and then make use of its functionality only if the package is installed. Then we don't bloat out its installation in cases where that's not needed.

[tractorcow]

Yes I suggest another env var; We can specify this in .env, but can also specify it on the command line during install / running vendor-helper.

[chillu]

Yeah I was missing the fact that composer includes constants.php as part of early-stage autoloading, which gives us an opportunity to parse .env - presumably before vendor-plugin kicks in. Meaning we can add an env var to control this behaviour. For global install with vendor-module-helper, we'll need to define the env var directly since that global dependency doesn't use the project's autoloading (for security reasons). composer install && export SS_RESOURCES_TARGET_DIR=public/resources vendor-module-helper link.

[tractorcow]

@chillu suggestion for solving non-vendor resources https://github.com/silverstripe/vendor-plugin/pull/10

[chillu]

Internal reference to SSP and CWP tickets:
https://silverstripe.atlassian.net/browse/AWSMP-407
https://silverstripe.atlassian.net/browse/CWP-1094

[tractorcow]

Discussed implementation details with @unclecheese and we've decided to make the public folder as light as possible; vendor resources will still be symlinked by the vendor-plugin module, but only to the base resources/ folder.

There will be a separate symlink task (Flushable in silverstripe) that will symlink / copy resources from base to /public. The only exception is assets, which will "really" be in /public/assets.

[flamerohr]

Oh ok, how come assets aren't treated as a symlink as well?
Is it because of the potential size of the folder and if it needs copying 💥 ?

[unclecheese]

Yes, but more importantly, you'd have to run the copy task every time the user made changes to the assets/ dir.

[tractorcow]

Yeah, because symlinks are just one option, some filesystems need to be copied, which of course won't work for assets.

[dhensby]

Discussed implementation details with @unclecheese and we've decided to make the public folder as light as possible; vendor resources will still be symlinked by the vendor-plugin module, but only to the base resources/ folder.

So this means we have resources in /[projectroot]/vendor/[vendorname]/[package]/[exposedpath] symlinked to /[projectroot]/resources/[vendorname]/[package]/[exposedpath] and then that is symlinked to /[projectroot]/public/resources/[vendorname]/[package]/[exposedpath]?

If I'm understanding that correctly, I'm not clear on the advantage of this double symlink-ing here.

As I see it, the "public" folder should be where all publicly accessible assets are stored, anything that doesn't need to be public goes in the parent directory.

[unclecheese]

It's important that the vendor-plugin remain SS unaware. Symlinking from vendor to public/ would require the plugin to know your SS_PUBLIC_FOLDER env setting, among other things, and we need to keep all that decoupled.

[dhensby]

And why is that? Because our other modules/tooling is going to need to be aware of this setting, so why is vendor-plugin not allowed to know about it?

[unclecheese]

My initial feeling was that there's a pretty reasonable use case for vendor-plugin having utility outside the SS ecosystem. All it does is expose a composer.json defined whitelist of vendor assets to another directory, and I can imagine lots of frameworks/projects benefiting from that, and that benefit to me seems more valuable than the double symlinking is costly.

But I didn't realise that it depends on type: "silverstripe-vendormodule. That makes it SS-specific by definition. So I'll wait to hear from @tractorcow on this one.

[chillu]

• Protected assets are placed outside of public webroot

I've moved this AC to https://github.com/silverstripe/silverstripe-framework/issues/7710 instead, to limit hosting environment changes on the short term (Damian discussed this with our infrastructure team)

[tractorcow]

There has been considerable discussion regarding the specifics of implementation of this feature; I've discussed at-length with @unclecheese and @chillu offline and will present the following two options for architecture: I'd like to request users to vote on these using 👍 or 👎 , and ideally with a comment below presenting your reasoning, or if you have any other ideas you would like to share.

[tractorcow]

Solution <<Unicorns>>: public folder is fixed per-project

Overview

The public folder is controlled by .env setting SS_PUBLIC_DIR, which is set by the installer project (before we did not provide a default .env file). This will default to public, but can be removed if necessary.

A physical public folder will be committed to source control, and all web-related files (e.g. index.php, .htaccess) will be committed here.

recipe-plugin module will be updated to support expose. Any project file in expose will be installed to the public path (if configured) instead of the project root. Files will only be copied (no symlink etc)

vendor-plugin will work the same way, but will conditionally expose to public/resources or resources based on the environment variable.

vendor-plugin will also be extended to support both silverstripe-module and silverstripe-theme support for expose, since now files in the base path may need to be exposed.

Both plugins will need support for reading .env files similar to read the configured public folder.

Any resources not in a composer dependency that need to be exposed will need to be comitted directly to the public folder.

If adding, removing, or changing the public folder, all migration is manual; The user will need to rename the folder and move any necessary assets into / out of the folder.

Advantages

• There is no duplication of files (except vendor resources)
• Files are pre-optimised for deployment; Most files will be in the correct location (e.g. public folder) after checkout from source control
• Files can have hard-coded paths, since public folder is fixed in source control. This can be useful for front-end build tooling (e.g. webpack)

Disadvantages

• Adding / Removing / Changing the public folder requires manual effort
• All environments must use the same public folder name (dev / testing / live)
• Project resources exposed to the web have to be committed in /public, which require themes to be split into two folders if adding it to the project directly.

[tractorcow]

Solution <<Pixies>>: public folder is a soft clone of exposed base path resources

Overview

The public folder is controlled by .env setting SS_PUBLIC_DIR, which is set by the installer project (before we did not provide a default .env file). This will default to public, but can be removed if necessary.

A public folder itself will not exist in source control, but is created automatically by a task (either ?flush task or composer vendor-expose) which builds these resources based on all composer-exposed resources as defined in either the project's composer.json, or all dependencies composer.json.

recipe-plugin will still install all project files to root.

vendor-plugin will be updated to support all module types, and also will expose root-project resources file expose config in the same way that silverstripe-vendormodule currently does.

Both plugins will need support for reading .env files similar to read the configured public folder.

When the user modifies the SS_PUBLIC_DIR value, the next time an update is done, the public directory will be rebuilt from all of these values. The update task in question could be either of (or both):

• A ?flush=all
• Running composer vendor-expose

Advantages

• Because public folder is buildable on deployment, you can have a separate public folder when developing locally, or on your release environment.
• "related" files (e.g. theme templates / public resources) can be physically co-located but still only have public assets exposed.
• You can make a file public / protected during development with a composer.json change instead of having to physically relocate it.

Disadvantages

• Because some files may need to be copied from root to the public folder, there can be extra disk storage used
• Preparing a checked out project for running can be slower, since it's necessary to build / copy these resources before the site can run

[kinglozzer]

<<Unicorns>>

Adding / Removing / Changing the public folder requires manual effort

Just to be clear, does “Adding / Removing” refer to opting in/out of the new “webroot subfolder” functionality? If so, I think requiring manual effort is fine - we have no way of knowing which files in someone’s project need to be exposed to the web and which need to be hidden.

If “changing” refers to having to manually rename the directory if you change SS_PUBLIC_DIR, I don’t see that as an issue at all - you’ll have to adjust your webserver config anyway, right?

<<Pixies>>

You can make a file public / protected during development with a composer.json change instead of having to physically relocate it.

What’s the use-case for this? Changing the root composer.json to expose a file? It’s faster and simpler to physically relocate it IMO.

---

I’m leaning toward <<Unicorns>> because my preference is for less “magic”

[dhensby]

I'd lean towards <<Unicorns>> too, but with modifications.

Configuration of webroot directory

I don't feel that the public dir should be configurable in the sense that the environment variable shouldn't declare the location/name of the public folder. I'd opt for convention over configuration for this. I struggle to see where it's appropriate to allow it to change based on an environment variable, these are hard configured into webserver configs (like apache or nginx) - change an environment variable won't update these server configs so I find it inappropriate that it should be dictated in such a way. I'd opt for having a simple boolean config somewhere or even just a detection such as "if there's a public folder then assume non-public project root".

Another way to think of this is: What use case is so common that it's worth the investment of time and complexity in our (and our users') tooling and infrastructure to allow dynamic setting of a webroot folder?

Version controlled public folders

There are files and folders that should live in the webroot and form part of a project (robots.txt, sitemap.xml, index.php, favicon.ico, ...). These files should be committed into a project's VCS just as the are now. If the public folder is not version controlled and is created dynamically on deployment then this adds an extra layer of complexity to storing and deploying these project files that logically should live in the public folder and not have to be specially copied or deployed.

Complexity / magic

I'm starting to get a little worried by how much preference we have for "magic" and trying to make everything just "work" without a developer doing anything. A default project that has a structure something like:

- app/
  - src/
    - ...
- cache/
- public/
  - assets/
  - resources/
  - .htaccess
  - index.php
- vendor/
  - ...
- compose.json
- compose.lock
- .gitignore

A project directory like this looks straightforward and manageable; I can also commit files to the public folder and not have to worry about some future magic that means they'll be ignored or deleted. I can commit my public folder and I get an expected outcome.

---

At time where I'm really hoping we can move away from idiosyncratic behaviour we seem to be brainstorming new ways of making SilverStripe more confusing and unexpected and "unique" and I'm not sure I can fully understand the motivation for it.

[andrewandante]

Haven't been able to think about this too hard today but I'm leaning towards Unicorns based on it being simplest and easiest to understand from a developers POV. Might edit this comment later once I've had a chance to dive in.

[unclecheese]

What’s the use-case for this? Changing the root composer.json to expose a file? It’s faster and simpler to physically relocate it IMO.

The reason the public/ folder is not in source control is because it's a byproduct of framework internals and/or composer tasks, not the developer. It's basically like a composer.lock file. There for viewing, but not for touching.

I'm leaning toward Unicorns, as well. I think @dhensby makes a good point about too much magic. It's a slippery slope. I think the recipe pattern, while I wouldn't call into question its benefits, sets a a precedent that makes more magic and idiosyncrasies feel more palatable, and we need to keep that in check. A developer coming into SilverStripe fresh, with no context or awareness of the motivation of history of this change is likely going to be really confused by a design like that. One of the things that really attracted me to SilverStripe as far back as 2.2 was that I could look at the directory structure and their contents and understand it straight away.

I disagree on Dan's point about making the folder name compulsory. Historically we've made few things like that compulsory, e.g. ASSETS_DIR, FRAMEWORK_DIR, THEMES_DIR, etc., and to leave PUBLIC_DIR out would feel like a glaring oversight, despite the use case being elusive. The use case for almost all those constants is elusive, but it's how we do it, and I would rather be consistent.

[tractorcow]

🦄 it is. :)

[tractorcow]

I don't feel that the public dir should be configurable in the sense that the environment variable shouldn't declare the location/name of the public folder. I'd opt for convention over configuration for this. I struggle to see where it's appropriate to allow it to change based on an environment variable, these are hard configured into webserver configs (like apache or nginx) - change an environment variable won't update these server configs so I find it inappropriate that it should be dictated in such a way. I'd opt for having a simple boolean config somewhere or even just a detection such as "if there's a public folder then assume non-public project root".

Another way to think of this is: What use case is so common that it's worth the investment of time and complexity in our (and our users') tooling and infrastructure to allow dynamic setting of a webroot folder?

The only challenge here is making it optional, and not breaking the "composability" of recipes. Remember that recipes both can be included into a project, but can also be a root project themselves. We'll need to solve the problem that both of these must work perfectly when there is or isn't a public folder, and also don't add a public folder unintentionally, but do create this by default for all new projects.

There are files and folders that should live in the webroot and form part of a project (robots.txt, sitemap.xml, index.php, favicon.ico, ...). These files should be committed into a project's VCS just as the are now. If the public folder is not version controlled and is created dynamically on deployment then this adds an extra layer of complexity to storing and deploying these project files that logically should live in the public folder and not have to be specially copied or deployed.

In the unicorns solution we will commit this to source control as you've described.

[flamerohr]

I'm leaning towards <<Unicorns>> there's less magic involved and makes more sense when first approaching it.
<<Pixies>> leaves a bit to be desired and feels less fleshed out compared to the other option, maybe it was because it's just a passing idea which I bounced off Damian last week...
it's a neat idea to stick to the paradigm of flushing - but I like my source control for the public root :D

Having a different webroot depending on environment could be a future enhancement.

[tractorcow]

Yeah, even though I was pushing pixies, forcing myself to critique and compare it to ingo's solution (unicorns) I started to see more and more flaws with it.

[tractorcow]

Proposed A/C and task changes:

Removed A/C / Tasks:

• Support custom naming of webroot folder
• Support web-root outside of BASE_PATH
• No automatic upgrade for existing projects

Added / changed A/C:

• Opting out on newly created projects is possible (may require manual effort) but the default is to use public folder.
• Folder is fixed to public folder. If the folder exists it will be used, and if it doesn't legacy non-public project is assumed.
  
  
  
  • Works with inlined themes (not composer dependencies) but these need to be manually installed to public folder if not using composer
  
  
  • Works with themes / non-vendor modules as composer dependencies (see https://github.com/silverstripe/vendor-plugin/issues/4) . May require per-modules changes to support 4.1, but should be trivial to upgrade a module for 4.1 support.
  
  
  • Documentation for adding / removing a public folder from a website is available.
  
  

New Tasks:

• [ ] Update recipe-plugin to conditionally move public files into public / root depending on project configuration.
• [ ] Ensure that new projects with all recipes (recipe-core, recipe-cms, installer) create public folder by default.
• [ ] Shift all public resources for all recipes (including recipe-core/index.php) into the public folder on each repo).
• [ ] Update vendor-plugin Auto-move resources/ into public/resources (read .env files in vendor-plugin?)
• [ ] Update vendor-plugin to expose resources for silverstripe-module and silverstripe-theme types as well.
• [ ] Update vendor-plugin to expose project resources not controlled by composer dependencies. E.g. to expose a custom-committed theme folder.
• [ ] Update module resource locator to work with files in public (e.g. public folder themes).
• [ ] Fix framework tests
• [ ] Test installer
• [ ] Test and fix IIS operation
• [ ] Adjust default gitignore paths
• [ ] Server installation docs
• [ ] Upgrading docs
• [ ] Check that #1384 has been addressed

[tractorcow]

Adding a new consideration; Modules that aren't managed by composer (either themes directly committed to the themes folder, or root-modules that have been added to source control) will need to have their exposed directories added to the root composer.json.

E.g.

{
    "extra": {
        "expose": [
            "themes/simple/style",
            "mymodule/css"
        ]
    }
}

The functionality will be provided by https://github.com/silverstripe/vendor-plugin/pull/10

We would extend the support of this plugin to these types:

• silverstripe-recipe
• silverstripe-module
• silverstripe-theme
• silverstripe-vendormodule

All of the above types (except silverstripe-vendormodule) would be skipped if not using a public folder.

Also, I would like to recommend that we use resources as the root for all symlinked / copied files; This means that a theme file themes/simple/style/style.css would be either symlinked or copied to public/resources/themes/simple/style/style.css.

Also a folder directly committed to root would be mapped as mymodule/css/style.css to resources/mymodule/css/style.css.

Also we will tweak the base path for vendor files to public/resources/vendor so that we have a measure of consistency within this folder.

[flamerohr]

public/resources/themes/simple/style/style.css

Would that mean that $ThemeDir would need to be aware of resources as well?

This does seem to lead down a slippery slope

[tractorcow]

Yep it will be. $ThemeDir will be conditionally resources/themes/mytheme (if using public folder) or just themes/mytheme (if not using public folder).

[unclecheese]

That all looks good so far. The only question I still have is if anything needs to happen on flush. One scenario I can imagine is that if my root module or inlined theme adds a new resource (or even changes one, depending on the enviroinment), the expose will need to be re-run. Are we mandating that the developer does this via composer, or should we also bake it into flush for convenience? I like the idea of doing it on flush because it's consistent with templates, another frontend resource.

[tractorcow]

The only question I still have is if anything needs to happen on flush.

You would need to run composer vendor-expose to update resources, but you have to do that anyway.

It would also happen automatically on composer update, I think. That would be a lot better right?

[chillu]

Great outcome, thanks for writing that up Damian! I've updated the ACs and tasks.

One scenario I can imagine is that if my root module or inlined theme adds a new resource (or even changes one, depending on the enviroinment), the expose will need to be re-run

That's only the case when you're using the "copy" method, and entries in the module's expose definitions are altered. We'd strongly encourage symlinks (or the Windows equivalent) for dev environments to avoid doing that every time you alter a file in an existing expose definition. If you really can't get symlinks working on your environment, there's existing file watcher tools you can chain to auto-execute this copy operation - that's out of scope for this issue though.

[chillu]

Works with inlined themes (not composer dependencies) but these need to be manually installed to public folder if not using composer

Just clarifying - we're not planning to put *.ss or *.scss files into public/ here, right?

Variation 1 (recommended): Store non-generated public files in public/, point frontend build tooling of generated files to public/ (incl. for inlined themes):

public/
  assets/
  css/
    main.css <- original file
  img/
    logo.png <- original file
  resources/
    myvendor/
      mymodule/ <- symlink
        extend.css
  .htaccess
  index.php
vendor/
  myvendor/
    mymodule/
      css/
        extend.css <- symlink
themes/
  inlinetheme/
    templates/
      Page.ss
    scss/
      main.scss
composer.json
webpack.config.js <- sets destination to public/css

Variation 2 (fallback): Use expose on inline themes

public/
  assets/
  resources/
    myvendor/
      mymodule/ <- symlink
        css/
          extend.css
    themes/
      inlinetheme/
        css/ <- symlink
          main.css
        img/ <- symlink
          logo.png
  .htaccess
  index.php
vendor/
  myvendor/
    mymodule/
      extend.css
themes/
  inlinetheme/
    templates/
      Page.ss
    scss/
      main.scss
    css/
      main.css <- original file
    img/
      logo.png <- original file
composer.json

[tractorcow]

Just clarifying - we're not planning to put *.ss or *.scss files into public/ here, right?

Well, it will be up to the devs. You can put your templates into non-public folder.

The next problem I have is how to ensure that the SSViewer.themes = ['mytheme'] config searches files both in the base path, as well as a separate public folder. Maybe a new pseudo-theme such as $public? It needs some thought.

However, these concerns are all PHP related... as far as the file structure is concerned, no we don't need to and should recommend leaving non-public assets out of public.

I definitely want to allow for your variation 2: We will need something like this in any case. The question is how we can make variation 1 work seamlessly.

I am thinking of something like:

Varation 1(b):

public/
  assets/
  resources/
    myvendor/
      mymodule/ <- symlink
        extend.css
  themes/
    inlinetheme/
      css/
         main.css <- original file
      img/
        logo.png <- original file
  .htaccess
  index.php
vendor/
  myvendor/
    mymodule/
      css/
        extend.css <- symlink
themes/
  inlinetheme/
    templates/
      Page.ss
    scss/
      main.scss
composer.json
webpack.config.js <- sets destination to public/css

Then you can reference these resources via inlinedtheme:css/main.css, or inlinedtheme:templates/Page.ss, where the internal resource system is able to look in both public and protected spaces, merging them together into one "logical" theme.

[tractorcow]

I've changed my mind... I don't think that we should merge public/themes/atheme into themes/atheme. We can just commit resources straight to public folder.

Instead I think we should add $public as a top level theme set, similar to $default. This will be an array of either [/public], or [] (if no public folder).

---
Name: mytheme
---
SilverStripe\View\SSViewer:
  themes:
    - '$public'
    - 'simple'
    - '$default'

You would add your resources directly to public/css or public/javascript instead. This would let you use themedCSS() with public folder, without needing a special theme structure.

You could of course ignore this, and just use <% require css() %> or <% require javascript() %> directly.

Another convention we should add is that you should never specify public/ as a resource prefix. Internally the resource loader should prefer that folder if it can find it, before it checks the base path.

[chillu]

Sounds good. My point in discussion with Damian was that many projects won't need themes to begin with, and we shouldn't force those semantics on them. It's perfectly fine to write <link rel=stylesheet href=css/main.css> if devs don't want to deal with themes at all. If they want to use themedCSS() to refer to "abstract" file locations (rather than a single concrete path), we should make it easy to do that with the same semantics (themedCSS('css/main.css')). Damian's inclusion order proposed above solves that.

[tractorcow]

I'm starting work on a PoC update of vendor-plugin and recipe-plugin now. ;)

[tractorcow]

WIP branches:

• https://github.com/open-sausages/vendor-plugin/tree/pulls/1/throwing-a-public
• https://github.com/open-sausages/recipe-plugin/tree/pulls/1/throwing-a-public
• https://github.com/open-sausages/silverstripe-framework/tree/pulls/4/throwing-a-public
• https://github.com/open-sausages/silverstripe-installer/tree/pulls/4/throwing-a-public
• https://github.com/open-sausages/silverstripe-serve/tree/pulls/2/throwing-a-public
• https://github.com/open-sausages/silverstripe-assets/tree/pulls/1/throwing-a-public
• https://github.com/open-sausages/silverstripe-cms/tree/pulls/4/throwing-a-public
• https://github.com/open-sausages/recipe-cms/tree/pulls/1/throwing-a-public
• https://github.com/open-sausages/recipe-core/tree/pulls/1/throwing-a-public

• https://github.com/open-sausages/silverstripe-simple/tree/pulls/3.2/add-exposes

  Install with:

  {
  "require": {
      "php": ">=5.6.0",
      "silverstripe/recipe-plugin": "dev-pulls/1/throwing-a-public as 1.1.0",
      "silverstripe/vendor-plugin": "dev-pulls/1/throwing-a-public as 1.3.0",
      "silverstripe/recipe-core": "dev-pulls/1/throwing-a-public as 1.1.x-dev",
      "silverstripe/recipe-cms": "dev-pulls/1/throwing-a-public as 1.1.x-dev",
      "silverstripe/assets": "dev-pulls/1/throwing-a-public as 1.1.x-dev",
      "silverstripe/framework": "dev-pulls/4/throwing-a-public as 4.1.x-dev",
      "silverstripe/cms": "dev-pulls/4/throwing-a-public as 4.1.x-dev",
      "silverstripe-themes/simple": "dev-pulls/3.2/add-exposes as 3.2.0",
      "silverstripe/serve": "dev-pulls/2/throwing-a-public as 2.0.0"
  },
  "repositories": {
      "recipe-plugin": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/recipe-plugin.git"
      },
      "vendor-plugin": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/vendor-plugin.git"
      },
      "recipe-core": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/recipe-core.git"
      },
      "recipe-cms": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/recipe-cms.git"
      },
      "assets": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/silverstripe-assets.git"
      },
      "framework": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/silverstripe-framework.git"
      },
      "cms": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/silverstripe-cms.git"
      },
      "simple": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/silverstripe-simple.git"
      },
      "serve": {
          "type": "vcs",
          "url": "https://github.com/open-sausages/silverstripe-serve.git"
      }
  }
  }
  

[tractorcow]

Ok, I've pushed some extra functionality to vendor-plugin to support more module types and also install to the public folder.

I still need to manually merge in https://github.com/silverstripe/vendor-plugin/pull/10, and also update the composer tasks.

[dhensby]

• Works with inlined themes (not composer dependencies) but these need to be manually installed to public folder if not using composer

Do they need to be manually installed to public if not using composer? I think we should be able to add arbitrary "expose" files to our root composer.json and then those will be exposed for us... It sounds like that's what's going to happen anyway so this AC appears to contradict the root composer exposing.

• [ ] Update vendor-plugin Auto-move resources/ into public/resources (read .env files in vendor-plugin?)

Do we need to read .env files? Don't we just need to check for presence of public?

All of the above types (except silverstripe-vendormodule) would be skipped if not using a public folder.

I feel it's simpler (at least logically) if we always expose all files always - especially from a tooling point of view. If I want to set up a CDN against my site it's much easier to say resources/* that list out all my legacy modules (which might change to vendormodule later) - it also makes resolution logic more simple I'd have thought?

[tractorcow]

Do they need to be manually installed to public if not using composer? I think we should be able to add arbitrary "expose" files to our root composer.json and then those will be exposed for us... It sounds like that's what's going to happen anyway so this AC appears to contradict the root composer exposing.

Yeah that'll be an option of course. I'll use your vendor-plugin PR to do that. We should prefer users commit directly though as convention.

Do we need to read .env files? Don't we just need to check for presence of public?

Obsolete task... we just detect public folder now.

I'm updating the task / plugin to just work on anything silverstripe-* instead of silverstripe-vendormodule specifically.

I feel it's simpler (at least logically) if we always expose all files always - especially from a tooling point of view. If I want to set up a CDN against my site it's much easier to say resources/* that list out all my legacy modules (which might change to vendormodule later) - it also makes resolution logic more simple I'd have thought?

I think just moving everything to vendor module fixes that right?

[tractorcow]

ok, plugins are now updated on those working branches. Updating tasks. :)

[tractorcow]

Note on this A/C:

• Update module resource locator to work with files in public (e.g. public folder themes).

Module resources aren't in public anymore, but the SimpleResourceURLGenerator has been updated to link to the symlinked location now. :)

[unclecheese]

Framework tests still failing, to address after break:

• RequirementsTest
• HTMLEditorFieldTest
• SSViewerTest

[tractorcow]

We probably want a mkdir public in .travis.yml as well.

[tractorcow]

Here are the failing framework tests: https://gist.github.com/tractorcow/502319508262d6e6946f02bc3339a302

[tractorcow]

Tests green so far. :)

[tractorcow]

Ok, all pull requests are up... I need to finish windows testing though.

[tractorcow]

Finally, I got a windows image working and tested it (albeit with sqlite3 only).

See https://github.com/tractorcow/silverstripe-vagrant-win for the vagrant image I used.

[bevanshaw]

Been reading these comments and wondering about the following use case: React project (containing SVG files) nested inside a SIlverStripe project. I have been using npm to create build files for my React project which automatically produces three folders in the build directory static/js, static/css and static/media (containing the SVG files). The first two folders static/js and static/css can be dealt with by using the Requirements class on the , adding the paths to the composer.json file and running composer vendor-expose. The media file is not so easy. Because the script npm run build automatically configures the relative path /static/media/icon.svg, it means I have to place a copy of the static/media folder manually in the public directory of my SS4 project. I want to place the svg files in the public/resources folder. But then I can point the virtual host at that folder without losing access to the symlinks for the static/js and static/css build folders. I am new to all this and have been advised to try writing a DOS batch script to automatically do this to avoid having to manually do this. This then creates new secrurity concerns and left me wondering why is this just not easier? Eventually my research led me here :-). Any thoughts about this would be great.

[robbieaverill]

I have to place a copy of the static/media folder manually in the public directory of my SS4 project

Is that a problem?

You can always inject props into your React app, such as a media directory which is determined in your backend

[bevanshaw]

What would that look like @robbieaverill? Would it mean creating the right folder structure inside the React project so that the build of the React project would result in a different path (i.e. not build/static/media pointing at URL/static/media)? Not sure I understand how a prop could be used to alter the build result?

[robbieaverill]

@bevanshaw I mean static/media/yourfile.jpg becomes a prop you pass into your React app, then you determine that path from the backend (SilverStripe app). Please feel free to ping me on the community Slack channel, or ask a question on StackOverflow if you need any more help. Good luck!

[bevanshaw]

Thanks a lot Robbie, I will have a think about your suggestion and post a question on StackOverflow if I am having trouble :-)

[bevanshaw]

Posted related question: let me know if it is too broad. Cheers
https://stackoverflow.com/questions/56052324/how-to-build-a-react-app-containing-svg-files-inside-a-silverstripe-project-so