Magento2: Plan for media storage API?

[owenhaynes]

Is there any plan for a media stroage api, was looking to see if it was possible to use S3/Cloud Stroage for media assets. So that Elastic Beanstalk can be used effectively.

I don't think storing the images in the database is the correct option when S3 storage is far cheaper and faster to run then a database.

Have had a look at the code and it seems very tightly coupled with File and Database, so creating a plugin seems a lot of work.

[ooxi]

:+1:

[alankent]

I will let the product folks comment in roadmap, but curious as to benefits of storing directly in s3 vs storing in DB and letting CDN take the load. So performance of DB not so critical.

(There is no big effort planned here internally that I can recall, but always open to ideas about putting in interfaces etc making it easier to replace our default functionality with other functionality via di.xml - so feel free to suggest where you would put such an abstraction layer. Start with comments, can see if any feedback, then move on to a pull request if you are keen. This is not specific to this request - this is true of making any replacement point easier. Suggest, confirm, code.)

[owenhaynes]

S3 is a lot simpler to set up for a small store, just looking at Amazon CDN looks like a bit of effort to set up. We also by using a database have the extra memory foot print in php by having to load the image into memory.

The MediaStorage module looks partway there to being an API, it just looks like some of it could do with be re-factored so that its more generic.

Most likely need an interface for the admin section for any set up if may require, as well as the interface for serving the files.

I think the biggest issue is that
MagentoMediaStorageHelperFileStorageDatabase
is used all over the place for things like order documents and admin data?

I understand them being need to be stored in the database, but this feel like it confuses order system data with public data, so would have to some how refactor this out in the process, so that media storage only covers public media.

[kandy]

Looks like simplest way to implement s3 in m2 is s3 stream wrapper and configuration media dir as s3://media/

PS: i don't try this

[jameshalsall]

Any news on this?

[Khaleel]

What you do for NGINX is create a directory and origin pull that directory. Example

https://subdomain.shop.com

This in vhost has a web directory of

/var/www/magento/htdocs/media

So Magento2 will always upload to the media folder - you can do the same (I think for pub folder for JS and CSS)

Thus in the CDN just make an origin pull with a cname

https://subdomain.shop.com/media/wysiwyg/sample.jpg
https://subdomain.shop.com/pub/media/catalog/product/8/7/871209028788_1.jpg

Then in Magento under the Admin > Stores -> Configuration -> Web.

Just update your media URLs

[craigcarnell]

I have the same issue except we use Rackspace CDN. For m1 I modified a free magento extension to do this (onepica) but that never got released for m2.

[Khaleel]

I am also using Rackspace CDN...

[craigcarnell]

@Khaleel The onepica module uploaded the files to the CDN using the API available for Rackspace. I modified it to work with the UK version

[Khaleel]

Sounds good. I too am using Rackspace UK as LON region. Also international API. It would have been nice to have official support from the Magento platform for CDN integration rather than modules. Varnish (personally not a fan) and things like caching come out of box. CDN should be default now across Enterprise.

[jameshalsall]

I'm working on a module for Magento2 that will offer S3, Google Cloud Storage for assets. Will post it here when it's done.

Sent from my iPhone

On 3 Feb 2016, at 10:23, Khaleel ibn Anwar Mughal [email protected] wrote:

Sounds good. I too am using Rackspace UK as LON region. Also international API. It would have been nice to have official support from the Magento platform for CDN integration rather than modules. Varnish (personally not a fan) and things like caching come out of box. CDN should be default now across Enterprise.

—
Reply to this email directly or view it on GitHub.

[ostark]

@jameshalsall +1

[craigcarnell]

@jameshalsall Sounds good, maybe I can extend it for Rackspace

[amansrivastava]

@jameshalsall Looking forward for the implementation. This will help me alot in using S3. Thanks in advance. :)

[jameshalsall]

@alankent @mazhalai

Whilst I think this is "achievable" with a 3rd party module through the combination of plugin code and completely extending core Magento classes and replacing them through DI, I don't think this is a good solution. Currently the Magento\MediaStorage\Helper\File\Storage\Database class does not implement any interface meaning that it's difficult to know exactly what needs to be implemented in a custom storage layer.

I'm not even sure why it's referred to as database storage when all of the files reside on disk, not in the database. The only information stored in the database are the asset paths relative to the media path, and this would continue to be the case if we swapped it out for cloud storage alternatives. So "Database" is a poor name for this class.

To allow a bespoke storage layer we will need interfaces creating that will allow us to define logic for

• creation of assets
• deletion of assets
• modification of assets
• listing of assets

Everything in Magento\MediaStorage\Helper\File\Storage\Database only handles the management of assets, not the fetching of them. To customise behaviour of fetching assets, I can see a couple of places that would need to be refactored for images (although I could be wrong so please correct me) but haven't found anything on generic file assets on my travels through M2 yet:

• \Magento\Catalog\Block\Product\ImageBuilder::create()
• \Magento\Catalog\Helper\Image::getUrl()

Again there needs to be a clear set of interfaces defined that handle the fetching of assets which would allow us to customise this behaviour. The workflow I was hoping to easily implement in a module would be:

1. Check if asset exists locally
2. If not, fetch it from storage bucket e.g. S3
3. Cache asset locally
4. Return local URL for asset

@alankent Your suggestion of just using a CDN doesn't really make sense, how is a Magento2 store currently meant to scale horizontally if all of the assets reside on phyiscal disk? The only easy solutions I can see right now are:

• s3:// stream wrappers which makes for an extremely slow store
• shared disk services like gluster or EFS (which still are not ideal)

Please let me know if I'm missing something obvious here.

I look forward to hearing your thoughts

[thaiphan]

Hey everyone,

Here is an extension that I've built that allows you to sync all your Magento 2 media files, e.g. WYSIWYG images, product images, category images, to Amazon S3.

https://github.com/thaiphan/magento2-s3

Regards,

Thai

[craigcarnell]

@thaiphan Thanks for that. If a file is not on the CDN does it still fall back to media?

[thaiphan]

@craigcarnell

There are actually different setups that you can use that allows fallback to media.

CloudFront

If you're using CloudFront then you can set different origin servers for your images. One origin server can be S3 (which Magento should automatically sync newly uploaded images to) and then you can set up Magento to be your second origin server for things like thumbnails (which needs to first be generated via Magento before being automatically synced to S3).

S3

If you're not using CloudFront then you can use S3's routing rules to allow you to fallback to Magento if a object doesn't exist in the bucket. Here's a quick article explaining how the routing rule engine works:

http://hipsterdevblog.com/blog/2014/06/22/lazy-processing-images-using-s3-and-redirection-rules/

Magento

Another method is if you're routing your media files through get.php then Magento will download an image from S3 if it doesn't already exist on the server.

I hope that was helpful!

[mrkhoa99]

@thaiphan +1

[peec]

@thaipan thanks, tested it and it works. Helpful hint is also just to set media url under Store -> General -> Media URL unsecure to S3 bucket or CDN url.

Core Team, are you going to officially support S3 and other storages ? Magento2 can't scale using regular file system storage.

[owenhaynes]

let alone scale with kubernetes or other docker containers management services

[piotrekkaminski]

Thank you for your submission.

We recently made some changes to the way we process GitHub submissions to more quickly identify and respond to core code issues.

Feature Requests and Improvements should now be submitted to the new Magento 2 Feature Requests and Improvements forum (see details here).

We are closing this GitHub ticket and have moved your request to the new forum.