Server: Integration with Azure blob storage

[shyamal890]

Will NextCloud support Azure blob storage as external storage in near future?

[MariusBluem]

Hm ... Let's put this as an enhancement. However - I do not see this in the near future 😁 ... Maybe you want to step in - or know somebody who wants so 😅

We have some external storage backends in Nextcloud, (Amazon) S3 seems to be finished for (primary) storage for Nextcloud 11: https://github.com/nextcloud/server/pull/768

This may be an alternative ... Sharepoint is also on the roadmap for 11, if you rely on Microsoft 😁

cc @icewind1991 @blizzz

[pascalsimi]

I find it rather puzzling that blob is not supported.
If you look at the gartner quadrant then there is Amazon and Microsoft as the big players.
Supporting only one of them cuts down potential user base by approximately 50%. Take into account how many personal computers actually run on Microsoft it's even more puzzling to me why Microsoft blob is not on the roadmap. But hey its just my thoughts. Thanks for all the hard work you put into next cloud. All you guys are awesome!

[gjonespf]

Would love to see this also. May be worth mentioning that (and I haven't tested this) but Azure File Store allows SMB access, so you could potentially run Nextcloud on an Azure VM with external storage mapped to the Azure File Store, as long as the VM and storage were in the same region (otherwise you need SMB3 encryption support, which isn't there yet in Linux). Something I will have to investigate.

[pascalsimi]

I have a few projects with Microsoft azure for processing and nextcloud for storage and sync right now and with their new push for openness on azure I think there is a good chance their evangelists might be interested in helping to implement that feature.
I'll mention it in the next meeting.

[pilcherd]

Support for Azure Blob would be good. Interesting about Azure File Store and SMB. Will look into that.

[MariusBluem]

https://github.com/icewind1991/files_external_azure

Digging for old code 😜😜 What was the state there @icewind1991 🤔🙈

[icewind1991]

Not working atm, will see if I can fix it when I get some time

[strategywise]

We're also interested in this ability. If the SMB mapping on the host box works would it in theory at least be able to read/write to the blob? If I have to get a blob mounted on the VM and it works well I don't think that would be a very hard work around for the time being.

[SpiraMirabilis]

I've been tinkering with this, using the S3 backend as a template. It's a bit of a work in progress. However, as a temporary measure you can use a Windows server (2008 or greater), with WebDAV authoring role for IIS and essentially act a proxy in between your users and azure blob storage, see this how-to: http://fabriccontroller.net/mounting-your-azure-file-shares-on-premises-through-webdav/ Edit: Actually, it looks like that only supports the Azure Files which is much more expensive than blob.

Should it support serving from the verizon/akamai CDN if it is linked to that storage account?

Also, either nextcloud will need dedicated access to the blob container, or maybe we can use a Service Bus subscription and just mandate that anyone adding/changing/removing a file triggers a message sent, that way we can confidently cache the metadata/contents and there wouldnt be polling which increases cost a lot.

[MorrisJobke]

This should be definitely be implemented as an app - see the files_external_dropbox external storage for example: https://github.com/icewind1991/files_external_dropbox

[CyrilHupor]

+1 on the feature which I believe is very important.

I can't however understand why this support should be an app when out of the box support is already included for AWS and OpenStack. Keep in mind Azure is leader together with AWS in the cloud space. I think this matter should be re-visited.

Btw, the reason as to why Blow Storage might be considered as opposed to Azure Files or additional storage on the VM is simply cost. Blob storage is much cheaper.

On another note, congrats to dev's on nextcloud, would just very much like support for Azure Blob.

[jbraes]

I am always amazed how people use software is such different ways.
running your "personal" cloud on-top of "public" cloud infrastructure.
Isn't the idea of having your own cloud infrastructure , to be independent of azure, Amazon , google etc , to have full controle over your hardware

[MourIdri]

+1

[wvh-shanee]

+1

@jbraes with e2e encryption the public cloud just ends up as a big storage location to stash encrypted files. Some prefer Azure to Amazon so it would be nice to support multiple storage options. I personally like the public cloud because of automatic backups, geographic redundancy, unlimited storage, etc. Running my own box at the house (or a VPS in the cloud) just adds overhead, because I then I have to deal with redundancy and backup myself. My father-in-law is tired of me swapping hard drives out of his gun safe. ;)

[SpiraMirabilis]

It adds additional flexibility. If you are running out of space and don’t
want to upgrade your server, you can easily leverage cloud platforms where
storage is cheap. If you are running into capacity issues with regards to
your servers bandwidth you can segregate highly popular files into a cloud
service so they are served without utilizing your servers bandwidth,
utilizing a cloud service can ensure important files are backed up offsite
based on the files taxonomy, or even backed up using AWS glacier so that
it’s impossible to change (for compliance/legal/preservation of document
reasons) based on taxonomy.

Hybrid solutions integrating a baseline of owned hardware with a backup of
cloud services for either load balancing, backup or reliability is in my
opinion a way to very efficiently leverage cloud services very cheaply.

On Thursday, March 15, 2018, jbraes [e-mail] wrote:

I am always amazed how people use software is such different ways.
running your "personal" cloud on-top of "public" cloud infrastructure.
Isn't the idea of having your own cloud infrastructure , to be independent
of azure, Amazon , google etc ...

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/server/issues/2027#issuecomment-373536747,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABf2gOLiAqEqvL8GRrMmxHIB1FoyLCznks5teuSegaJpZM4KqhH3
.

[ohthehugemanatee]

@MorrisJobke Can you give some insight into your decision process here? Nextcloud supports AWS and Openstack out of the box. Why not the number 2 biggest cloud provider?

[MorrisJobke]

@MorrisJobke Can you give some insight into your decision process here? Nextcloud supports AWS and Openstack out of the box. Why not the number 2 biggest cloud provider?

As this sounds like a nice feature, there are currently no plans to implement such a feature. This does not mean we don't want this feature, but it is simply not on our roadmap for the near future. If somebody wants to implement this feature nevertheless we are happy to assist and help out. I hope that explains a bit why it isn't there yet.

If you wish to have this feature implemented by the Nextcloud GmbH there is the option for consulting work on top of your Nextcloud Enterprise subscription to get your features implemented.

[powareverb]

Thanks for the clarification @MorrisJobke is what I figured. You guys only have so much time in the day. I'll be looking into Minio on blob storage, as the s3 backend for Nextcloud seems more stable now. Might be worth looking at for others.
https://azuremarketplace.microsoft.com/en-us/marketplace/apps/minio.minioblob

[ohthehugemanatee]

Thanks for the clarification.

For those who are curious, there are ways to mount Azure Blob storage directly on your VM or in k8s. That's what I ended up doing. Honestly if you're going to use it as primary storage, this is a better option than implementing in the application layer IMO.

@see

• BlobFUSE driver howto
• BlobFUSE driver for Kubernetes
• Dysk project "Attach Azure disks in < 1 second. Attach as many as you want. Attach them where ever you want. dysk mounts Azure disks as Linux block devices directly on VMs without dependency on the host."

Hope this unblocks someone else in the future!

[z2z]

Maybe in future, use of https://flysystem.thephpleague.com/docs/ it already has a lot of adapters.

[mstenz]

also instead of BlobFUSE you could potentially map Azure Files with SMB directoy to the host vm.
But as you can mount it only with one access rights you need to fix some error.

Please change the permissions to 0770 so that the directory cannot be listed by other users.

f(\OC::$server->getConfig()->getSystemValue('check_data_directory_permissions', true) === false) {

I have not checked further but i think you can change the falue to false in the configuration which will fix this issue, but as i have not yet done it (still in the process) there may be issues afterwards.

[MourIdri]

It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other)

[SpiraMirabilis]

Interesting. I wonder how efficient it is with regards to unnecessary
metadata requests which Azure charges for. Ideally if you’re using blob you
would cache the metadata aggressively for that reason.

On Wed, Jan 8, 2020 at 3:44 AM chapeau [e-mail] wrote:

[image: AZUREBLOBNFS3]
https://user-images.githubusercontent.com/19348103/71972009-1ca82780-320c-11ea-99aa-ab571c21f699.JPG

It seems that it is possible to mount Blob natively via NFS now. Cheaper
and easier than object storage (Blob, S3 or other)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/server/issues/2027?email_source=notifications&email_token=AAL7NADARSB62Z2HAHAUHQ3Q4WVATA5CNFSM4CVKCH32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIL6ZZQ#issuecomment-571993318,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAL7NAEVI6GUV7I4BDLQXBLQ4WVATANCNFSM4CVKCH3Q
.

[MourIdri]

Interesting. I wonder how efficient it is with regards to unnecessary metadata requests which Azure charges for. Ideally if you’re using blob you would cache the metadata aggressively for that reason.
…
On Wed, Jan 8, 2020 at 3:44 AM chapeau @.*> wrote: [image: AZUREBLOBNFS3] https://user-images.githubusercontent.com/19348103/71972009-1ca82780-320c-11ea-99aa-ab571c21f699.JPG It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2027?email_source=notifications&email_token=AAL7NADARSB62Z2HAHAUHQ3Q4WVATA5CNFSM4CVKCH32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIL6ZZQ#issuecomment-571993318>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7NAEVI6GUV7I4BDLQXBLQ4WVATANCNFSM4CVKCH3Q .

It is all about $$$ I guess, I use to have a 28 Millions requests per week on my S3 bucket for around 1000 files. In that case you just get the data and the metadata will be limited to files's GID/UID and embedded in the offering. Let's see .