Server: Move into the future: Replace PHP with Golang.

[Wehzie]

Replace PHP with Golang

Something is happening: ownCloud replaces PHP with Golang

In this interview in German language the ownCloud CEO's Gerlinger and Dyroff discuss why ownCloud is replacing its PHP architecture with Golang.

What problems does Golang solve?

Golang brings speed improvements; these are not only demonstrated by benchmarks but also by the file sync app Syncthing which is built on Golang. PHP has problems dealing with large files. Hopefully, Golang will also increase stability.

How should Nextcloud react?

The ownCloud CEOs hint at a continued development of the PHP based ownCloud X/10 until 2020. It is unclear from the article when a new Golang based version will be available. The ownCloud Golang implementation will be released under an Apache 2.0 license. To me a main question arises: Can Nextcloud wait until the ownCloud code is available?

Alternative

Alternatively, if Nextcloud can't wait on ownCloud an own development process could be started. From a commercial perspective I understand that it is imperative to employ engineers that built the backend to support paying customers.

Conclusion

I don't believe that a PHP based Nextcloud can survive next to an ownCloud built on a Golang backend. While more things than fast file synchronization form the Nextcloud ecosystem, speed and reliability of core services must form the foundation. Maintaining speed and reliability into the future is supported more by Golang than it is by PHP.

Edit, 31. January 2020:
The ownCloud Golang project is called ownCloud Infinite Scale.
The main GitHub repo is found under https://github.com/owncloud/ocis

[aignerat]

Hey Wehzie,

that's very interesting, I believe the fastest approach is to go modular. Determine where PHP has its weaknesses in Nextcloud and try to translate that packages to Golang.

A similar approach was made by ownCloud for the WOPI-integration, they use a phyton-middleware for filelocking, fileoperations and communication (REST/SOAP) with the WOPI-Server and the PHP-app, a PHP-app for configuration and communication with the phyton-middleware and the filestorage.

Regards chris

[clementduveau]

Hi,

Nonetheless, Go is way less popular than PHP. Switching language can also reduce dramatically the community able / that want to maintain Nextcloud because, all of the sudden, everything is different.

Therefore, taking the risk to switch in order to improve Nextcloud could be the best choice on a long term.

I am just exposing my opinion.

Regards,

[tobiasKaminsky]

Additionally one of the main benefits of Nextcloud is that it can be run (for home users) on nearly every web space / system.
This will not be true if we switch to Golang.

[Wehzie]

Thank you for your replies,

@tobiasKaminsky The minimum requirements of Golang show that system flexibility is given. I don't think the additional flexibility of PHP is needed and I don't think that it outweighs the performance of Golang.

I have raised this discussion on Golang because ownCloud is going to rewrite their code, not because it was my idea to rewrite Nextcloud. There is going to be Golang code one way or another. This move by ownCloud is going to define the future relationship between the two companies' communities as code bases drift apart. I created this issue to discuss how we react.

[aignerat]

Conclusion

I don't believe that a PHP based Nextcloud can survive next to an ownCloud built on a Golang backend. While more things than fast file synchronization form the Nextcloud ecosystem, speed and reliability of core services must form the foundation. Maintaining speed and reliability into the future is supported more by Golang than it is by PHP.

I still think you are exaggarating on this point of view. Unquestionable golang is faster and used in many projects, but PHP is fast enough for most usecases of nextcloud. I think owncloud is trying to
rush too fast. I don't think it's possible to get a fully converted golang-version of owncloud ready in 2020. Additionally the challenge consists of PHP-developers changing to golang, that fact alone lowers productivity for month as Tobias mentioned, not even considering the lower availability of golang-developers.

[ben423423n32j14e]

Can't say I agree with replacing PHP with Golang, I actually think it could be very bad for the NextCloud project since there are so many people already experienced with PHP but I have never heard of Golang until I read this thread! My opinion is stay with the language where the most people are able to contribute.

[helmut72]

@Wehzie Thank you for writing down my thoughts.

PHP is fast enough for most usecases of nextcloud

I'm unsure if PHP or any other Scripting language is a good core for fast reliable file handling. Syncthing flies on a Raspberry compared to Nextcloud.

Additionally one of the main benefits of Nextcloud is that it can be run (for home users) on nearly every web space / system.

This only matters on cloud space, when no binary can be executed. But the focus of Nextcloud is privacy and the home users I know, install Nextcloud on their own Hardware.
Current restrictions of a web space hoster shouldn't be a reason not to think about the future. In early years, web space hoster also doesn't provide what is now standard.

This will not be true if we switch to Golang.

Many Golang projects only need one binary to start and most of them runs better and feel more stable on small Hardware compared to their alternatives, for example Gitea vs Gitlab, Miniflux vs TT-RSS.

[small1]

Switching language will in the end make the project to loose developers. Everyone in the project now knows php. And for most us cases as said before php is all that is needed.

Rewriting everything is a HUGE and complex project that will go on with less developerst than before .. so according to me its not something to even consider. The benefits is to few for such a high cost and penalty that comes with it.

[wucke13]

I'm a seafile user for many years, since I switched from owncloud in the Version 6 days due to very slow sync speed. Seafile was absolutely superior back then, regarding sync speed. In the last months I had more and more trouble with seafile, causing me to research the current state of alternatives. From the moment of the owncloud annoucement onwards, I'm seriously consider moving as soon as it will be feasible to use their golang backend.

I'm sure of these two things:

1. PHP is a really bad choice for writing a storage back-end. It is amazing how far nextcloud (and owncloud) have come performance wise, though it is (and was) far below the bar set by seafile or syncthing (or any other storage/sync software, which offers persistent server processes).
2. Golang is very easy. Like, too easy. I've written small service oriented apps in days, without prior experience in go.

Is PHP really that important for nextcloud devs? According to github, PHP makes up almost 60% of nextcloud, but almost 40% is JavaScript. If certain core functionalities are rewritten in Golang, offering nice AJAX/JSON/Whatever web-API's, a fair amount of the code would still be in the same programming language as before (JS). And I would assume that browser sided functionality and UI will change far more, so once there is a core in golang, there will be few reasons for most developers to touch the core.

TL;DR I will never again use a file synchronization service with a PHP core. But I would be happy to use nextcloud/owncloud in a few years!

[aignerat]

I'm a seafile user for many years, since I switched from owncloud in the Version 6 days due to very slow sync speed. Seafile was absolutely superior back then, regarding sync speed. In the last months I had more and more trouble with seafile, causing me to research the current state of alternatives. From the moment of the owncloud annoucement onwards, I'm seriously consider moving as soon as it will be feasible to use their golang backend.

I'm sure of these two things:

1. PHP is a really bad choice for writing a storage back-end. It is amazing how far nextcloud (and owncloud) have come performance wise, though it is (and was) far below the bar set by seafile or syncthing (or any other storage/sync software, which offers persistent server processes).
2. Golang is very easy. Like, too easy. I've written small service oriented apps in days, without prior experience in go.

Is PHP really that important for nextcloud devs? According to github, PHP makes up almost 60% of nextcloud, but almost 40% is JavaScript. If certain core functionalities are rewritten in Golang, offering nice AJAX/JSON/Whatever web-API's, a fair amount of the code would still be in the same programming language as before (JS). And I would assume that browser sided functionality and UI will change far more, so once there is a core in golang, there will be few reasons for most developers to touch the core.

TL;DR I will never again use a file synchronization service with a PHP core. But I would be happy to use nextcloud/owncloud in a few years!

Hey wucke,

thx for your reply, you're right in one point, PHP is slow on fileoperations compared to golang. I still believe switching to golang is a high risk as PHP is fast enough for most operations and there are so many functionalities and apis that have to be considered and tested again. Nextcloud says they have 6.6 million lines of code, according to your numbers it would be almost 4 million loc that have to be rewritten. The core alone has 50k LOC.

I just don't see the urgent need to keep everything bleeding edge, just take a look at facebook, they use partly compiled PHP, maybe that's a way to consider instead of switching a huge project to golang. I personally think that owncloud does the step to go because they lose to nextcloud-functionality in so many other points.

regards chris

[wucke13]

high risk

Definitely.

I personally think that owncloud does the step to go because they lose to nextcloud-functionality in so many other points.

Sounds reasonable to me! This really is about making choices. I'm not interested in having an Eierlegende Wollmilchsau, I want solid file synchronization with versioning and a way to share/collaborate with others. Currently, there is only one open source competitor in this area that I'm aware of (seafile). Personally, I do consider some of the functionality you mentioned bloat. The market for lesser bloated file sync and share services is pretty small. The nextcloud project will have to decide how they prioritize the needs of their customers. I think nextcloud offers the better product currently, but I'm not willing to use either one of those (nextcloud/owncloud) due to my disbelief in the fundamental design approach. If owncloud will offer a similar feature set as back in the version 5 days on a high performance core, that will eventually cause me to switch. So all the extra functionality, and a fair amount of code is pointless (to me). Please don't misunderstand this, I'm in no way part of either owncloud or nextcloud community, and this is only my personal opinion on that. There is no urge to follow it in any way, just take a note that there is at least one user who is considering the php everywhere as a strict deal-breaker, but who doesn't consider most of your features (which hopefully is what makes up most of the code) as necessary.

As a small side note: maybe it is not going to be necessary to rewrite all the php code. Seafile for example uses a file storage daemon (called seafile) which only handles the libraries of persons, file sync, and the storage-backend. The whole web ui (called seahub, which is the part with heavy development and >90% of the changes in the last years) is written in python/django. So maybe it is feasible to only replace the 50k LOC of the core without replacing everything which is php. If this holds true, most of the arguments about loosing all the php devs and the work which was already done will loose in weight. A stricter separation in core and web ui might help to optimize the architecture and to implement stricter decomposition, which in general may increase code quality. For example running only core to do file sync between clients but not web ui would be an interesting use case?

Whatever it is that nextcloud will do regarding this topic, I wish you good look on your path!

[helmut72]

PHP is slow on fileoperations compared to golang

But file operation is the core feature of Nextcloud. This should be fast and rock-solid. Nothing against Nextcloud, they do a great job, but because of PHP file operations feels sometimes sluggish.
And it's not only core file operations. Just compare Roundcube Mail with Nextclouds Mail Addon. Speed difference is HUGE.

just take a look at facebook, they use partly compiled PHP

I guess no one want to run Facebook clusters. At least for me. I don't have a free power source at home.

Nextcloud says they have 6.6 million lines of code, according to your numbers it would be almost 4 million loc that have to be rewritten. The core alone has 50k LOC.

Reminds me of "to big to fail". In 5 years they maybe have 20 million lines of code. Nextcloud/Owncloud started in a time where PHP was one of the common solution. Now it's soon 2020.

[aignerat]

I just try to bring in my experience and point of view to give a good overview to everyone for further discussions.

I totally got your point Helmut. I analysed the code myself several times, I couldn't find the 6,6 mio LOCs, I only found ~ 1.5 mio, I guess the rest ist summed up by the ~100 apps that I never use. It's still important to know that the files "app" is the core-functionality that handles most of the fileoperations.

The challenge I see with golang is the communication with the PHP-libaries, there are so many 3rd-party-extensions to make everything possible, I don't think that most of these libraries are available in go yet. I seriously doubt that rewriting this code to enable it to be used as API is the way to go. Now we talk about ~ 300k LOC. I would be really impressed if owncloud manages to rewrite everything that's really needed in GO, I prefer the refactor-on-demand approach of NC. Changing a whole system of this size at once eats up a bunch of manpower.

[5dao]

1, install PHP runtime is so cumbersome,especially on my nas !

but ,go bin run anywhere!,,,on my pc,on my old notebook,on my nas,on my wifi router,,,,

2,php version update is so terrible,,,,,php 7, on my pc win7,,so bad,,,on my old ubunut 14 so bad ,

bug ,go bin from win2000 to win 10 ,,from ubunut 8 to ubuntu 19

so,,,,please replace php with golang ,,,,,!!!

[lachmanfrantisek]

Switching language can also reduce dramatically the community able / that want to maintain Nextcloud because, all of the sudden, everything is different.

Additionally the challenge consists of PHP-developers changing to golang, that fact alone lowers productivity for month as Tobias mentioned, not even considering the lower availability of golang-developers.

Switching language will in the end make the project to loose developers. Everyone in the project now knows php. And for most us cases as said before php is all that is needed.
Rewriting everything is a HUGE and complex project that will go on with less developerst than before ..

I agree with the difficulty of the change and effect on the community, but I would like to point out that there are also people from the opposite side (at least me..;-) that don't (want to) know PHP and will contribute only if it will be written in something different/better/modern.

Personally, I don't want to spend time learning php, but some languages are beneficial to learn.

---

Rewritting of the core part(s) would be nice:

• It will not take unrealistic amount of the time.
• We will not break the compatibility with all the apps that were created.
• The php developers will not go away and we can even rise the number of develoeprs. ("Divide et impera."..;)

[kiwy42]

I'm not part of the dev base, just a regular user.
However it seems that the move to GOLang is also make to offer a full CS3 Api compliance.
It is a project to standardize storage sharing and application connection between cloud app.
The goal of Owncloud is to be compliant with this API and they decided to do so by using modern coding language.
If there's an important move here, I think it is the use and implementation of this (I hope) future standard cloud API.

[Quix0r]

I won't be able to run NextCloud then on my server as I don't have Golang there and to much is already running on it. And please also remember that NextCloud is not only about sharing files, there are tons of (even official) addons, that are not into file sharing only. And also consider that most shared hosters (read: virtual server) provide PHP, not Golang.

[Quix0r]

@small1 exactly also my view! Next day, people want Python or Ruby (On Rails) or Java or or or ...

[Quix0r]

@aignerat they seem totally have forgotten the 3rd-party developers. You are going to loose a BIIIG peni.... eh, user base. ;-) Holy shit, what did my girlfriend write me?! ;-)

[helmut72]

I won't be able to run NextCloud then on my server as I don't have Golang there
(...)
And also consider that most shared hosters (read: virtual server) provide PHP, not Golang.

?? You don't need any compiler. Golang projects provides binaries for nearly every OS/CPU, because cross compiling is very easy. Installing is also easy: copy one file, starting it, done.

[kesselb]

OK. If @Quix0r cannot use Nextcloud anymore we should close this issue right now! We need more of these constructive posts :rofl:

[kiwy42]

Owncloud is influenced by its business base, they at the moment need a strong standard API to access file between application in the cloud.
The consortium behind CS3 API is powerful and will do anything possible to have that API everywhere. it happen that they decided to use a modern efficient robust architecture to offer this API and PHP with all the services it offers me in the past is a langage that should never had such success and is not a langage of the future.
The question for the core devs of Nextcloud is IMHO do they want or not to be part of the CS3 API project and if yes, is it possible or reasonnable to continue with PHP (IMO PHP is not the best to serve modern Rest API) .

[aignerat]

@Quix0r well it won't happen soon anyway so don't mind, atm only owncloud tries to go golang

[Quix0r]

OK. If @Quix0r cannot use Nextcloud anymore we should close this issue right now! We need more of these constructive posts rofl

I'm just saying it. Many users won't speak up because they don't know this ticket.

@Quix0r well it won't happen soon anyway so don't mind, atm only owncloud tries to go golang

Yes, such "changes" (it is not a simple change) must be discussed with the whole community, not just a few.

And besides: PHP is not that bad after all and it is common with many (to my knowledge) hosting companies.

[aignerat]

I had a talk with a developer of NC at the last conference, he mentioned that at least atm PHP is the way to go, and there are no plans to switch to golang in the near future.

[aignerat]

The question for the core devs of Nextcloud is IMHO do they want or not to be part of the CS3 API project and if yes, is it possible or reasonnable to continue with PHP (IMO PHP is not the best to serve modern Rest API) .

Exactly what I thought, I guess rewriting some crucial core-parts would make sense, on the other side PHP is fast enough to handle most scenarios.

[Quix0r]

@aignerat and it is improving each version.

[triangletodd]

For people saying that you wouldn't be able to run Nextcloud if it were written in Golang: Golang applications compile to a self contained binary and do not require Golang to be installed on the machine you plan to run them on.

[stonyz]

I'm a java developer, also use php for some web project. I think we should switch to golang. PHP may not be the best choice for backend service, even though it's good enough on web.

[Quix0r]

Have it been asked how many PHP/Golang developers you will loose/gain? Just asking, because I don't know Golang but Java and PHP.

[Quix0r]

So Golang runs no where but just the plain machine? I guess you have to run some container or at least a web server like Apache/Lighttpd is.

[helmut72]

I guess you have to run some container or at least a web server like Apache/Lighttpd is.

Like PHP. You can run any programs written in Golang. Console, Server Daemons and with or without Container, Webserver...

Have it been asked how many PHP/Golang developers you will loose/gain?

Have it been asked how many users move away if core services are slow?

[wucke13]

Have it been asked how many users move away if core services are slow?

I have not been asked, but I moved away exactly because of that.

So Golang runs no where but just the plain machine? I guess you have to run some container or at least a web server like Apache/Lighttpd is.

You guess wrong. You can compile a self contained binary from golang code and that binary can serve as it's own webserver. Obviously for production you probably would want a reverse proxy like nginx in between, to mix in simple SSL cert management and other web services on the same ip, but that's about it.

Have it been asked how many PHP/Golang developers you will loose/gain? Just asking, because I don't know Golang but Java and PHP.

This question is not super relevant in the context of knowing Go. I would advice anyone who fears the unknown to educate himself for some ten minutes. Golang is so primitive as a language that almost everybody who has used __any__ procedural language before will be able to use it instantly. Golang in a way was designed from Google for Google so that coding rookies can get started fast, without any further idea of higher level programming concepts. The only context in which that question makes kind of sense IMHO is: How many people know Go, and don't want to write a program in it?

I would prefer a bunch of __other__ language because they offer __more advanced__ semantics, like true generics, macros and so on. But whoever argues "People will move away from XYZ, because they don't know Go" did not investigate any time in understanding what Go is, and who it's made for.

Edit:

I don't want to blame or insult anyone. I simply cannot imagine that anyone who actually knows a bit of Go would claim that a lack of Go experience is a good reason not to start using it. Whoever has used some exceptions or OOP modelling is far beyond the possibilities of Go.

[aignerat]

Hey Wucke,

sure GO is no complex language like c++ that is rocket-science compared to PHP. It has some true advantages like beeing able to write good code with ~30% less LOCs

It's good to discuss PRO/CONS here, still I can just repeat myself that atm there is no plan to switch to golang in the near future. As far as I understood, security has more priority than plain performance. It would be helpful if anyone could bring up a POC to show the advantages and a simple integration of a core service. Generally rewriting and implementing crucial performance-functions in GO will make the structure of the system more complex, therefore more room for errors.

[monotok]

Looks like Pydio rewrote their software from PHP to Golang, looks like they were in a similar position. https://medium.com/@charles_93287/why-we-rewrote-pydio-in-golang-723d6071d30c

Seems like they were glad of the switch and had to overcome issues like training php developer in Golang etc.

Anything is possible :)

[wucke13]

It has some true advantages like beeing able to write good code with ~30% less LOCs

Just wondering, what makes you think that? I would have guessed the opposite, since go is very verbose and offers none of the classical shortcuts for writing less code (dynamic types, generics, macros or any other form of code generation or syntactic sugar like uniform function call syntax or mixins) + very explicit and verbose error handling. Of course it may be that the very complete (std-) library accommodates for that overhead, but I haven't heard of any other claim about that?

I can just repeat myself that atm there is no plan to switch to golang in the near future.

Is this a nextcloud official statement?
If yes, this should be outlined more visible IMO, as it has an impact on the scope of this discussion.

[aignerat]

It's no official nextcloud statement, but a developer at a conference told me that at the NC-Conference in september 2019, because I was interested in the change too.

I don't use go, I mainly code in PHP, that's what my colleges said, who code a lot in GO. And it's the goal of google to have less code with full functionality and stability while delivering maintanable code, could be 20% or 40%, of course it's an estimate.

[go2sh]

One thing never mentioned in this thread is how the nextcloud project actually works. Though nextcloud is open source, most of its development is done by developers employed by the nextcloud GmbH. The company has a business model, customers, contracts etc. Switching the language and start all over again is by no means just a technical decession.
There are countless reasons to stay with php from a business perspective. You have a well tested proven code base, throwing that away will hurt. Current setups with maybe some custom integrations have to be migrated, it would take a lot of time and money to recreate them. You have to invest a lot of money in the first place to create a new product and thus reducing the cash flow, if you don't have an investor, it will become difficult. ( The investor and venture capital thingy is one of the reasons why nextcloud was founded )
From a technical point of view there is only one thing with the language, that php currently cannot solve: the C10k problem. But you can solve it with hardware. I would guess in most commercial projects with nextcloud, hardware cost will be a very small portion of the overall budget. ;-)

[aignerat]

From a technical point of view there is only one thing with the language, that php currently cannot solve: the C10k problem. But you can solve it with hardware. I would guess in most commercial projects with nextcloud, hardware cost will be a very small portion of the overall budget. ;-)

And don't forget that it's not about 10k users, but concurrent users, allthough if you use mobile + webdav + web you need 3 connections, it's still safe to say, that 20k Users on 3 channels will never cause the c10k problem as PHP will run and close the connection when everything is loaded. Ajax calls are really quick and only load what's needed, most processes are speed up by redis and php-fpm. Another thing that is a larger problem is the database. The standard of pgsql is 200 concurrent connections, it's safe to config 500 cc, the documentation says, you shouldn't use over 1000 cc. I allready ran into that problem with 200 cc.

[wucke13]

From a technical point of view there is only one thing with the language, that php currently cannot solve: the C10k problem.

Yes and no. From a business way kind of yes, throwing more money is easy. But if some project offers the same features as another at 1/20 of the resource impact, that is a potential problem.

Edit:
I do not mean to say Go is exactly or even roughly 20 times faster than PHP. Clearly that is subject to the given setup. With the primary caching technologies of PHP (APCu, OPcache, ...) one can speed up the execution of an existing app quite a lot (without modifying it's code in most cases). Though configuring a PHP application for high performance __is__ hard. To get the best bang for the buck (read: the given hardware), one has to investigate quite a lot. In the meantime someone gets maybe 4 times the performance without doing more than something like ./myapp -p 8000. Just a wild guess: go being 20 times faster may be quite realistic in comparision to the most basic Nextcloud installation that takes a similar amount of time (just installing Apache with mod-php) and unzipping Nextcloud to /var/www. Of course PHP apps can do a lot better than that most basic setup, but that _a lot better_ comes at a higher cost (of time and knowledge) than the out of the box performance that a go app achieves (if it is not horrible messed up, finding extreme negative examples is possible in mostly any argument). And this higher cost is something that both individuals and businesses eventually consider when making there choices. TL;DR The maximum performance one can achieve with money is not the only thing to consider - performance per investment (time, knowledge, hardware, ... you name it) is important at some point too.

[go2sh]

I doubt that you can reach 1/20, what I have found online: You can do 1/2 to 1/4 better with golang in real world scenarios. Also there is some much more to nextcloud, e.g. fast db, fast fs, fast cache, etc.

I just want to make a point, that golang doesn't magically solve every problem. You still need a lot more.

Also as I said, I think, the guys at nextcloud gbmh have a good look at their competitor and they probably got feedback from their customers.

Also take a look at the progress of both owncloud core and there new stuff, its not that fast and they are far far away from an enterprise ready product.

[Quix0r]

The question that bothers me is, how much PHP developers/contributors/plugin writers will you use compared gaining in Golang/Java/$Language$.

[aignerat]

The question that bothers me is, how much PHP developers/contributors/plugin writers will you use compared gaining in Golang/Java/$Language$.

What do you mean? Use as many as possible. Need hopefully the same amount or slightly less.. Get depends on the community.

[ben423423n32j14e]

Personally I'd pick PHP over Golang every single time because of the number of developers who are already familiar with PHP allowing a greater number of people to contribute. It's not all just about performance and to be honest PHP has been improving greatly, I don't think there is that much difference, definitely not enough for me to even have the slightest interest in the project moving to Golang, I don't think thats ever going to happen.

[monotok]

PHP is fine for the frontend, after moving my nextcloud instance to LXD I then enabled Redis cache etc. This greatly improved the webdav access time and web front end.

However I do think the core syncing code needs to be greatly improved and I don't see the issue with this moving to another language if it greatly improves the performance.

I recently installed seafile to test the difference; 4000 small files in a sync folder.
Nextcloud took 7 mins in total.
Seafile took 12 seconds.
Seafile is not golang but C++.

I still use nextcloud for mobile sync, small amounts of files etc as it's sync isn't that fast and generates conflicts with lots of small files. I know nextcloud is more than file sync but its core usage is file sync and probably needs rewriting in something else.

[aignerat]

thx for the test monotok, but you can't say that seafile is 50 times faster than nextcloud. You need to take in consideration if seafile has the same quality of logs, if it is as secure as NC and if you can access the data from the same channels. If you want a plain test of the core, then disable all apps, it should speed up the process.

Please line out your test a bit further to have a better comparison and maybe find out some issues that could lead to performanceupgrades. Especially PHP 7.2->7.4 should speed processes up a lot.

[monotok]

It was only meant to highlight the difference in sync speed compared to other similar solutions. Nextcloud is far more feature rich.
The web interface hasn't got any noticeable difference in speed, one is php and the other Django/python.
For a bit more context:
Nextcloud PHP version: 7.2.24
Both running in an LXD container on a VM with Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (4 cores) and 8 GB RAM.
Both have their data stored on a ISCSI disk using multipath active/active (2x1GB links).

I can try the same test with php 7.4 at some point.

[aignerat]

sure, I just wanted to point out it's similar to using a standard database and compare it to a tuned database. Could be comparing apples and oranges in that case. You can speed up a lot with the config.php (especially filelocking, files_check_changes)

[wucke13]

thx for the test monotok, but you can't say that seafile is 50 times faster than nextcloud.

He __isn't__. He's saying that there is a significant difference in a particular test case on a particular machine.

quality of logs

Very high quality logging isn't necessary but if one has a lot of issues? No matter what, disable the logs, make a benchmark and show how that makes it better. If your application suffers from writing to much logs, your application is probably poorly optimized though (or uses a language which is inefficient for I/O).

if it is as secure as NC

Both PHP and C are great for introducing security issues. More code is harder to audit. NC has a lot more code. Just compare seafile CVE's with NC CVE's. Sure, NC is used more.
Edit: As there is a widely used sync service, written in go: syncthing CVE

If you want a plain test of the core, then disable all apps, it should speed up the process.

Isn't the argument mainly about "We have so much stuff written in PHP ..."? If you ditch all that, the argument is defeated. Like: why would one choose NC if it is stripped down to the core functionality that other services deliver too, but faster?

I just wanted to point out it's similar to using a standard database and compare it to a tuned database.

Which is not the point. No matter how much you optimize, seafile will outperform nextcloud by a significant amount (at least ATM). Speeding up "a lot" is not sufficient. Not to say that you can tweak seafile too. Just a base install of both speaks a clear word (performance wise clearly in seafiles wins, featurewise clearly NC wins). If you tweak _both_, this will hold true.

[aignerat]

Not explicit, but still it implifies seafile is quicker than nextcloud by a huge margin under some circumstances, that's obviously the case so it would be good to know why and if there is an easy way to catch up without losing much functionality.

You can set the loglevel to 4, so only fatals are tracked, logging on huge instances of at least warning-level is very good to my mind to see if there are minor problems.

I tested syncthing too, it's really very fast, still the syncing follows simple rules and could be extended for collaboration.

When it comes down to pure performancetesting of the core, it's the only way to disable all the apps not needed to have a somewhat compareable outcome, many hooks of the apps produce database-entries, surely it's slower when you have 5 entries in your database per file compared to probably 1 or none. The golang discussion was mostly about how much gain is possible when rewriting some core-modules and the next step would be to point out the effort to integrate that rewritten pieces of code to the rest of the php-backend.

Could be true, I didn't test seafile so far and I don't know what exactly is done in the database. You're absolutely right, that most installations are about optimizing I/O as this is most probably the bottleneck.

[wucke13]

I tested syncthing too, it's really very fast, still the syncing follows simple rules and could be extended for collaboration.

Absolutely. I would really anticipate a syncthing with finer grained permission setup and sharing options (e.g. have a share read write for authenticated users and read for everyone else).

logging [...] at least warning-leve[...] to see if there are minor problems.

That is one of the issues I suspect resolve itself. PHP doesn't really encourage writing correct source + setup is so complicated that __a lot__ can and will go wrong. PHP scripts simply are not a very robust way of writing complex applications.

it's the only way to disable all the apps not needed to have a somewhat compareable outcome

Why is that? My expectation would be that nothing (blockingly) is in the way of syncing files. If I use the NC client to sync a bunch of files, what a difference do any plugins make?

database-entries

database queries? Or is it really that a hook of an app likely is to insert new data in a table? Anyway, most of these hooks probably can be done in an async fashion. Go could help here, as it natively provides the means to do things async with channels, which is both convenient and most likely faster than blocking IO.

I didn't test seafile so far and I don't know what exactly is done in the database

Nothing fancy, really. An interesting side note: seafile doesn't have one database, it has three. One for ccnet, the user service for seafile, one for seafile itself and a third one for seahub, the seafile web interface. They take the decomposition game serious! _So, one could adopt NC to replace seahub to get a sweet interface with many diligent PHP developers on a fast core_. /s

[5dao]

I won't be able to run NextCloud then on my server as I don't have Golang there and to much is already running on it. And please also remember that NextCloud is not only about sharing files, there are tons of (even official) addons, that are not into file sharing only. And also consider that most shared hosters (read: virtual server) provide PHP, not Golang.

nextclouds is a NAS not web public services,so shared hosters is not better choice

golang can run anywhere ,php not

[aignerat]

Absolutely. I would really anticipate a syncthing with finer grained permission setup and sharing options (e.g. have a share read write for authenticated users and read for everyone else).

That's the challenge you got when you build a software for experts and normal users that expect everything to function automatically. NC has some good stuff like the owner and the sharee on a federated share (2 different instances) edit a textdocument at the same time, the sharee gets a message that it couldn't be saved and the owner's edits are permitted. I haven't looked into the collaboration texteditor of nc18 yet, that should even bring up more possiblities.

That is one of the issues I suspect resolve itself. PHP doesn't really encourage writing correct source + setup is so complicated that a lot can and will go wrong. PHP scripts simply are not a very robust way of writing complex applications.

it's not about correct source, there are a lot of 3rd-party-extensions for embedding external storages, handling encodings, vcards, signature etc. if something isn't working I'd like to know why, most log entries at least give you an idea of the problem, it isn't possible to test everything in advance, as you can't be sure of what kinds of data the user has to work with.

Why is that? My expectation would be that nothing (blockingly) is in the way of syncing files. If I use the NC client to sync a bunch of files, what a difference do any plugins make?

A plain speedtest should be measuring under the same circumstances, do I get the right idea that you are just talking about the client? I thought you are talking about the servers (as it is nextcloud/server). The client of NC could really need some more love, but the I/O happens mostly on the server-side, the sync-client is surely not written in PHP too. I guess we could talk for hours about this issue and wouldn't really get to the point, so I would assume it's better to accept it as it is, standard seafile is faster at syncing than standard NC. Still standard mysql is faster than standard pgsql too, still I wouldn't have a good feeling about 1 TB-databases in mysql, especially with a lot of joins needed to get some reports.

database queries? Or is it really that a hook of an app likely is to insert new data in a table? Anyway, most of these hooks probably can be done in an async fashion. Go could help here, as it natively provides the means to do things async with channels, which is both convenient and most likely faster than blocking IO.

PHP has a real disadvantage, there is no real threading, every request needs a new database-connection. That could be a point where GO could improve the solution by far, still this is only assumption and it could be a gain of only 10%, maybe the "time-loss" is on another part of the sync-process. (That's why I'd like to see plain NC vs plain seafile and see what's the difference in functionality), it's hard to get real numbers for performance, my experience is, that a solo-server can easy handle 200-500 Users based on the sizing, cpu is nearly irrelevant, as long as it's not a singlecore, 8 GB RAM for Redis should be fine. It really depends on your usecase. NC is mostly used for documents, it wouldn't even matter, when the syncing needs 10 minutes, because if you need realtime-syncing you probably want another ESB-based solution anyway, if you fancy workflow and nice functionality NC is probably the way to go.

Nothing fancy, really. An interesting side note: seafile doesn't have one database, it has three. One for ccnet, the user service for seafile, one for seafile itself and a third one for seahub, the seafile web interface. They take the decomposition game serious! _So, one could adopt NC to replace seahub to get a sweet interface with many diligent PHP developers on a fast core_. /s

K, from a developer-side I got to say, that NC has some legacy-code that's rather hard to adopt, but the rest gets better ever release. There is no penetration-test that finds out real hard issues, only minor stuff.

[Scrumplex]

I think this effort could start with something like a daemon, that handles heavy tasks. A daemon that does the stuff cron.php is running currently, or things like thumbnail generation.

[helmut72]

As we have learned from here, PHP is fast enough. Even Nextclouds High Performance Backend for Talk use PHP. Oh, wait.
https://github.com/strukturag/nextcloud-spreed-signaling

[small1]

And that backend was developed by a company called struktur from scratch. And not something that has been written for over over 10 years ....

[fbettag]

For anyone commenting „i cannot run Golang on my server“, you guys are so badly misinformed it’s beyond me.

Golang compiles to binaries, thus you can just build binaries for every OS and ship those. This is a far better solution than getting all required stuff for PHP setup and updated from a deployment perspective.

[go2sh]

Most go binaries are static linked. This implies huge responsibilities on both the user and the developer to use and provide secure software. Using a package manager and dynamic linked binaries reduces this risk.

And in general: Be polite. There are many more possibilities than just go as there are many more languages and libraries that can do the same.

[fbettag]

Most go binaries are static linked. This implies huge responsibilities on both the user and the developer to use and provide secure software. Using a package manager and dynamic linked binaries reduces this risk.

And in general: Be polite. There are many more possibilities than just go as there are many more languages and libraries that can do the same.

You sir, are also badly misinformed. Golang does not use shared libraries (except libc). All „the libraries“ you talk about are Golang code getting shipped with the binary.

It would really help if y‘all understood what you were talking about.

[go2sh]

I just did a quick google search dynamic library golang. The top 5 search results all show, how to use dynamic libraries in go. (as plugins, as 'real' shared library, as shared c library) Yes, the typical golang way is static linking. This is why I wrote 'most'.

And all that doesn't change my criticism: Static binaries put a burden on both the person building it and the person using it to keep its dependencies up-to-date and to fix possible security bugs. With a dynamic linking, you can just swap one library restart your service and you are good to do. Especially with packages supplied by a distribution, where many people work on keeping stuff up-to-date.

[helmut72]

@fbettag

Golang does not use shared libraries (except libc).

Be fair. This is the same what he wrote:

Most go binaries are static linked.

@go2sh

This implies huge responsibilities on both the user and the developer to use and provide secure software.

Not a security problem. Recompile or pull latest Binary. It's not that you need to get 1000s of files and being lost in space.

Package managers must be also used at all. How many old versions are running out there?

[fbettag]

I just did a quick google search dynamic library golang. The top 5 search results all show, how to use dynamic libraries in go. (as plugins, as 'real' shared library, as shared c library) Yes, the typical golang way is static linking. This is why I wrote 'most'.

And all that doesn't change my criticism: Static binaries put a burden on both the person building it and the person using it to keep its dependencies up-to-date and to fix possible security bugs. With a dynamic linking, you can just swap one library restart your service and you are good to do. Especially with packages supplied by a distribution, where many people work on keeping stuff up-to-date.

In my 7 years of deploying production golang code, i have never seen this being used. Sure you can do it, but putting something in this discussion that affects 0.01% of all golang code out there is not helping this either.

As for keeping shared libs up2date, this is just factually wrong. You will need to keep your glibc up2date no matter what, thats what your package manager usually does out of the box.

There is no additional security implication here from golang. It would only strengthen this project’s security as nobody would have to run, maintain or patch a buggy PHP interpreter, which has been the biggest turn-off for me so far.

So i would be ecstatic to see this moving forward. This also means you could deploy nextcloud really everywhere without having to care about how to install a runtime like PHP on every operating system.

[wucke13]

Just a small note: @go2sh frames it like it is the accepted truth that static linkage is implying a lot of security issues, and dynamic linkage solves this problem. My perception goes slightly towards _you can't say_. For one, there are common situations where dynamic linkage in fact does not do what @go2sh described, which is especially dangerous if users assume that just swapping the dynamic lib for a new one fixes a security problem. Also, one must ask how many (maybe security related) bugs might be prevented if the code running on the dev-machine is exactly the same as the code on the user machine, versus the wide variety of different combinations given the different sets of shared libs shipped by different distros. To my knowledge most modern languages aim towards static linkage (Rust, Go, Crystal _?_, ...) for a reason. Also interesting: the fully statically linked linux distro Alpine has by far fewer vulnerabilities as nextcloud, despite being used widely among all the container folks for example.

I'm not saying static linkage is more secure, but it certainly offers other nice perks with IMHO irrelevant disadvantages (come on, binary size doesn't matter, storage is cheap nowadays) and without being significantly less secure.

[go2sh]

Your comparison of nextcloud and alpine makes no sense. Alpine is a distribution with nearly no own software and alpine is not fully statically linked. Check the package archive. There are alot of dynamic libraries and packages depending on them.

I'am not against static linking, but you have to be very careful. I found a lot of github issue in go packages with update xy because its in our production code and we need to patch it by hand. Going the way of php or python by providing an interpreter with an well maintained set of features, which can be updated independently of your software makes it easier to update more quickly without waiting for all maintainers of your package dependencies.

@fbettag Please, did you read my criticism? You can argue against php. I'am fine with that, I can easily live without it. But every peace of software has its drawbacks and go is not the answer to all questions. go and especially the libraries you statically link into your final binary (will/might) have security bugs in it. Every time someone somewhere in your dependencies tree fixes something, you need to recompile your binary, ship it and replace it. And the fix needs to propagate through your dependency tree as everybody needs to create some sort of commit/release, since each package pins a dedicated version of its dependencies. For php and its core libraries, you can update them independently of your application (mostly through very good distribution packages( and the app developer don't need to worry about that.

[fbettag]

Your comparison of nextcloud and alpine makes no sense. Alpine is a distribution with nearly no own software and alpine is not fully statically linked. Check the package archive. There are alot of dynamic libraries and packages depending on them.

I'am not against static linking, but you have to be very careful. I found a lot of github issue in go packages with update xy because its in our production code and we need to patch it by hand. Going the way of php or python by providing an interpreter with an well maintained set of features, which can be updated independently of your software makes it easier to update more quickly without waiting for all maintainers of your package dependencies.

@fbettag Please, did you read my criticism? You can argue against php. I'am fine with that, I can easily live without it. But every peace of software has its drawbacks and go is not the answer to all questions. go and especially the libraries you statically link into your final binary (will/might) have security bugs in it. Every time someone somewhere in your dependencies tree fixes something, you need to recompile your binary, ship it and replace it. And the fix needs to propagate through your dependency tree as everybody needs to create some sort of commit/release, since each package pins a dedicated version of its dependencies. For php and its core libraries, you can update them independently of your application (mostly through very good distribution packages( and the app developer don't need to worry about that.

Ok, so for the record. your usually golang binary (at least on OpenBSD) links with:

        Start            End              Type  Open Ref GrpRef Name
        0000000000200000 0000000000db3000 exe   2    0   0      bin/aerc
        00000002025ba000 00000002025c7000 rlib  0    1   0      /usr/lib/libpthread.so.26.1
        0000000275bba000 0000000275cae000 rlib  0    1   0      /usr/lib/libc.so.95.1
        000000029908d000 000000029908d000 ld.so 0    1   0      /usr/libexec/ld.so

So if libc or libpthread have a severe security vulnerability, then the nextcloud golang code would be the least of your problems, because then at least 50% of all your userland are somewhat a problem all of a sudden.

[fbettag]

Your comparison of nextcloud and alpine makes no sense. Alpine is a distribution with nearly no own software and alpine is not fully statically linked. Check the package archive. There are alot of dynamic libraries and packages depending on them.

I'am not against static linking, but you have to be very careful. I found a lot of github issue in go packages with update xy because its in our production code and we need to patch it by hand. Going the way of php or python by providing an interpreter with an well maintained set of features, which can be updated independently of your software makes it easier to update more quickly without waiting for all maintainers of your package dependencies.

@fbettag Please, did you read my criticism? You can argue against php. I'am fine with that, I can easily live without it. But every peace of software has its drawbacks and go is not the answer to all questions. go and especially the libraries you statically link into your final binary (will/might) have security bugs in it. Every time someone somewhere in your dependencies tree fixes something, you need to recompile your binary, ship it and replace it. And the fix needs to propagate through your dependency tree as everybody needs to create some sort of commit/release, since each package pins a dedicated version of its dependencies. For php and its core libraries, you can update them independently of your application (mostly through very good distribution packages( and the app developer don't need to worry about that.

as for the comment about the dependency tree, it seems you have never used vendored golang packages. All your dependencies are inside your code repo, there is no global plugin you need to somehow magically make available like geoip2 when it's not available in your package system. All these issues that make for more issues like "it does not run on Ubuntu 16.04 because they have an old php-xyz module that doesn't support X.", are suddenly gone. BOTH in deployment AND in development.

[wucke13]

Plus probably there are tools for updating (transitive) dependencies automatically. At least that is for example the default behavior for Rust. A quick glances at time between successive releases of Nextcloud tells me that frequently updating wouldn't be a huge difference anyways. But theses points are closely related to how and how much you use 3rd party dependencies. Go already has a rich std lib and despite an ORM there is few things that are necessary to provide a neat webapp. So yeah, if you do it in a bad way, it can be a real big issue. But if you manage it carefully (which is necessary for PHP as well, btw.) these issues are no blockers.

[PopeRigby]

Would Rust be another possible option?

Jun 12, 2020, 2:47 AM by [email protected]:

>
>
>
>

Plus probably there are tools for updating (transitive) dependencies automatically. At least that is for example the default behavior for Rust. A quick glances at time between successive releases of Nextcloud tells me that frequently updating wouldn't be a huge difference anyways. But theses points are closely related to how and how much you use 3rd party dependencies. Go already has a rich std lib and despite an ORM there is few things that are necessary to provide a neat webapp. So yeah, if you do it in a bad way, it can be a real big issue. But if you manage it carefully (which is necessary for PHP as well, btw.) these issues are no blockers.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, > view it on GitHub https://github.com/nextcloud/server/issues/16726#issuecomment-643182481> , or > unsubscribe https://github.com/notifications/unsubscribe-auth/AE7GLJC7NCSEXCAJFA7K5ODRWH2T3ANCNFSM4ILEOQSA> .

[wucke13]

I would love to see it with Rust, and rust would be fit for nowadays. However, given the overall skepticism towards switching languages at all, that will very likely have to happen in a fresh, unrelated project.

[kescherCode]

To me, replacing PHP with a compiled language would be a welcome change.
File handling is somewhat sluggish with PHP, and this is an area where Nextcloud should shine, as it's centered around files.

I wonder if Nextcloud could just be slowly transitioned to another language, using both PHP and ?
Also, I wonder how plugins would be made with such a change.

[JonathanTreffler]

We are already in a migration phase in the frontend, where more and more is getting migrated to Vue.js.
I don't think this project can handle vanilla js -> vue.js and php -> golang at the same time.

I personally think that at some point php will just be too old to find good developers, because nobody cares about php anymore. But that is going to take a long while.

In a optimal world it would be possible to support both PHP Apps and Golang Apps at the same time, but that would be very hard (if not impossible).
It would be awesome to have a golang framework to interact with the Nextcloud Server Backend written in PHP, but that would cost a lot of development time.
I don't think it is a good idea to rewrite the backend (at least not in the next 4 or so years), but being able to write Apps in Golang would be awesome and at least a tiny bit more realistic than a complete rewrite.

[michaelstingl]

PoC how PHP backend and Golang apps/extensions can co-exist:
https://owncloud.github.io/ocis/bridge/

[jospoortvliet]

In short, there is nothing a Go rewrite would give Nextcloud users that we can't already have with some pragmatic improvements. We already use Go in a few places, as mentioned, and the performance issues that exist (some corner cases) just haven't been bad enough to motivate anyone to fix them - no need to rewrite all of Nextcloud for that, in any case.

With regards to surviving next to an Go-based ownCloud - I doubt it'll change anything.