Orm: Some questions about the future of the Doctrine Project

[javiereguiluz]

Hi Doctrine Project managers (@Ocramius, @beberlei and @guilhermeblanco),

This morning, in the Symfony Slack chat, I posted the following message:

I'd like to ask your opinion about the health of the Doctrine Project. It's a critical piece for Symfony and I'm concerned about it: the number of contributors looks alarmingly low, the development of new features is not very impressive lately and it accumulates more than 1,000 open issues. Thanks!

I won't copy the entire discussion because there were a lot of messages, but at the end someone suggested to ask you directly about the future of the Doctrine Project. So here are some questions:

1) Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?
2) Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.
3) Is there an estimated date to publish the 3.0 version?
4) Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?
5) Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away? How could we help you with that?

Thanks for your time!

[Ocramius]

Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?

We're planning to release 2.6 and call it the last 2.x version, freezing any request for improvements.
ORM 3.x is being developed by @guilhermeblanco when he has free time.

Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.

No roadmap: the roadmap is "we do what we can, when we can", sorry.
We're mostly thinking of:

• dropping features
• performance optimizations (especially considering PHP's new engine)
• internal cleanups to reduce complexity

Most public API will stay as-is until we got to a decent cleanup state and can take decisions about the future of the interfaces.

Is there an estimated date to publish 3.0 version?

No - at this rate it will be likely no earlier than 2018, unless we just do a release due to a forced BC break/security issue leading to BC break.
We cannot commit to any schedule, since we're not a company and we do not profit from the tool.

Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?

I think the amount of prototyping work done by @guilhermeblanco doesn't play well with coordination work. Yes, he is going "cowboy-coder" here, but I really do trust him on most design decisions behind the scenes, as he's been very meticulous and informed with his work.

We also don't have the time to coordinate work, and the domain knowledge behind the ORM internals is huge and sadly only written down in the Java world.
Assuming we had the willpower for it (not the case at the end of a work day), nobody in the team would really be able to follow somebody's work closely, as the amount of issues being spawned already completely clutters our ability to work on new stuff in our free time.

Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away?

We had more than 1000 open issues for ages - the fact that we moved them to Github a bit over a year ago simply exposes the number to the "general public". There are no critical issues or known vulnerabilities at the moment, so there may be a big purge after the API changes (3.x).

How could we help you with that?

We could need somebody to go into open PRs and find good reasoning for rejecting them. Yes: I do intend reject there, as most issues on the ORM starting from version 2.4.0 are edge-case scenarios, mapping weirdness, exotic relational databases and structures.

The ORM is an extremely stable tool, so I wouldn't worry too much about the amount of issues, rather than the quality/relevance. Also, we are extremely alert on security issues, which we also didn't have in a while.

If somebody wants issue rights and go in and do a close-spree (with careful explanations and cross-referencing), then I'd gladly grant those.

Other than that, I can suggest sponsoring (actual 💰) @guilhermeblanco (if he's interested - this is my idea, not his) for working full-time on the ORM, and that's a quite substantial investment.

[javiereguiluz]

Thanks for your detailed answer! The status of the Doctrine Project is more clear now 👍

[lcobucci]

@Ocramius do you think we could/should make this status a bit more visible?

[Ocramius]

@lcobucci just do it if you feel like it 👍

[lcobucci]

@Ocramius ok, I will think on a nice way and propose it 😉

[guilhermeblanco]

Hi @javiereguiluz!

I'll answer the questions from my own POV. I think it's ok if we answer individually on our own perspective and then take this discussion offline and come up with a project response.

1. Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?

We have enough fixes/features on master branch that we could release 2.6 today. AFAIR, there was one outstanding fix needed for SLC to be reviewed and merged (which by now it already is), so there're no blockers to release 2.6.

I'm focused on working purely on 3.0. However, I couldn't touch it since October due to a variety of personal reasons (I switched jobs, onboarded a huge project on initial phase which required some extra dedication and now took a month of well deserved vacation) and will resume my development once I get back home.

1. Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.

We share a Google Doc describing our wish list and that is what we're doing on ORM.NEXT Project. Not everything is listed there, since as we evolve, we might change plans. There're thousands of lines of code to be written and thousands to be dropped.
I started a new branch called develop. You should be able to use it today (with some BC breaks already highlighted) if you want. I've lost count of how many unknown bugs (and edge cases) I've fixed so far, but you can count 100+.

As it currently stands, develop has embeddable support removed (actually commented out) since most of it was done in a dirty hooked way. Properly supporting it (and addressing most issues), would require a full re-development of the feature, so I removed its support for now.

I have yet to review 3.0 issues and close/answer them properly and annotate with similar elements on ORM.NEXT project. Should do that any day tbh.

1. Is there an estimated date to publish the 3.0 version?

No, but I was committed to dedicate 2-3h daily on this project to work on 3.0.
Keep in mind Doctrine 2 took an year of planning and 2 years of development. We don't need to redo all the work, but I expect it to take at least full 2017 of development before it hits early releases of 3.0.

1. Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?

There're a couple blocking changes required before we unleash the ability to split work with others.
Right now I'm in the middle of a massive change (transforming ClassMetadata mapping from associative arrays to OO). I've done most of the work already, but any other person that decides to jump in now would have to rewrite a lot of code until I finish this epic.

The change to ClassMetadata will unleash changes to be made everywhere. One good example is the ability to compile metadata PHP classes that can take advantage of opcache and remove the need for metadata cache forever. As I work on OOing Metadata, I'm also ensuring that Export to PHP is adequate, so consuming it later would be a joy.

1. Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away? How could we help you with that?

Most of the issues are actually user issues, not bugs on ORM. As we evolve, we add new ways of implementing similar features, which lead people to use them in funky ways.
Our main wish is to reduce the amount of weirdness people can do while using Doctrine, which should save us substantial time and effort (well, and save us from some grey hair).

[Ma27]

as I already mentioned in the Slack discussion, I try to comment on some issues in order to help, but my time is quite limited right ATM.
I hope that I'll have a little bit more time this year, so I'll try to spend 1-2h in a week by looking through the issue tracker and checking if I can do anything.

[Jean85]

I will do the same! Happy to help such a great project.

[Ocramius]

Closing here - answers were given, I suppose :-)

[guilhermeblanco]

Looks like it's well deserved some updates around this.

I've been dedicating 15-20h/weekly on develop branch since February. Right now almost the entirety of ClassMetadata has being moved to object orientation. Our performance tests highlighted an improvement of close to 10% and memory optimization close to 20%.

Right now I'm in the process of re-building the ClassMetadataFactory and also the way we load, cache and build ClassMetadata instances. It's expected that we have a significant performance improvement over this, since we'll no longer have to validate and complete arrays. It'll also be opcached.

As soon as this work is finalized, we'll be able to unlock parallel efforts of optimizations and improvements. One of the big challenges I have today is to break down Embeddables and Inheritance across the board. I have a detailed plan on how to build them, but the amount of interdependencies are preventing me to start this work. Working on ClassMetadataFactory will help reducing most of them.

I've been discussing extensively of having improved Metadata implementors as a way to strategically optimize code that is duplicated across the board on UnitOfWork, Persisters and SqlWalker. This should have invisible changes to end-user, but will unlock the possibility of specifying how the eager fetching could happen (either through join or through follow-up select).

Right now UnitOfWork is also doing a lot when comparing changesets. We've discussed internally how we could make it OO, and also delegate the change detection to Metadata implementors. This is also a couple months ahead of the game.

We made a decision of trying to ditch Query\Parser and leveraging Hoa\Compiler. Anyone should be able to work on this today, but it's non-trivial and required deep understanding of how we built our current parser and also some SQL standards.

I'll attempt to keep this issue with some background information, but ORM.NEXT project is the way to get a more accurate status of the project.

[mnapoli]

@guilhermeblanco would crowdfunding or sponsoring help in any way, or are you committed to some kind of full-time job (i.e. can money help free some time for you)? Given how widely used Doctrine is, I think a few companies and individual could help financially.

[Ocramius]

@mnapoli we discussed crowdfunding a few times, but never got to actual concrete details.

For now, I personally applied to http://prototypefund.de/ to develop #5550 in the second half of this year, but that's just a personal bet.

If that fails, I'll probably catch up with the rest of @doctrine/doctrinecore and set up a non-profit organisation for Doctrine, so that we can accept funding.

[pstephenson02]

I've heard good things about https://www.patreon.com/ as well to help with sustaining contributions. I believe Laravel uses this and I'm sure a number of other open source projects.

[Ocramius]

Yeah, but since Doctrine isn't a legal organisation yet, we'd first need to
go through that step.

On 12 Apr 2017 21:53, "Phil Stephenson" [e-mail] wrote:

I've heard good things about https://www.patreon.com/ as well to help
with sustainable contributions. I believe Laravel uses this and I'm sure a
number of other open source projects.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/doctrine/doctrine2/issues/6211#issuecomment-293688883,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJakPvwXGcg2VrwJNnGYSStErazF1Iqks5rvSu9gaJpZM4LZrXN
.

[psihius]

@Ocramius I'd suggest taking steps and setting up infrastructure for that. Doctrine is a huge and very popular project that is one of the pillars of the PHP world. If somethong happens to it, it's gonna be a mess, especially if it results in project splitting into forks and all that stuff. There also needs to be a knowledge base built up and docs really need a lot if work (they are vague in a lot of places and sometimes you can't even understad the params of some methods even looking into the code)

[Ocramius]

I'd suggest taking steps and setting up infrastructure for that.

Please scroll up by 2 comments: https://github.com/doctrine/doctrine2/issues/6211#issuecomment-293382782

[CarlosEduardo]

Great job guys! CC: @guilhermeblanco @Ocramius @lcobucci

[weaverryan]

I noticed there's no develop branch (anymore) - has the 3.0 branch moved somewhere else? Thanks :)

[Majkl578]

@weaverryan Develop has been merged into master which became 3.0-dev, see #6955. 2.x branches are in feature freeze now.

[weaverryan]

@Majkl578 - I finally found that - sorry for not noticing it :). Very exciting! Thanks!

[iluuu1994]

Has Patreon been reconsidered? I'd love to donate a couple bucks a month to bring Doctrine forward.

[addiks]

Has Patreon been reconsidered? I'd love to donate a couple bucks a month to bring Doctrine forward.

https://www.patreon.com/phpdoctrine/overview

[iluuu1994]

@addiks Awesome!

It's quite hidden. Consider linking it on the home page and in GitHub's README file?

[timonf]

Is this the official Patreon? Who made this?

[Ocramius]

Yes, it is the official patreon, created by @jwage