Language: Allow access/visibility to private members of libraries

[icatalud]

Currently there is no way to access private members of external libraries, the classes and class instances coming from them.

If I'm wrong, you may stop reading, but I couldn't find a way.

Private members should by default somehow be accessible unless explicitly specified by the owner of the library in the pubspec file or something like that and it should be explicit in LICENSE file of the library that it is not allowed to use the code in a different way than what it was designed for originally.

By default, I believe over 90% (have to be proven) of the library owners are ok with people freely adapting their open source libraries. Private members are most of the time created in order to avoid confusion, by helping and directing the users of the library to the members that are designed to be utilized. That doesn't mean there aren't other clever or creative way to use them, which would require modifying behavior through private members. In fact there is often a thin line on whether choosing to make a member public or private. Many public members where not thought to be externally utilized when a library is created, but still are left public "just in case". The punishment for not making the right choice in terms of members privacy is too hard, there is no fall back mechanism for the users. Constraining the freedom of maneuvering through object/libraries by not giving any possibility to access private members, goes in detriment of the expansion of the language in general, since it hinders possibilities of usage and extension creation of libraries.

The protection of "bad usage" should be given by compiler warnings that can be skipped through annotations, like @accessPrivate. IDEs can easily hide private members, by not showing access to those members unless the user writes an underscore _.

This would produce conflict in the language since currently you may create private members with the same name as the one in the external library without it being overridden. Personally I find that a bad confusing pattern and would choose to eliminate it. But if it sticks, it can still be worked around by writing for example "private object._fieldName" or a special annotation which would imply access to the private/invisible member _fieldName instead of the "visible" one. It should be possible to use this keyword/annotation multiple times in case of inheritance through nested libraries.

I believe modern languages should go more in the lines of freedom, thinking more positively of users as responsible and intelligent beings.

[leafpetersen]

I don't think there's any chance we're going to move in this direction. Library private names are currently essentially the only mechanism that library developers have for preventing implementation details from leaking into the public API. Dart already has the problem that almost any change to a public API is a breaking change. Making private names accessible would double down on this by making changes to private implementation details also a breaking change.

[icatalud]

I don't think there's any chance we're going to move in this direction. Library private names are currently essentially the _only_ mechanism that library developers have for preventing implementation details from leaking into the public API. Dart already has the problem that almost any change to a public API is a breaking change. Making private names accessible would double down on this by making changes to private implementation details also a breaking change.

Saying "double down" the problem is exaggerated, since in practice most people will you use the public interface. That's the reason why accessing private members would require extra syntax, such that people is aware that they should avoid doing that unless strictly necessary. It's most likely that most libraries will keep being used the same way, but it will add an extra amount of usages, that will be mostly people that takes a deeper look and have good ideas on how to use those members. People that uses this, should be aware that they are making a choice that doubles down the risks of problems in future library updates.

It should be well noted in the documentation that you should avoid using private members, unless strictly necessary.

[rrousselGit]

So, what that means is that private wouldn't be private anymore?

If you want public unrecommended variables, maybe you want @protected instead, or prefixing the name by $.

[icatalud]

Exactly, there shouldn't be a private without fall back. Why would you want that? You are not responsible on how people use your library, you just give your official "how to" and the rest is up to the user. It should be users decision what is the best use they can make of a library.
Imagine you find a library that suits perfect your needs, except that you need to access a private member. The work around is having to redefine functionality on your side for hundreds of lines, mainly copied from the library or simply modifying the library yourself and having it on your project instead of importing it from the official source.
The creator of the library doesn't mind that you access a private variable. If that works for you, that is your issue, not theirs. Unless they explicitly write in their project that by no means they want people touching their library.

Most open source library owners are happy to see people use their library in whichever way they want. The whole point of open source is free to use code, so why forcibly put constraints to limit the usage? The constraints should exist to direct and help the user to use the library, but not to forbid using code. That is silly. You can make it as complicated as you want, in order to discourage people from using private members, but there should be a way around it.

[icatalud]

I guess the question is the following. If we are trying to create a more transparent open world, why are we providing tools to darken libraries? What do you gain from it? If something breaks because people use it, they know they are responsible, they were warned in many ways that what they are attempting to do is undesired.

Also, like I said before, I think most people use private members as a directive, to help the IDE point you out the members you should access, but very very few people actually want to darken the code by all means (if there are any).

[icatalud]

@rrousselGit Giving it a second thought, I guess you response about the @protected annotation goes to the cheese. It's far too easy to create private invisible members. I think most people is using private members in their libraries for readability of exposure of the library functionality. At least that's why I was using them, it makes clear to me which variable I should be accessing from the outside, but by no means I want to completely hide the variables to users. I didn't thought about @protected, but even then I would create private variables with underscore anyways because I like the notation, it's easier and cleaner to me.

Perhaps it is a good thing to expose a way of letting people create private invisible members, but it should be hard, like adding two or three extra annotations, so it makes it very clear that the author really has the intention of not letting people touch their library (it's unlikely that is a common requirement). I do not think that is the intention of most private members created so far.

By the way I started learning dart not long ago, because of flutter. I was trying to adapt the flutter library to have a behavior closer to my expectations and I found myself unable to do want I wanted because I couldn't access private members and because I couldn't mutate objects created by the flutter library. I cannot hide that I felt frustrated when I found this constraints. Since I saw the dart language is on development I came to comment this. I like the language syntax and would love to see it thrive, however I really believe it's a mistake to constraint the language so much, it feels too much like Java. In my opinion, flexibility attracts people (makes it more suitable to different tastes) while forced rigidness discourages them.

[munificent]

In my opinion, flexibility attracts people (makes it more suitable to different tastes) while forced rigidness discourages them.

You are thinking of flexibility only from the perspective of a library consumer, not a library maintainer.

If I maintain a library and any private method can be called, then I have lost the ability to change or remove that method. It is effectively part of the API and it is a breaking change to remove it. This makes my life as library maintainer harder. Because of that, libraries evolve more slowly, which in turn also harms library consumers. Users want thriving up to date libraries, and encapsulation is the primary tool we have to give library maintainers the freedom to grow their libraries quickly and safely.

If something breaks because people use it, they know they are responsible

This is not true once you consider the real world where people have large, complex dependency graphs. Let's say I have myapp. It uses some package foo, which in turn uses another package bar. I don't know anything about bar. It's an implementation detail of foo.

Unbeknownst to me, foo calls some private method in bar. A new version of bar comes out that removes that method. This is just a minor version release since — as far as the maintainer of bar knows — there are no breaking changes. The next time I run pub upgrade, I get that version of bar. Now my application is broken because foo is trying to call a method that no longer exists. I don't maintain foo, don't know about bar, and didn't do anything wrong. My application is broken, and I don't have any easy way to fix it. I certainly can't fix foo, because that's not my code.

When you multiply this scenario by the hundreds of packages a typical application uses, I hope it's clearer that encapsulation is very important to ensure code you don't know about doesn't spontaneously break your app.

[icatalud]

Thank you for your thoroughly explanation @munificent , it makes very clear the reasons of the choices you have made for the language and the direction where you are pointing it (you want it to be very reliable and consistent). I was not aware of this huge dependency graph problem, probably because I have never maintained a big library. Almost always I have worked more superficially in short term projects. It's interesting, because we are on completely opposites sides of development chain. How do they deal with dependencies problem in python or javascript (or it actually has been a permanent huge annoyance) I'm curious ?

It makes sense the intention you have to provide reliability in the library dependencies. Essentially you believe that if you give any chance to access those private members, no matter how much extra verbosity, warnings, etc, developers had to deal with to access them, people will use them anyways and some popular libraries will use them too (without users knowing). I can believe that is true (although it shouldn't be common if enough barriers are put), I do not know what the programming languages history have to say about this. But let me include the other side of the coin, which is developers that like to try new stuff by quickly prototyping. You miss a huge exploration space for the libraries by not giving any possibility of flexibility on aspects like access to private members (there is also object mutation and classes wrapping). Sometimes you start using something new, you play around with it a little bit and soon there is something that you would like to be different, better according to your limited perspective. Now if you are doing something small, you would like to achieve this by doing little effort. But if doing this small adaptation requires that you have to write more lines than the actual project you are working on (for example because you cannot access a private member), many times you will be feel discouraged to attempt this modification. Writing 10 times more lines not only makes it harder because you have to understand deeper things, but it also makes the modification more prone to errors, making it actually more than 10 times harder to achieve the intended result. If you knew that what you are attempting to do would be used a lot, perhaps you would make the effort, but you cannot know that beforehand. Only a minor percentage of small extensions or modifications of bigger libraries end up popularizing. They popularize because they represented the users preferences. Think of this exploration space like a calibration of a library, it actually helps the libraries improve more than causing problems. There is a trade off, but there must be a better middle ground were you give free maneuvering under certain constraints without jeopardizing reliability in the dependency graph.

A library is like any other tool, the owners create it thinking on one usage, but one cannot think on everything. Many tools have been customized, and from these customization new products arise or improvements are made afterwards to the original product. Which one is the best tool is determined by the users, supply and demand, we should always take that into account and not cover our eyes from it. Freedom is the best way to explore (as long as we are able to provide it).

Unbeknownst to me, foo calls some private method in bar. A new version of bar comes out that removes that method. This is just a minor version release since — as far as the maintainer of bar knows — there are no breaking changes. The next time I run pub upgrade, I get that version of bar. Now my application is broken because foo is trying to call a method that no longer exists. I don't maintain foo, don't know about bar, and didn't do anything wrong. My application is broken, and I don't have any easy way to fix it. I certainly can't fix foo, because that's not my code.

In the dependency graph wouldn't you always get the maximum possible version of the libraries? So shouldn't foo be pointing out to a version of bar that works for it? But I guess it's too easy to write ^ notation referring to the current version of the library. Since private members are not reliable, perhaps there should be a possibility of creating "prototype libraries" that are able to use private members of another library conditioned to a fixed version of that library (cannot use ranges). It could even be hard coded in the import, like "import 'package:foo/foo.dart v1.2.1'" and because of it you gain access to all members, not just the visible ones. That feels similar to copy pasting the library though, but that feels wrong and clumsy.

[pulyaevskiy]

You miss a huge exploration space for the libraries by not giving any possibility of flexibility on aspects like access to private members

There is a nice way to achieve easy exploration:

1. Fork the library
2. Modify it to your liking
3. Add dependency_override to your pubspec.yaml which points to your fork.
4. Experiment as much as needed.
5. (Optional but recommended) submit a pull request to the original library repo with changes from your fork.

As a package maintainer myself I really appreciate this flow and step 5 in particular.

[icatalud]

That is actually an alternative. Sorry for not taking a deeper look. It is not as easy as one would like it, having an overhead of extra steps, but perhaps is for the good, since it encourages you to update the library itself.
Thanks,

[icatalud]

@pulyaevskiy I had second thoughts about your comment after I went to try your steps, however the options dependency_overrides and dev_dependencies give a hint of what would be great to do. Although the option to do want you want exists with anything, the idea is that the effort to get there is small when there is a desire to achieve something small. In the measure of reasonable a language should attempt to provide assistance in common user requirements. In the case of what I was trying to achieve, it was just accessing the private _debugLifecycleState in flutter elements to know when they are in defunc state. I was just trying something out, I find it hard and unlikely to believe it will be possible or even a good idea to push a PR to flutter to make that variable somehow visible (it would probably require a huge debate and I would have to inform myself very well about a lot of details before I can attempt that). Yes there are steps available to do it, like the ones you described, but what is the overhead of a very simple initial request, which in terms of code is immediately at reach, but you cannot reach it unless you do a lot of mechanical work which also requires understanding more things if you have never done something similar before. It greatly distracts you from your creation process (your fun).

Everyone has different styles of development, in my case I like dividing the process in two phases: the creative one where you play around quickly testing stuff, and the formal one, where you take decisions based on the trials and start structuring and formalizing what you did. I like static typing because I feel it makes me more productive, however I miss from dart some sort of "dev mode" where you can escape a little bit from the chains of production reliability. Once you finish your idea, you devote some time to properly finish your work by fixing this unsafe "dev mode" paths you took. You also have motivation to correctly structure you work once you are happy and proud of your product.

In what respects privacy of members, I do not know how common or if it would be worth opening a dev feature to give access to private members. But I guess the only way to know is to hear the users opinion. In my case I didn't develop much before I felt I needed this, however I might have gotten unlucky. I do believe in what I said earlier though, which is that most private members are left private because the owners didn't want to expose the users to more members that are not intended to be used by the designed API, but that does not take into account people that is taking deeper looks and have ideas that escape the expectations.

The introduction of the dependency_override is probably the way to go, but perhaps allow a little bit more flexibility. Say you have a dependency_override with a fixed version of a library, then maybe you could add the dev keyword, like "import dev 'package:foo/foo.dart'" which allows you to access to whole library as if it was in your code. Since you have an override and an import dev, your project is obviously flagged as dev.

[munificent]

The path @pulyaevskiy suggests is what I like too.

You are definitely right that when prototyping or hacking something together, sometimes you just want to be able to muck with the code and ignore rules like privacy. In a language like C++ or C# where you may not have access to some third party package's source code, only the compiled binary, this can be a real problem.

But Dart packages are distributed in source form. This means that the language can be strict about privacy. If you want to make something public, the code's right there. You can always edit the text files.

[icatalud]

I know the code is there, it's just a matter of accessing fast and easily to those members when needed. How simple you should make it, is in the end a matter of personal preference. I stated my reasoning, others might agree. The annoying thing about downloading the package is that it takes you out of dart code (you have to go to git, do some stuff and modify library, just to access one member).

In summary, my preferences are the following:

• There should be a way to access private members (explicit and verbose). Many alternatives were presented to allow this, some more restrictive than others, but libraries that use private members should definitely be flagged as dev libraries.
• It should not be allowed to reuse private members names when extending classes (because you can access them).