Crystal: [RFC] Providing extension methods in crystal

[Timbus]

Hello all

I come with a request to support "extension methods" in Crystal, as well as a proposal on how they could be implemented.

I define "extension methods" as methods that don't (typically) modify the state of the object they are attached to, nor really make use of OO. Some good examples of existing stdlib methods that should
be extension methods include Int.times, the '.to_{x}' methods, Enumerable.map and .each..
Basically, anything that would work as a function in a different language.

My reason for desiring a way to safely provide these methods (ie: no monkey patching), is so that
libraries can provide (and use) richer, cleaner APIs without fear of stomping on other libs
and breaking end-user's code. Examples of some possible extensions are things like list operations
(zipwith, cross), type conversion helpers (.to(AnotherType)), and quality-of-life methods
(transform_keys, select_random, rspec's old, better looking should method)

I honestly want Crystal to flourish, and for this I believe third party libraries need to be able to offer
the same rich features and functionality that the standard library provides.

This is my giant wordy proposal on a way to achieve this with a combination of lexical-scoped methods and UFCS (including the above rationale, you can skip it): https://gist.github.com/Timbus/1f278441e7ecffbb67b664a49e18adcb

EDIT: Updated proposal following my own and ysbaddaden's comments: https://gist.github.com/Timbus/51d54cc7a66e69f36f8263a5d01032cf
I have separated out the nonessential extra features, but left them as features that can be added in the future if needed.

I'd like to point out that this proposal is only a suggestion. Many languages facilitate extension
methods using alternative mechanisms to UFCS, and if any of them would make for a cleaner, more
"crystally" solution, then I'd be very than happy to see it added instead.

Thanks for reading. You may now start yelling at me in the comments below.

[Timbus]

After sleeping on it, the only reason my current proposal has import-ish behaviour is to provide a saner alternative for including into Program, as shown at the bottom of the module docs: https://crystal-lang.org/docs/syntax_and_semantics/modules.html
Additionally, the fact UFCS only works on imported methods seems a little bit underwhelming (I limited it to imports for sanity/safety reasons)

I could also propose a tweaked version that only works as extension methods, which better suits Crystal anyway and might seem less complicated:

using Base64
"a string".encode64 # Still looks good to me.

It's basically the same thing, except it doesn't mimic any form of 'include' behaviour. I believe would this make it almost identical to C# extension methods, with a tiny bit of extra sugar.

Just getting my own conversation started here.

[RX14]

I think we shouldn't add any major new features to Crystal before 1.0. We should focus on getting what we have now robust enough to be a good foundation to build on later. And even after 1.0 I think I would definitely favour more simple - if more limited - proposals.

[Timbus]

This is entirely backwards compatible, and only serves to enhance the way Crystal already works, ie: Method-heavy syntax.
I can't see how this would need to wait for a 1.0 release?

[RX14]

@Timbus if it's backwards compatible it can wait for a 1.0 release. And if it can then it should, since time and resources are limited.

[ysbaddaden]

Sorry, too complex. This is over engineering. It even proposes to create aliases when crystal has a strong no aliases stance... please, no.

Maybe using a module to mixin its public methods (and ivars) without including submodules is a good idea. For example the following ends up with a Foo::Serialization type that can't be trusted (the includes may be inversed):

class Foo
  include JSON::Serialization
  include YAML::Serialization
end

:heavy_check_mark: having a using Module at the module-level would avoid this (yay).
:white_check_mark: maybe this could evolve to allow module-level monkey patching (ala Ruby refinements).
:x: scoped imports of specific methods that can be aliased then ignored? nope.

[Timbus]

If a large application is using several libraries that try to implement the same extension methods, renaming/ignoring allows that to be easily resolved.
But instead of arguing for why I suggested it, I guess it is better to ask you this: Why is it a good thing to have imports that cannot be renamed, or removed from the scope?

I don't think this is over engineered, I tried to document and cater for problems that this feature might introduce. If I didn't you would have called it broken/problematic.

[bcardiff]

I understand there are two concerns here.

1. Workaround potential collisions of method names
2. Ensure that some features are added without breaking encapsulation

The first issue don't play well with the global namespace nature of modules and classes IMO. It is a fact that collision may happen. Shards should prefix their code, user might need to use :: to disambiguate, etc.

I'm still in favor of the global namespace, without explicit import of the modules in every file. It is less pure than modules in Haskell/Clojure/ES6 for sure. But I don't think this has been a real issue in RoR ecosystem. Sometimes it's harder to know what code is been included, but with the right tools is easier to navigate imported definitions.

For the second issue, again, with right tools it can be checked that some modules do not access directly to ivars. Eventually this could be hooked somehow in the compiler itself maybe. I would find that a bit more crystal-way IMO, annotating a module with a @[Pure] or something like that and analyze during or after compilation if the criteria is met.

Maybe an outcome of this is a guideline of how to code shards splitting core functionality and extensibility to aim for code easier to maintain in some aspects, but having in mind that global namespace.

---

It was great to see the proposal detailed in such way!

[Timbus]

The first issue don't play well with the global namespace nature of modules and classes IMO. It is a fact that collision may happen. Shards should prefix their code, user might need to use :: to disambiguate, etc.

Module methods, and therefore imported methods, are still global with this proposal. I don't know what you mean by "play well", though.
using just lets everyone pick and choose what they want from the global namespace, preventing the majority of method collisions.
For the remaining few cases, I proposed a renaming/ignoring mechanism, which would have been a godsend if ruby had it when activesupport decided to break my code just the other day. Again.

Further, a person making shards cannot be held to the impossible task of 'never making the same extension method as someone else'. How can you ever know what namespaces and method combinations are currently in-use, within an individual user's library collection?
Only the end-user can be the tie-breaker, yet Crystal currently gives them no way to do it.

I'm still in favor of the global namespace, without explicit import of the modules in every file.

Agreed, and I believe it is the main reason ruby refinements are so unpopular. That is why I proposed a hook to add default imports when you 'require' a file.
Such a mechanism could be used for other interesting things too, but I leave that to the imaginative reader.

---

For the second issue.

This proposal is somewhat about separating encapsulation, but only for practical reasons. I actually don't really care if helper methods access @ivars, that's not really a concern that has bitten me in the real world. The first issue has bitten me.

Maybe an outcome of this is a guideline of how to code shards splitting core functionality and extensibility to aim for code easier to maintain in some aspects, but having in mind that global namespace.

This is good and I support it, but I don't think it's enough, and I'd rather the language had the tools to better facilitate it =/

[rdp]

I always wish I could add to "core libraries" without having my monkey patch methods leak to the global namespace, somehow...whether this or any other proposal. :)

[chocolateboy]

Previously:

• Encapsulated Require System
• [RFC] On matters pertaining to reopening and redefining
• Scoped imports

And a related proposal for Ruby:

• Improve namespace system in ruby to avoiding top-level names chaos