Meteor: Investigate swapping Underscore w/ Lo-Dash or upgrading Underscore

[jdalton]

_86 Upvotes_ It's been mentioned a few times, so I figured I'd track progress/questions with an issue.

_85 Upvotes_

[felixrabe]

Any updates on this?

[richieb]

Any update. I need some of LoDash's methods, including _.cloneDeep()

[glasser]

In Meteor, you probably want EJSON.clone.

Lodash continues to be something I/we are curious about but never seems to
rise to the priority level of, eg, making the Mongo connector scale better.
Maybe next bug week.

[richieb]

Thanks very much, David. I wasn't aware of EJSON.clone.

[jdalton]

Thanks for the interest in lodash. Let me know if you all revisit the issue in the future.

[richieb]

Hi, JDalton,

I am confident the Meteor Devs (also known as MDG, for Meteor Developer Group) will no doubt add LoDash to Meteor. But they started with Underscore as the safe bet at the time, I suppose. On top of that, I think they like to do their due diligence to make sure any package they add to the core Meteor platform is sufficiently tested and reliably integrated.

Therefore, I think they really just need a little push and the the time to include LoDash, but they wouldn't act unless there is enough interest. And based on the activity on this Github issue, there is not enough interest to rise to "sufficient."

LoDash is amazing and incredibly useful, btw. I have been using it with Backbone and Ember. And now that I am using Meteor, I expect to continue to use it. Thanks for your work on LoDash.

[glasser]

Lodash looks great. But just like every time we upgrade a Node minor
version, the day I swap in lodash will be a day where I do a full automated
and manual QA of Meteor. Which is something we're trying to make more and
more automated, but we're not there yet.

Minor underscore version bumps have caused regressions for Meteor. Swapping
out the entire implementation probably will too. (Usually it's a a matter
of a place where Meteor needs an improvement because it was using
Underscore wrong.)

[jdalton]

Minor underscore version bumps have caused regressions for Meteor.

Yap, that's because Underscore, and Backbone as well, don't follow semver.
Lodash follows it though, which is why we've pushed passed v2.0 ;)
We'll keep at it on our end. Ping me for questions/issues :email:

[dalgard]

+1

[reezer]

+1

Doing some benchmarks and profiling most time in our application is spent on underscore (and minimongo). So I am sure performance of many applications could be improved by switching. See the lodash website for in-depth benchmarks.

[sbking]

+1

[ghost]

:+1:

[yasinuslu]

+1

[dohomi]

+1

[mxab]

:+1:

[frozeman]

+1

[Lobosque]

+1

On Sun, Mar 16, 2014 at 5:15 PM, Fabian Vogelsteller <
[email protected]> wrote:

+1

Reply to this email directly or view it on GitHubhttps://github.com/meteor/meteor/issues/1009#issuecomment-37768810
.

[Lauricio]

+1

[ghost]

+1

[MauroJr]

+1

[aknuds1]

+1

I find lodash a great replacement for underscore, and use it whenever I can. I'm having problems fitting it into Meteor though, due to that integrating underscore already :(

[bestwestern]

+1

[ByScripts]

Voting for this too. Lodash is better and more reliable.

[darkadept]

I want Lodash too please!

[fungilation]

+1

[timbrandin]

+1

[ghost]

By the way. You can use Lodash instead of Underscore for your own code without problems. Just remove the dependency on standard-app-package and depend on the listed packages instead (select the release version you use). Make sure to remove underscore from your package.js. Then you can use the Lodash atmosphere package with mrt add lodash (or create your own).

I hope that helps some people, because I myself realised it quite late that there is no namespace conflict with _ because of the way that Meteor bundles the packages.

[pajooh]

+1

[RichardLitt]

+1

[jaulz]

+1

Replacing underscore with lodash can easily be achieved using the following approach:

1. Install lodash by mrt add lodash
2. Somewhere in your code simply use _ = lodash;

Though that also means that underscore is still loaded but later replaced by lodash.

[lifeinchords]

+1

[shaneGirish]

+1

[wizonesolutions]

+1

[rijk]

+1

[spencern]

+1

[subhog]

+4

[kyv]

+1

[evenfrost]

+1

[andresberrios]

+1

[janbaykara]

+1

[tkh44]

+1

[JamesMGreene]

:+1: for the :heart: of :godmode:

[bobiblazeski]

+1

[igregson]

+1

[avital]

Lots of people have been +1'ing this ticket, looks like people really want this. As Glasser describes, this isn't a trivial update. To help us prioritize, it would help to understand what particular problems you are having that switching to Lo-Dash helps with. Can some of the +1'ers elaborate?

[raix]

@avital The problem I had with underscore was https://github.com/meteor/meteor/issues/594 - Using _.each for an universal iterator had its issues - it ended when @glasser wrote a modified version of each to solve this. Imho it would have made more sense to use the lodash iterators instead. (more details in #594) - I do fully understand the implications switching regarding QA etc. a scary job.

I've tried to highlight the pro/cons/status in https://github.com/raix/Meteor-community-discussions/issues/8 and keep a track of things.

[evenfrost]

@avital As for me, the biggest advantage is speed. I believe this is mostly because Lo-Dash uses fast primitive loops and conditions instead of falling back to relatively slow native JS implementations (forEach-map-reduce etc.) as Underscore does. And this gap increases for web apps being developed with aim to provide really fast and pure experience on any platform — not talking about PhoneGap-like webviewers, but about REAL seamless (homescreen) web apps. And from this point, strict Meteor dependency on Underscore has always been a fly in the ointment (another one is jQuery, though this is off-topic for sure).

[bobiblazeski]

@avital I switched to lodash due to lack of cloneDeep and so far haven't looked back.Everytime I need some new feature I just look at documentation and 99% find it there. Lodash is very fast, feature rich and very high quality code. Due to my lisp heritage I code in heavily functional style and use lodash extensively where other people would use for loops (Steve Apter http://nsl.com/ style) and so far lodash always delivered both stylistically and performance wise. Here's how my code usually looks like

function raze(arr, keys) {
    return _.reduce(keys, function (res, key) {
        return res.concat(_.map(arr, function (d) {
            return {
                Year: d.Year,
                Name: key,
                Value: d[key]
            };
        }));
    }, []);
}

function  distribute2(people, amount){
        return _.reduce(_.sortBy(people,'need'),function(memo,person){      
           var ration = Math.min(person.demand,memo.amount);
          memo.amount -= ration;
          memo.people.push(_.assign(person,{ ration: ration}));
          return memo;
        },{ amount: amount, people: [] }).people;
      }

Just give lodash a shot and I'm sure @jdalton will help you in any roadblock you hit on. Whatever bug you fixed on underscore there's a big chance its already fixed on lodash.

[timbrandin]

For me, I haven't used it as I like to stick to Meteor defaults, but I like
that its feature rich, well documented and has deep cloning. :)

Cheers,
// Tim

2014-11-24 21:35 GMT+01:00 slobodanblazeski [email protected]:

@avital https://github.com/avital I switched to lodash due to lack of
cloneDeep and so far haven't looked back.Everytime I need some new feature
I just look at documentation and 99% find it there. Lodash is very fast,
feature rich and very high quality code. Due to my lisp heritage I code in
heavily functional style and use lodash extensively where other people
would use for loops (Steve Apter http://nsl.com/ style) and so far lodash
always delivered both stylistically and performance wise. Here's how my
code usually looks like

function raze(arr, keys) {
return _.reduce(keys, function (res, key) {
return res.concat(_.map(arr, function (d) {
return {
Year: d.Year,
Name: key,
Value: d[key]
};
}));
}, []);
}
function distribute2(people, amount){
return _.reduce(_.sortBy(people,'need'),function(memo,person){
var ration = Math.min(person.demand,memo.amount);
memo.amount -= ration;
memo.people.push(_.assign(person,{ ration: ration}));
return memo;
},{ amount: amount, people: [] }).people;
}

Just give lodash a shot and I'm sure @jdalton https://github.com/jdalton
will help you in any roadblock you hit on. Whatever bug you fixed on
underscore there's a big chance its already fixed on lodash.

—
Reply to this email directly or view it on GitHub
https://github.com/meteor/meteor/issues/1009#issuecomment-64259518.

Cheers,
// Tim Brandin @timbrandin https://twitter.com/#!/timbrandin
http://timbrandin.com
http://studiointeract.com

[patmoore]

I would like to use lodash ( and will if I get time to test and make conversion ). I WAS worried about interfering with Meteor functionality but I see from earlier comments that lodash can be used without messing with Meteor.

For myself, I would be happy with an official way to use lodash in our code. I have no need to care about what Meteor uses internally - if I can independently use lodash.

[dwelle]

+1

[glasser]

At some point we should move off of Underscore 1.5.2, either to its latest version or to Lodash. Both of these changes will require thorough QA, of course.

[JamesMGreene]

At some point we should move off of Underscore 1.5.2, either to its latest version or to Lo[D]ash. Both of [T]hese changes will require thorough QA, of course.

:wink:

[rclai]

+100,000,000.

There, it's gonna happen. :P

[hanchang]

@glasser , I really hope the decision is to move to LoDash and off of Underscore!

For those of you who want to switch before it's officially supported, I just ran meteor add stevezhu:lodash in my Meteor project root, added _ = lodash as the first line to my lib/constants.js (or any other initializer) and all seems to be well.

[ai10]

+1
comments re lodash 3.0 announcement
https://news.ycombinator.com/item?id=8952291
third party opinions can provide perspective absent from autobiography.

[jdalton]

A better source for info may be the official lodash v3.0.0 release notes, though comments from non-JS users and anonymous trolls can provide some amusement.

[rgoomar]

The new changes for lo-dash 3.0 are awesome! And huge performance boosts.

[JamesMGreene]

So excited for lazy evaluation in Lo-Dash v3.0.0! Now if only the Meteor core team would get on board.... :wink:

[digioak-zz]

+1

[donflopez]

+1 too!

[sabind]

+1

[fermuch]

+1!

[matteodem]

+1

[robertlagrant]

+1

[AlexYursha]

+1

[comerc]

Wanted translators: https://github.com/comerc/lodash-ru

[orlaqp]

+1

Please add it and let me/us know if there is anything we can do to help with the transition. Main reasons for me:

• Speed
• API consistency
• cloneDeep :-)
• plus all the new goodies from version 3.0.0 (https://github.com/lodash/lodash/releases/tag/3.0.0)

[javierbyte]

+1

[Tarang]

Just one thing to be careful with lodash is its not 100% compatible with underscore. Lodash always produces what .chain() would produce if you use _(xx) so it may have some issues on existing projects.

i.e
_([1,2,3]).map(function(x){return x}) => [1,2,3] //Underscore
_([1,2,3]).map(function(x){return x}) => (Wrapper Object) //Lodash

[jdalton]

With v3 Lodash has discontinued its Underscore compat build in favor of devs using Lodash modules to supplement Underscore until the full transition to Lodash is doable. That said, if Meteor
required it, we'd work something out for it.

[JamesMGreene]

@jdalton: Is there some v3.x-based documentation that explains potential conflicts/issues if using LoDash as a drop-in Underscore replacement (without the compat build)? Their base APIs are obviously very similar one way or another but I'm curious about the actual gotchas when using the full build.

[awatson1978]

Lodash looks great. But just like every time we upgrade a Node minor
version, the day I swap in lodash will be a day where I do a full automated
and manual QA of Meteor. Which is something we're trying to make more and
more automated, but we're not there yet.

What exactly would a fully automated and manual QA of Meteor look like? There's obviously TinyTest on the package side. And on the end-to-end side, the Meteor Cookbook has some two dozen example apps that are running on Travis.

https://github.com/awatson1978/meteor-cookbook

I'm still in the work of writing tests for all of these example apps (and will be for the next few months). But I suspect something like these benchmarks are what @glasser and folks are looking for with regard to the QA process needed to do the LoDash refactor?

If people can start brain storming the edge-cases that we would need to test to migrate to LoDash, we could start writing tests for them. For instance, _.each() would need to iterate over an array. Clinical Workqueues has a tagging infrastructure that uses _.each(), as I recall, so we could use it to benchmark that functionality. And so, on for each of the Underscore functions.

Do we just go through the entire Underscore API docs and find an applet or demo that implements that API? Or should we create a kitchen-sink style app that uses every underscore call, and use that as a benchmark?

[jdalton]

@JamesMGreene

Is there some v3.x-based documentation that explains potential conflicts/issues if using LoDash as a drop-in Underscore replacement (without the compat build)? Their base APIs are obviously very similar one way or another but I'm curious about the actual gotchas when using the full build.

I've created a wiki page for migrating that lists known differences.

[JamesMGreene]

Thanks, @jdalton! Very useful and detailed information as per usual. Seems like pretty minor differences overall but a few little potential gotchas — and mostly just more LoDash benefits. :wink:

[matteosaporiti]

+1 for improved speed (third party benchmark where lodash is one of the fastest library while underscore is one of the slowest), better documentation and many useful additions (cloneDeep being one of them).

[schrislambert]

+1

[madvas]

+1

[benstr]

I felt left out... throwing my hat in. +1

[yoh]

+1

[smowden]

+1

[rclai]

Is there a way to swap underscore to lodash as a local hijacked underscore package in order to test? I tried to copy the files of the underscore package into a local package and simply replaced the underscore files/includes with lodash files and for some reason when starting Meteor it is complaining that there is a circular dependency issue with underscore and meteor (the package). I expected it to just work since local packages take precedence right? So my hijacked version of underscore should override the other one?

I'd really like this thread to be less +1s and more of figuring out what initial steps we can take to start testing this at least unofficially.

[jonagoldman]

+1

[tungv]

+1

[firdausramlan]

+1

[mitar]

Please at least upgrade the underscore to the latest version.

[tmikoss]

+1

[jonagoldman]

Guys at meteor should see this talk (and everyone really):
https://www.youtube.com/watch?v=cD9utLH3QOk (Lo-Dash and JavaScript Performance Optimizations)

[mazswojejzony]

+1

[voxlet]

+1

[bryankennedy]

It seems like one step forward would be getting this on the Meteor roadmap. I'm guessing that until we see it on there, it isn't a priority for the MDG.

[rclai]

Hey guys, so I've created a repository with a solution to actually replace underscore under the hood with lodash:
https://github.com/rclai/meteor-lodash-replace-underscore
Please feel free to contribute or let me know of any suggestions.

[stachrom]

+1

[Tarang]

Gosh I wish there weren't so many '+1's in here. It makes it really difficult to scroll through the actual issues that need to be addressed with the differences between the two.

I've been trying to narrow down some of the differences so a 'drop-in' replacement can be used such as the one by @rclai or any others on Atmosphere.

In addition to the chaining issue with the previous comment I made there is another difference:

Some methods on Lodash inconsistently exit the chaining method to give out values. Examples of this are .reduce. There are also a few more methods on Lodash so .sum also exits the chain:

Underscore (Chains)

_([1,2,3]).map(function(x){return x}) => [1,2,3]
_([1,2,3]).map(function(x){return x}).filter(....) => [1,2]  //Assuming 3 is filtered out
_([1,2,3]).map(function(x){return x}).min() => 1
_([1,2,3]).map(function(x){return x}).sum() => [Error] no method 'sum'

No method on sum- but that's not an issue if lodash were to be the replacement

Lodash (Chained already by using the _(xx) notation)

_([1,2,3]).map(function(x){return x}) => [Chain object]
_([1,2,3]).map(function(x){return x}).filter(....) => [Chain object]
_([1,2,3]).map(function(x){return x}).min() => 1 //Chain is exited - inconsistent
_([1,2,3]).map(function(x){return x}).sum() => 6 //Chain is exited - inconsistent

So Lodash does use chaining with the _(xxx) notation and returns chains but often can exit the chain on certain methods instead of keeping the chain.

[jdalton]

@Tarang

Your summary of Underscore chaining is incorrect. Without chain() Underscore won't make it passed the first method call. So your example of _([1,2,3]).map(function(x){return x}).min() is incorrect as map would return the unwrapped array which would lack a min method.

Also Lodash chaining behavior is more jQuery-like. Methods that operate on and return arrays, collections, and functions can be chained together. Methods that return a boolean or single value will automatically end the chain returning the unwrapped value. Explicit chaining may be enabled using _.chain.

[Tarang]

@jdalton Yes, you're correct I mistakenly copied out the wrong line. The inconsistency with Lodash's chaining issue still remains though which was what I was trying to demonstrate using the underscore example.

We need to find a way to allow Lodash to work as a drop-in replacement for Underscore that way MDG would perhaps look into the issue a bit deeper. It may be similar to JQuery and the design could be more consistent with other public libraries, but this argument isn't helping the case about existing projects using underscore at the moment as they would be rendered incompatible.

If .chain() forces explicit chaining the _(xx) chaining style is a bit confusing and inconsistent. I think one of the things that could help is this notation followed underscore's lead. At least this would be needed for it to be a drop-in today replacement over underscore a viable replacement in the future.

[jdalton]

The inconsistency with Lodash's chaining issue still remains though which was what I was trying to demonstrate using the underscore example.

Your wording makes it sound like Lodash has a problem with its chaining. It's not an inconsistency, as in some oversight or bug, it's a different behavior.

We need to find a way to allow Lodash to work as a drop-in replacement for Underscore that way MDG would perhaps look into the issue a bit deeper.

That's not a good route. Lodash 3 has features that would be nixed if it were a drop-in replacement.

If .chain() forces explicit chaining the _(xx) chaining style is a bit confusing and inconsistent.

I disagree. It gives devs an intuitive-feeling chain by default but allows them to explicitly force chaining if they want. It also allows writing code that works in both Underscore and Lodash.

I think one of the things that could help is this notation followed underscore's lead.

Lodash is leading now.

[Tarang]

Your wording makes it sound like Lodash
lodash is leading now.

Apologies for this. I'm looking at it from the point of view where I can use Lodash instead of Underscore in my apps. I prefer Lodash.

This issue isn't about Lodash vs Underscore which is alot of where you're basing your arguments on. I think we all know Lodash has plenty of advantages over Underscore.

The problem is over the past 2 years there are tonnes of Meteor apps people have made using Underscore and I'm quite sure the primary reason MDG can't immediately swap Underscore with Lodash is these apps would suddenly start crashing due to issues such as the chaining issue as the code would operate as unintended.

This issue will remain +1s unless we can find a meaningful way to transition these users to Lodash without breaking their projects. It's the performance that is the advantage, perhaps the syntax but a broken app is a regression when the argument for it is an all out transition to Lodash with its syntax as it stands now.

[jdalton]

but a broken app is a regression when the argument for it is an all out transition to lodash.

Save it for a major bump. Roadmap that bump.
Even jQuery 3.0 is introducing back-compat breaking changes ;)

[Tarang]

@jdalton Meteor is used by lots of people and version 1.0's motto was there would be no major breaking changes going forward. JQuery & Underscore are the 'core' 3rd party packages used. Realistically the JQuery issue is still to come up forth and the package included with Meteor will probably not be upgraded as soon because of this.

If you look at everything MDG has done so far with breaking changes they have mostly put out paths for backwards compatibility with the exception of Meteorite's upgrade where they & Percolate actively engaged package maintainers to update their code and even helped them in many cases. The other was the file scoped variables introduced where eventually they added the compatiblity folder to ensure no one was left behind.

You can't expect everyone to suddenly adapt to use Lodash in the way you expect for the sake of syntax. There are a good deal of Stackoverflow questions and blog posts that break down suddenly. Lets not forget Meteor is a learner's platform too.

Perhaps eventually users can use the Lodash syntax but there needs to be a realistic upgrade path. Where the user can remove a compat like package (like this one: https://github.com/meteor/meteor/blob/devel/packages/deps/package.js) to get it to behave naturally.

I think alot of us, you included, want to start using Lodash with Meteor today and not on Meteor 2.0.

[jdalton]

You can't expect everyone to suddenly adapt to use Lodash in the way you expect for the sake of syntax.

I don't expect anyone to suddenly adopt. This issue is more than two years old now. The version of Underscore Meteor uses is almost as old. There's nothing sudden about this process.

I think a lot of us, you included, want to start using Lodash with Meteor today and not on Meteor 2.0.

I have no preference. If compat is a concern put Lodash on the roadmap for 2.0.

For a start MDG could maintain an official Lodash package, as they do with Underscore, and move on from there. Since Lodash follows semver they can update it more often without risk of compat issues and then separately, over time, work it into a future release.

[Tarang]

@jdalton If this is your point of view why cannot Lodash live as an Atmosphere package? Why does it have to be an 'official' package released by MDG. There is nothing that could stop you releasing Lodash on atmosphere or using the current atmosphere packages for Lodash if someone wanted to use Lodash?

I'm very against the stance you're taking. You're promoting Lodash on the point of view of evangelising it. This is very unfair to the people who actually build apps with the tools you make who are out to build apps.

This issue is more than two years old now.

I think there's a reason for this. There is no upgrade path for existing users. An outright update in the way you want would need a lot of effort to have two separate forks of meteor using both Lodash or Underscore. Not to mention people's apps. Naturally the issue laying around here without any action is simply the path of least resistance.

If there was an upgrade path there would be not a single reason to stick to underscore. If we can find ways to replace the current underscore package & find out with a bit more certainty on what will break.

What I'm suggesting is two packages, a Lodash package and a compatibility package that together can be used to replace Underscore. With the removal of the compatibility package, Lodash would exist in its natural form

[jdalton]

If this is your point of view why cannot Lodash live as an Atmosphere package?

An officially supported MDG Atmosphere package would be great.

I'm very against the stance you're taking

I think my stance is pretty fair.

I think there's a reason for this.

I get how compatibility works. That said, two years is probably a bit too long.

What I'm suggesting is two packages, a Lodash package and a compatibility package that together can be used to replace Underscore. With the removal of the compatibility package, Lodash would exist in its natural form

Meteor should stick to official released versions and not try to manually augment the supported lib/framework.

[robertlagrant]

@Tarang What's the difference between:

1. As per your suggestion, using a compatibility package and then at the next major version of Meteor (or whenever you'd switch to full Lodash) devs changing their code
2. Carrying on using Underscore until the next major version of Meteor (or whenever you'd switch to Lodash) and devs changing their code then, which I believe @jdalton is suggesting

I don't understand the advantage of 1) - can you elaborate?

[Tarang]

@robertlagrant There would lots of performance enhancements. Underscore is heavily used in Meteor core. Minimongo (Meteor's client side implementation of mongo) would see a performance boost, probably in a very noticeable way. That's just minimongo. Blaze, DDP would be others.

In general the prime advantage is performance. I'm assuming we're after Lodash's performance more than its syntax.

The other thing is with 1) we can get this today & yet get to switch to its syntax at our own pace (as devs).

[ghost]

I think it's no problem today to use lodash for your own Meteor app code. You can just load lodash besides underscore and use it in your own code. Meteor Core still uses underscore in this case. I've done that in a project in the past (see my comment on the top: https://github.com/meteor/meteor/issues/1009#issuecomment-49179113).

So I think the open questions are:

• Will the MDG switch to lodash in Meteor Core for increased performance (like Tarang just said in https://github.com/Tarang)?
• Will the MDG promote lodash as default?

I think that apps should still be able to use the current underscore package for its own app code. Besides from about 60kB more to load on the client, I don't see any issues in loading lodash and underscore. Meteor already handles exposing dependencies to packages and the app without global namespace conflicts.

[mazswojejzony]

I am a newbie Meteor/JS developer but maybe the way to go would be to define underscore/lodash-like adapter (think: interface in Java) which could be implemented using any 3rd party lib like underscore, lodash or maybe even lazy.js?

I guess many methods on the adapter class could basically reference actual implementations on the target lib; proxy/adapting code would only be needed when there would be no 1-1 mapping between adapter class method and any of the 3rd party lib methods.

Assuming this is doable (and makes sense performance-wise) Meteor and packages code would only depend on the adapter class making it fairly easy to replace 3rd party libs under the hood.

[BenDiuguid]

+1

Would love to see the switch!

[syzer]

+1

[ccorcos]

+1

[bryankennedy]

There's some new discussion on merging lodash and underscore - https://github.com/jashkenas/underscore/issues/2182

Might be relevant to this discussion.

[rgoomar]

@bryankennedy I'm really hoping that they merge. There are benefits to both and not having to choose one over the other is nice.

[pthrasher]

@rgoomar I'm not quite sure what benefits underscore has over lodash. Also... Since lodash encompasses almost all of the behavior of underscore (and then some) for me there's no choice. Lodash wins in almost every category (speed, docs, modularity, test coverage, and more) and I use it every time instead of underscore. My hope is that they do not merge.

[jdalton]

@pthrasher

My hope is that they do not merge.

We've just opened talks. We share much of the same goals, much of the same API (~60%), and much of the same contributors. Merging makes sense as it would reduce a lot of duplicated effort on both sides and simplify choices & support for developers. It's still early but I can't imagine sacrificing speed, docs, modularity, test coverage, etc.

[pthrasher]

@jdalton As always you're on the ball with regard to github. ;-)

I don't mean to fracture the discussion by posting here as well. If those key items won't be sacrificed, then so be it. It took me a while to decide to switch from underscore to lodash. Lodash was the underdog for a while, and I certainly didn't jump on board as an early fanboy.

I'm always willing to be proven wrong, but it appears to me that since the APIs have so much in common, the reasons to like lodash are generally the reasons to dislike underscore. That mismatch makes me worry a bit about a possible merge.

No reply is necessary, but feel free to reply to me in jashkenas/underscore#2182 if you think this conversation shouldn't be in this thread.

[jdalton]

I don't mind a bit of duplicate posting :P

I'm always willing to be proven wrong, but it appears to me that since the APIs have so much in common, the reasons to like lodash are generally the reasons to dislike underscore. That mismatch makes me worry a bit about a possible merge.

Yap, we'll have to see how it shakes out. Because it's still early the details of merging haven't shaken out yet so it's not clear if Lodash would merge into Underscore, or if Underscore would be a build target of Lodash, or some other configuration. What is clear though is combining our efforts and reducing fragmentation is a win for developers (if we can do it without sacrificing quality).

[pthrasher]

What is clear though is combining our efforts and reducing fragmentation is a win for developers (if we can do it without sacrificing quality).

I definitely agree with that.

_edit:_ clarity

[ebradbury]

+1

[VincentDauliac]

+1 It would be great !!
I hope to see Lodash in meteor's roadmap soon.

[Siyfion]

At the very least, could we not have an "interim" release with the latest version of underscore, it's such a pain to not have some of the newer stuff.

[Alfwich]

+1

[jxm262]

+1

[uNmAnNeR]

+1 underscore!!!

[maccesch]

+1 lodash

[KonstantinKo]

+1 lodash

[eXon]

+1 lodash

[tomericco]

+1 for lodash!

[maartenbusstra]

+1 lodash

[churcho]

+1 lodash

[joaobarcia]

+1

[nhducit]

+1

[fabiospampinato]

+1

[nanamkim]

+1 lodash

[Kamshak]

+1 lodash

[hiphapis]

+1 lodash :+1:

[catgualter]

+1 lodash

[tagrudev]

+1 lodash

[ccorcos]

Ok. this ticket is just a tease at this point...

[MeKarina]

I don't think we should push to this extent. Maybe the reason MDG not to swap or upgrade because current underscore is sufficient enough, and there are more priority and unreplacable task such as project mongo

If you need more power (lodash), there are lodash community package. I use https://atmospherejs.com/stevezhu/lodash. As I write it use lodash latest version (3.10.1). You only need to install, put _ = lodash; in you lib file, do a migrate which quite fast and easy, and all working fine

I believe MDG know what to do now and what to do next. I believe MDB team,,,

[jdalton]

@MeKarina

If you need more power (lodash), there are lodash community package.

No offense to stevezhu or the five other folks who have created Lodash packages but a handful of packages maintained by non-Lodash and/or non-MDG folks is not the way to go.

I believe MDG know what to do now and what to do next.

IMO MDG kind of painted themselves into a corner by using a modified-Underscore and now no one is happy. The MDG Underscore package is using a version of Underscore that's more than 2 years old and they have yet to step up and maintain an official Lodash package despite Lodash now being the most depended on npm package :cry:

[jxm262]

@jdalton , @MeKarina This has been discussed quite a bit on the forums I believe.

Echoing @jdalton , using many of the existing lodash packages is _not_ the same as replacing the default used underscore. Without explaining it all in this issue, you can take a look at meteor-lodash-replace-underscore to see why. IMO it's important to actually move to lodash as the default, or at least make some sort of way to swap it out as the default. I'm not even interested in upgrading the underscore version at this point since I (and many I know) only want to use lodash.

[MeKarina]

Yes I agree the most ideal is to swap to lodash. Swapping to lodash will be a breaking change. If it happen is might be 1.3 (soonest). Before that, MDG should consider to coordinate with community because a lot of community package use underscore. And, in reality, this is a painful job, i know that because i did several community projects...

[jdalton]

To get things moving in the right direction it'd be great if MDG had an officially supported Lodash package.

[Pushplaybang]

+1 for an official lodash package. the performance difference looks insane.

[zasquez]

+1 lodash

[JamesLefrere]

+1 lodash

[veeramarni]

+1 loadash

[derwaldgeist]

+1

I just fell into the trap that _.pick() does not support a predicate function in Meteor's version. Please at least update it to a more recent one. Or even better, use lodash.

[dcsan]

meteor seems to be using a really old version of underScore,

_.VERSION
"1.5.2"

which is from 2013??!?
1.5.2 — September 7, 2013 — Diff — Docs

I really want the new _.mapObject in 1.8

1.8.0 — Feb. 19, 2015 — Diff — Docs
Added _.mapObject, which is similar to _.map, but just for the values in your object. (A real crowd pleaser.)

http://underscorejs.org/#changelog

is it possible to swap out / replace meteor's underscore without causing the above mentioned problems?
or maybe I can add lodash to my project and use that instead - without nameclashes?

[derwaldgeist]

which is from 2013??!?

Have you noticed the creation date of this issue? ;-)

[Pushplaybang]

@derwaldgeist - I'm surprised with clearly so much community interest and constant activity since then that this has still not got a milestone, associated ,or at least a "we're working on it" statement.

[derwaldgeist]

Maybe it's just because underscore is used that much in the core that they don't want to risk any breaking changes? Who knows.

[rozzzly]

I think 1.3 modules will make this more possible/likely.

here's how, and why:

Say you're writing fresh code, using es6 modules with import/export, etc and you have globals hidden, they must be explicitly imported. ie: if _ global is hidden, so it is undefined, throws error if you try to use _.reduce(...), etc. Now all you gotta do is:

import _ from 'lodash'

then start building with lodash. Seriously, ignore everything else. You don't need everything to change over at once; if one package, or part of core is using lodash, the "traditional" bits of code don't even notice and keep on using their global underscore reference.

Not foolproof by any means, but better than embarking upon refactoring some breaking changes across (nearly) the entirety of meteor core, as well as a huge majority of community packages.

---

Meteor's got 99 problems,

_that people actually want to invest time into_

but a _ ain't one.

_* drops mic *_

---

This isn't a fun change, it's a huge ugly one. You won't find anyone saying we need lodash over NPM modules, Blaze2, a new data layer/graphql/etc. It's always been on the back burner and always will. _Meteor has too much room to grow right now™_ Each of us is thinking:

It's not _broken_ so there's no way in hell I'm going to waste my valuable time being the only one to start to refactoring this entire thing.

This issue has been open for 3 years and there hasn't been _(AFAIK)_ any PRs, or people volunteering to _take-one-for-the-team_. As it stands, it's just not going to happen. But what if suddenly a few of flow-router, simple-schema, accounts-ui, etc started to use lodash? Then might you seriously consider making the switch, not needing to refactor your entire project all at once just to make it run?

I bet you would

lodash is awesome, the switch is simple (personal experience), why wouldn't you

When people actually adopt, then we can start contributing little bits of refactored things into core, public packages, new examples, etc, etc, etc... _and the snowball starts to roll down the mountain_

---

it is incumbent upon _us_, the community... upon _you_, to make the change.

We can't sit and wait for MDG, or some GoodGuyGreg to do it for us.

All the code I've been writing for the past several months has been pure lodash. Almost no complications, but lots of awesome. Let's do it together. I have bits and pieces of it posted publicly, but the (private) packages I've written will be OpenSauced in WithinTheNextMonth™ now that they actually work, lol. You don't have to be the first one to attempt it; jump on in, the water's nice.

thanks guys.

[cesarve77]

+1

[nchenbang]

I can confirm meteor 1.3 will solve this problem. All you need is npm install your latest version of underscore or lodash and import the function you need at the top of the file where you need to use it. Something like this:

import { map } from lodash

Note that currently Meteor underscore package will take precedence over local node_modules/underscore but it'll be fixed in the next release of 1.3 beta. The is the related issue in case someone is interested https://github.com/meteor/meteor/issues/5933

[derwaldgeist]

@nchenbang Thanks, this sounds promising!

[ghost]

I think MDG still need to extract their modifications to underscore, so they can finally update underscore to the latest version.

[wmertens]

How about using an AST transformer to convert MDG underscore calls to the equivalent lodash calls? It is only in a couple of cases that the API differs, right?

https://github.com/facebook/jscodeshift

Then you can change meteor-core, bump the major release version, and provide the script for people that need to perform the same changes on their own code.

[tcastelli]

+1 for lodash :)

[bySabi]

I create a PR proposal for split Meteor underscore on a internal underscore-base package and a wrapper.
With this we can have more room for experiment with modules, updates, ...

https://github.com/meteor/meteor/pull/6494

One step ahead on this subject.

[cyclops24]

+1 for lodash

[vlasky]

In a Node.js app, CPU time is the most valuable resource to be maximised. Lodash has higher performance than underscore, so it will help achieve that goal.

If the MDG don't want to permanently replace underscore in the core with lodash, then they should implement a mechanism to allow the user to make that choice for themselves.

[rdnorthup]

+1

[jaredtmartin]

+1
The outdated version of Underscore doesn't have _.mapObject :-(

[vlasky]

It is clear that people want underscore to be replaced with lodash - mainly for improved performance.

Therefore, it is a good candidate for the Meteor roadmap:

https://github.com/meteor/meteor/blob/devel/Roadmap.md

[nyxynyx]

+1
+256
+1048576

[glasser]

Despite the huge number of people saying vaguely that lodash is "faster" than Underscore over the years on this thread and pointing to various benchmarks, there haven't been too many concrete points about what that means.

Here's something I learned today on an old Meteor project that uses Underscore instead of Lodash: Lodash provides linear-time algorithms for large arrays passed to functions like _.difference, _.uniq, and _.union, where Underscore only provides quadratic algorithms for these functions. Using _.union in this particular project has apparently been leading directly to outages.

Had I understood that back when I was personally one of the active maintainers of the Meteor framework, I think that would have been much more motivating than the hundreds of low-information "+1", "it's better", and "it's faster" comments here. Unfortunately, I don't personally have the time to work on this change now, but I suppose add me to the chorus of upvotes at last :)

[mitar]

Maybe the easiest thing would be to swap underscore with lodash in one release candidate, allow developers to run their projects with it, and report if anything breaks.

[jdnichollsc]

My solution using lodash with Meteor ES2015:

meteor npm i lodash --save

and import lodash /imports/startup/client/index.js

import lodash from 'lodash';
window._ = lodash;

Regards, Nicholls

[vlasky]

@jdnichollsc we are talking about using lodash in the Meteor core packages including those on the server side.

[awatson1978]

I once tried the underscore to lodash refactor, just to get a sense of the scope of the problem. It's a gnarly refactor. The API for underscore is large, and used extensively throughout the Meteor codebase. As I recall, there were thousands of broken reference the first pass through; which were actually fairly straight forward to clean up; but once the references were cleaned up, there were hundreds of broken calls; related to dozens of mangled files. It was layers and layers of refactoring.

The best way to do the refactor, in my opinion, is to divide it into parts, and be slow and consistent. @jdnichollsc is probably right. The best bet is to bring both lodash and underscore into the codebase, and rewrite one package at a time. Separating the packages out into independent NPM packages is probably a good stepping-stone. If a package can be installed independently via NPM, it's probably decoupled enough that it's underscore references can be cleanly replaced with lodash.

[stubailo]

I wonder if TypeScript or Flow could be helpful in using a computer to identify places where the API is different.

[awatson1978]

API differences are documented in the Migrating page.

Underscore _.compose is Lodash _.flowRight  
Underscore _.contains is Lodash _.includes  
Underscore _.each doesn’t allow exiting by returning false  
Underscore _.findWhere is Lodash _.find  
Underscore _.flatten is deep by default while Lodash is shallow  
Underscore _.indexBy is Lodash _.keyBy  
Underscore _.invoke is Lodash _.invokeMap  
Underscore _.mapObject is Lodash _.mapValues  
Underscore _.pairs is Lodash _.toPairs  
Underscore _.pluck is Lodash _.map  
Underscore _.uniq by an iteratee is Lodash _.uniqBy  
Underscore _.where is Lodash _.filter  
Underscore _.isFinite doesn’t align with Number.isFinite  (e.g. _.isFinite('1') returns true in Underscore but false in Lodash)  
Underscore _.matches shorthand doesn’t support deep comparisons (e.g. _.filter(objects, { 'a': { 'b': 'c' } }))  
Underscore ≥ 1.7 & Lodash _.template syntax is _.template(string, option)(data)
Lodash _.memoize caches are Map like objects  
Lodash supports implicit chaining, lazy chaining, & shortcut fusion  
Lodash split its overloaded _.head, _.last, _.rest, & _.initial out into _.take, _.takeRight, _.drop, & _.dropRight  (i.e. _.head(array, 2) in Underscore is _.take(array, 2) in Lodash)

As I recall, this was the second or third step in the process; and where I hit a wall (in part, because I didn't have the above documentation).

The first step was to replace the Underscore references with Lodash. That breaks the builder, because it can't load up all the files. Once all the references are updated, then Meteor will start going through the build pipeline. Unfortunately, the build pipeline involves things like _.contains calls and _.each calls which exit by returning false. So then it starts complaining about the API not working. When you fix some of the API calls, it will get through one of the steps in the pipeline, but then complains about subsequent steps.

I'm handwaving a bit, since it's been at least a year since I tried that refactor, and I don't recall all the details. But it's a metric ton of errors.

[stubailo]

I wonder what the impact on user apps would be of a change like this. I suppose as long as we provide a shim to use underscore for parts of the app during a transition period it could be fine, but if it's this hard for the Meteor code base it could be even harder for apps.

[mitar]

BTW, this is the most upvoted issue for Meteor.

[abernix]

Just as a reminder: You can and have been able to use lodash in your own projects for a very long time now (since Meteor 1.3) and if you'd like to use lodash today, do not let anything stop you! While underscore is used extensively by Meteor internals, you can still meteor npm install --save lodash and import lodash from 'lodash' in any project and enjoy its _.glory (_not_ a Lodash method, surprisingly!) I realize that's not the point of this issue but it's worth clearing up (again) the confusion which has and does exist!

I'm here to instill hope that this has not been forgotten and is in the works! Getting a bit more detailed: For those interested in how Underscore weighs in within the Meteor code, I'll point you to the Underscore tally counts I generated in researching this project a couple weeks ago:

• Totals method counts (inclusive of below)
  
  
  
  • Method counts in core packages
  
  
  • Method counts in meteor-tool
  
  

As has been stated throughout this issue, especially by those who have made the attempt to undertake this project (very much appreciated, to those who have tried!), there are a number of obstacles: very old underscore, Meteor-modified, different names, incompatible API, etc. Each underscore method has considerations which must be made and the maze of "change log" entries between "Meteor-mod-Underscore 1.5.2" and [email protected] is extensive. A word of caution to anyone just switching a dependency like this and seeing what happens: prepare yourself for a lot of bizarre edge cases. There is plenty of documentation out there which advises against exactly this.

While one option for this was to openly and willingly accept PRs for this transition and given the vast scope of Underscore in Meteor and the large potential for manual error in making the (sometimes complicated) changes, or even reviewing the changes, I'm currently pressing forward with an automated process to avoid both the time-sink the high margin-of-error.

I am (currently) in the process of writing AST transformations which will programmatically (using recast and jscodeshift, amongst other tools) go through and remove Meteor's dependency on underscore and replace it with lodash or native ECMAScript methods, on a case by case basis. (It's worth pointing out that @wmertens made a wise suggestion about using AST transformations earlier in this thread).

I'm aware of existing codemods which attempt to make this transformation automatically, but in trying those, I've certainly ran into many bumps, some of which are unique to the Meteor core (though rest assure, I'm taking solace and inspiration from their existence). We also have considerations of making sure we play nicely with those who want to keep using underscore and ensuring efficient bundling (a current Roadmap goal).

I'm hope that some of the tooling I'm building will be helpful to others too (even if this migration might be less common now than in the past) and I hope to share more info soon!

[pozylon]

In my opinion, while swapping underscore with lodash is a waste of time, swapping underscore with native functions is a plus.

https://github.com/you-dont-need/You-Dont-Need-Lodash-Underscore

In propably less than 2 years somebody will open an issue with the title: "Investigate swapping lodash with xyz"?

[jdalton]

@pozylon You might check out this post. Lodash is a collection of ~300 modules, considerably more than the handful of ES5/6/7 methods, and offers features beyond those of their built-in counterparts. Lodash works great combined with ES6 and there's even a lodash-es package. You might also dig that Lodash v5 is being written in ES6/7 too.

Unlike Underscore, which has been dormant for ~2yrs, Lodash has evolved with the language and ecosystem. This is also why Lodash continues to gain users with over 1,000 new package dependents each month for the last 2 years, more than 30,000 packages directly depending on it, and more than 110,000 packages (30% of npm) impacted by it in some way.

[sebakerckhof]

@abernix How will it be packaged? As a meteor package, an npm dependency in the package.js or just as peer dependency so users can use their own version (within limits)?

It'd be interesting to have separate counts for files that are server-only and files that are included in the client bundle. I think one of the major concerns is bundle size. Although this could be alleviated by only requiring the necessary lodash modules, at least until there is support for tree-shaking.

[pozylon]

@jdalton I truly appreciate what you have done for the whole js community and I use lodash myself.

Still I think getting rid of dependencies is preferrable over replacing dependencies if +/- same amount of time has to be invested. Lodash is more popular than ever and exxtremely well maintained but underscore too had some fans before the 2y rapunzel sleep...

[jdalton]

@pozylon

Still I think getting rid of dependencies is preferrable over replacing dependencies if +/- same amount of time has to be invested.

Lodash is modular, so folks can use as little or as much as they'd like. It makes sense to use something that's battle tested rather than reinventing the wheel of fundamentals.

Lodash is more popular than ever and exxtremely well maintained but underscore too had some fans before the 2y rapunzel sleep...

Judging by how long this issue has existed maybe Lodash's future is the least of the worries here. I understand your concern, but I think it's a bit premature/unwarranted (has the slight ring of NIH).

Update:
The work @abernix has put in is a pretty rad solution.

[mitar]

@abernix, are you thinking about something like what @pozylon posted above? So swapping underscore with ES6, and then having Babel translate that back to backwards compatible things if needed?

[abernix]

@sebakerckhof That's a great question! I've gone through a couple iterations/preliminary attempts at figuring the best solution for this. While a peer-NPM dependency implementation might be one way to avoid duplication and bundle bloat, the current implementation of Meteor's NPM peer dependency system wouldn't trickle up as far as the Meteor server bundle itself – a place where a lot of current _ references live and a reason why underscore is still included in Meteor itself right now.

Currently, I'm creating a lodash Atmosphere package, however the usage recommendation will be different for lodash than it was for underscore.

The current underscore loads the entire Underscore library into your application, regardless of which actual methods you use. The lodash package will only bundle and load what you use, will a little help from the developer.

Internally, the lodash package will contain files for each Lodash method, provided in one of two automated ways (at lodash publish-time): either by using lodash-cli to generate modules directly, or possibly just by offering providing a stub which re-exports lodash/moduleName from the NPM itself via Npm.depends – I haven't decided yet. Either way, it will not explicitly add them to the package using api.addFiles (which would eagerly load them).

The responsibility of the developers of Meteor apps (and packages) will then be to import or require the lodash package and specific method (or core) in a similar fashion as they do with other Atmosphere package exports by using the meteor/ prefix. They can either use core (for the entire core build) or ideally, Lodash's cherry-picked methods:

import methodName from 'meteor/lodash/methodName';

or

import _ from 'meteor/lodash/core';

In a similar way as to how underscore is used now, this will allow a single lodash library to exist in the bundle and be used by the Meteor core packages, developer packages and developer apps but allow the import scanner to only bundle lodash methods which are actually used by your app. This means that if you don't use _.deburr (maybe you should, it's cool!) then Meteor won't bundle deburr.

Also, since Meteor 1.5 will support dynamic imports, you could benefit further in some cases if you loaded more beefy lodash methods like debounce or template using dynamic imports:

import("meteor/lodash/debounce").then(debounce => {
  // Some debounce-y fun, if that's even possible.
  debounce(fun, 2000);
});

In a future reality, where Meteor moves more toward full NPM integration, a peer NPM dependency setup will be the natural course for this (as with the rest of Meteor packages). This might seem like a counter-measure to that right now, but I can assure you that this transition does not need to be any more complicated than it is already (nor wait and take any longer). Additionally, in the future, developers will only have to remove the meteor/ prefix to move to a true NPM peer dependency setup, but keeping in mind any incompatible lodash version changes, of course.

@pozylon Somewhere out there, some day, maybe a version of ECMAScript exists that might maybe natively provide all of the features that lodash offers, but I can assure you that it's not any time soon. Removing lodash entirely would be detrimental to the Meteor framework and cause us to have to duplicate a substantial amount of functionality that it provides. I do think that embracing newer ECMAScript standards is a direction that we will proceed in, but Lodash does offer benefits, like chaining which still come in very handy. That said, in cases where it makes sense I am selectively replacing things with native functions. I'll point out that the list of "You Don't Need Lodash Underscore" methods is quite short compared to the list of Lodash functions.

[sebakerckhof]

Yep, seems like a clean way forward.

[jdalton]

@abernix

Internally, the lodash package will contain files for each Lodash method, provided in one of two automated ways (at lodash publish-time): either by using lodash-cli to generate modules directly, or possibly just by offering providing a stub which re-exports lodash/moduleName

You'll have more tree-shaking and optimization opportunities going thelodash/moduleName route than lodash-cli. The cli will not make the jump to v5. It's being retired in favor of modern bundlers.

[abernix]

@jdalton

I did move in the direction of using lodash/moduleName stubs rather than generating actual lodash source modules with lodash-cli however I'm currently generating them by utilizing the includes export from lodash-cli's listings.js as a "dictionary" of lodash methods. Since this will be deprecated in the v5-future, do you have a best suggestion to use as a reference for auto-building our lodash module stubs? Looks like things are a bit under-construction at the moment, so happy to revisit that when it's finalized if you don't have an answer yet!

We won't need this stubbed-pattern long-term, but I'd like to easily generate new versions of the Meteor lodash package (ideally with fp modules as well!) until we can make that full switch.

[jdalton]

@abernix Using the mapping of lodash-cli is fine for v4. If the transition takes a while and v5 is out, there will be other options like using a babel-plugin. You can base it off of babel-plugin-lodash.

[hwillson]

To help provide a more clear separation between feature requests and bugs, and to help clean up the feature request backlog, Meteor feature requests are now being managed under the https://github.com/meteor/meteor-feature-requests repository.

Migrated to https://github.com/meteor/meteor-feature-requests/issues/48.

[abernix]

Just as an update on the status of this (important) task before the updates move over to the feature request repository...

I did complete the work to replace underscore with lodash in Meteor. The only hesitation to applying that officially was that it actually _increased_ the resulting client-side bundle size by slightly more than we would have liked. This was particularly true since the realization was had during the 1.5-push which was directly focused on _decreasing_ client bundle sizes.

I'm aware of the reasons and I will develop a plan soon to resolve that and get this released. This will be even easier with the bundle-visualizer in Meteor 1.5.