Node: Request for several features in Node core

[ghost]

We have recently implemented a very large project in node.js, and while the memory still fresh I wanted to summarize why node.js was perceived as a nice toy by most of the developers, and not as a mature language suitable for such a projects.

Don't take me wrong, I'm not here to criticize, or to start another holy war. My goal is to list the things we encountered in a hope that they will be given a priority in the next releases.

I cannot reveal the details of the projects, but lets say it was your average web application + algorithmic part. Naturally, we didn't have any expectations from node when it came to the algorithmic part, but we hoped the web part will be easy to implement. How naive of us, huh?. :)

The major problem with node:

Most of the NPM modules are heavy, slow, not well thought out, and resemble the classical dependency hell. Therefore you are forced to constantly reinvent the wheel.

Here is our reinvention list:

1. CSV parsing
   2, XML/HTML parsing
2. IPv4/IPv6 operations
3. parsing of HTTP headers
4. RSA keys
5. SSL certificates (read ASN1)
6. MySQL drivers (yes we tried mariasql)
7. Sendfile
8. SMTP support
9. Asynchronous logs
10. Zip/Unzip

Its a basic and widely used stuff, and it was extremely discouraging to find out that node does not support natively anything in that list. Basically, we've spent most of our time doing stuff we shouldn't have been doing.

Cheers,

[evanlucas]

Node is meant to be the small inner essentials. We try to keep things that can be done outside of core, outside of core. We've found that doing that tends to help foster experimentation within the ecosystem. Having packages in the npm ecosystem instead of in core allows those packages to version themselves independently. To me, this means that a bug in a downstream dependency can be fixed in minutes vs hours.

Most of the NPM modules are heavy, slow, not well thought out, and resemble the classical dependency hell.

That is a pretty broad statement...

[sam-github]

Not sure what you hope to achieve here, node has never had those features, and isn't likely to (well, maybe sendfile).

I doubt this is a useful issue, but I am personally curious to hear people's node experience. Maybe you can expand each item with what the problem was that you had, and how you tried to resolve? For example, what was was the problem with the npm modules?

As for unhappiness with the modules in the worlds largest module collection...

Heavy? No idea what that means, bits are weightless. Since I understand you to be asking for a kitchen sink of features to be added into node core itself, you surely aren't talking about feature bloat, are you?

Slow? well, it is javascript, and node doesn't execute its js faster than other modules.

Classical dependency hell? Not sure what you mean, classical dep hell is when sink needs pipe 1.x, and stove needs pipe 2.x, and so you can't use sink and stove in the same kitchen app. node doesn't have this problem, of course, pipe 1.x and 2.x can co-exist, so there is no "classical" dependency hell. What kind of hell did you experience?

Also, I don't have a clue what 2 or 4 are, node clearly supports IPv4 and v6, as well as RSA keys.

[addaleax]

@stiels I’ve edited the issue title to better reflect what you’re asking for here, the original title didn’t really say much (and was far from funny or anything, if you were going for that).

[ghost]

Node is meant to be the small inner essentials. We try to keep things that can be done outside of core, outside of core. We've found that doing that tends to help foster experimentation within the ecosystem. Having packages in the npm ecosystem instead of in core allows those packages to version themselves independently. To me, this means that a bug in a downstream dependency can be fixed in minutes vs hours.
Not sure what you hope to achieve here, node has never had those features, and isn't likely to (well, maybe sendfile).

I understand your rationale, it correlates very well with my rationale when I choose C/C++ for certain projects. However, the same rationale makes node sort of "hang" between 2 worlds. One world is C/C++ where I'm ready to sacrifice development time for the sake of performance boost. The other world is PHP where I'm ready to sacrifice performance for the sake of development time. In node we had to sacrifice both.

For example, what was was the problem with the npm modules?

Lets take something simple - CSV parsing. If I remember correctly our benchmarks of the existing modules gave about 15min per 1 GB, we reduced that time to about 4min per 1 GB and we're not a bunch of geniuses. You guys could take it to the next level.

To me, this means that a bug in a downstream dependency can be fixed in minutes vs hours.
Classical dependency hell?

Just installed mariasql + libxmljs to remind myself why we got rid of those two. That gave me a total of 52 modules that are in the dependency chain. Some of them were last updated 2-3 years ago. You definitely have an active community and the largest collection of modules out there, but people come and go, leaving abandoned code behind.

Also, I don't have a clue what 2 or 4 are, node clearly supports IPv4 and v6, as well as RSA keys.

There is a basic support indeed. But for one it cannot generate RSA keys, and has nothing that deals with IPs on a binary level (masks etc)

I’ve edited the issue title to better reflect what you’re asking for here, the original title didn’t really say much (and was far from funny or anything, if you were going for that).

Fair enough. :)

[ryan-copperleaf]

Ideal packages can be hard to find, but moving everything into core isn’t the right solution. (Maybe you could publish your faster CSV parser and have it become the standard solution for Node?)

Just installed mariasql + libxmljs to remind myself why we got rid of those two. That gave me a total of 52 modules that are in the dependency chain. Some of them were last updated 2-3 years ago.

This isn’t a problem by itself; much of Node is backwards-compatible.

[cjihrig]

I don't think there is really anything here that Node core is going to directly address, so I'm going to close the issue. Please feel free to continue the conversation, or reopen the issue if there are specific actionable items.

[sam-github]

Node should be able to generate RSA keys IMO, its on my TODO list, feel free to open a feature request for that, with some background as to why, it helps to know there is demand when PRs are reviewed.

If you made CSV parsing faster, PR your improvements to the module you used, or publish your own. Maybe some node collaborators have the talent to do either better than yours, maybe not, but I'd say that if they had the desire, they would have done so already and PRed the improvements to an npm module, or published their own.

bitwise IP ops: open a feature request issue, though unless it is required to be written in c++, it sounds like a candidate for an npm module. You'll have to make the case it needs to be in core, as opposed to npm where it can be versioned and published independently of node, and be useful across multiple node versions.

if your mariasql dep worked, who cares when its deps were updated, whatever they are needed for they apparently are doing. How often do you think python's ElementTree library gets updated? Some things really can be "done". Especially when they are small, well designed, single purpose modules.

[refack]

@stiels please don't take the "close" as a discouragement. Node-core has allot of room for innovation, your issue was just too big to handle.

You should follow up with @sam-github 's last suggestions, so we (node people, you, and the rest of the community), could benefit from your experience.

If you can't be bothered, ping me, I'll at least spin-off some smaller issues.