Node: Require from internal folder

[streamich]

This might be a somewhat controversial proposal: allow users to require from internal folder, like require('internal/fs.js').

One can already use the hidden modules in the /lib folder, like:

require('_http_agent');
require('_http_client');
// etc.

However, I am not aware of a way to access the modules in /lib/internal folder. Does anyone know a way to require those? Are there any issues with that?

I know that in this form it would conflict with the internal package, so the exact syntax has still to be figured out.

This could be provided as "use at your own risk".

[refack]

It's available as an undocumented cli switch. I'm not sure I want to name it explicitly here, but if you grep /test/ for internals you'll find it.
I'm closing as "done" 😉

[edit]
Feel free to reopen if you think I'm wrong or if you have more information.

[streamich]

1. What is the CLI switch?
2. Is there a reason why accessing the /internal folder should be allowed only with a CLI switch, for example. as mentioned above the user still can access the require('_http_agent') module, as one example. As I understand it, there is no way to require from the /internal folder only by accident, because module.js resolves in such a way that it will search for internal "package" in ./node_modules/.

[bnoordhuis]

We added lib/internal + the switch precisely because people kept mucking with node's internals. The switch is there for our test suite, don't use it.

[refack]

To elaborate on @bnoordhuis comment.... Since JS does not have a formal notion of private encapsulation, we need to "invent" mechanisms for separating supported APIs (public) and unsupported endpoints (private / protected).
If there was no CLI switch a module author could use those unsupported endpoints and thus inadvertently expose their users to breakage. The CLI switch makes sure the end user explicitly gives consent to run fragile code and TMMV.
The most impactful such story is graceful-fs which is a useful and super popular library, but was dependant on unsupported endpoints. When those changed in node-core the fallout was huge (which eventually lead to the whole internal mechanism and the institution of the CitGM project).

@streamich if you feel that a specific endpoint (or groups of endpoints) should be made public, feel free to raise a PR and IMHO it will receive proper discussion.

[streamich]

@refack, thanks. Would be great to expose the internal/errors.js in the future for other libraries to be able to throw similar/same errors that Node does.

[refack]

@refack, thanks. Would be great to expose the internal/errors.js in the future for other libraries to be able to throw similar/same errors that Node does.

That's coming, probably even in an npm package.
Ref: https://github.com/nodejs/CTC/blob/master/meetings/2017-07-19.md#nodejsnode-1

[szmarczak]

We added lib/internal + the switch precisely because people kept mucking with node's internals.

What do you exactly mean? @streamich's reason is very legit.

[devsnek]

if a specific api is needed it would more constructive to discuss exposing that specific api.

[szmarczak]

I'm particularly interested in errors and url-to-options. But I believe other Node.js power users would want other modules too.

[bnoordhuis]

@szmarczak Can you file issues for each?