Tiddlywiki5: Use node_modules as plugin source

[danielo515]

Hello,

This is more like an open discussion about how could we make this possible. Any suggestion and/or instructions about how to make this will be welcome.

The fact is: it is hard to distribute tiddlywiki plugins, especially for development. If you want to use a third party plugin on a wiki under node js there are not many comfortable ways. Cloning the repository of the plugin and making a symlink to the dist folder of the plugin is one of them. Another option is to manually import the plugin, but then you will have to manually do it every time you want to update the plugin, and pack it with your plugin when you distribute it.

Wouldn't be ideal to just declare your plugin dependencies on a package.json? Something like this:

"dependencies": {
   "tiddlywiki":"^5.1.15",
   "tiddlymap":"*",
   "another-plugin":"1.0.0",
   "d3":"*"
}

This will make starting developing tiddlywiki very easy, all nodejs developers will feel right at home, and instructions about how to use x or y third party plugin on your nodejs tiddlywiki will be much simpler. Also the distribution of specific configurations will become a trivial task. Not to mention that having a bunch of packages related to tiddlywiki on npmjs.org would make tiddlywiki feel more alive and will increase discoverability .

Currently the format of plugin folders is very strict and restrictive, making a configuration like the one described above a manual hard task.

I know about the environment variable TIDDLYWIKI_PLUGIN_PATH, but just pointing that variable to node_modules is not enough, because the naming restrictions plugins have:

• All the plugins of the same vendor should be under the same folder
• No recursion is possible at the moment
• You will need to add each plugin path manually to the variable, totally defeating the purpose

Those restrictions would make impossible to distribute individual plugins as node packages. Maybe it would be possible to distribute collections of plugins. This second approach have several disadvantages:

• You are forced to download an entire plugin collection just for one plugin
• You can not decide which version of certain plugin you want, you are forced to use the one contained at the collection
• I would be forced to name my plugin with a quite weird namespace. I don't want to publish to npm a package named danielo515 containing all my tiddlywiki plugins. I should use tw5-danielo515 and then name mu plugins tw5-danielo515/tiddlypouch and so on.

Thoughts? Corrections ? suggestions?

[pmario]

Cloning the repository of the plugin and making a symlink to the dist folder of the plugin is one of them.

wrong way! We have TIDDLYWIKI environment variables. see: http://tiddlywiki.com/#Environment%20Variables%20on%20Node.js
It's simple to set them, so tiddlywiki will find them automatically. .. I do it this way and it works flawlessly.

[pmario]

This will make starting developing tiddlywiki very easy, all nodejs developers will feel right at home,

IMO the main problem here is, that the node directory structure and TW directory structure don't match atm. ... IMO the node_modules structure and the existing dependency resolutions sucks. Especially on windows systems, where it causes path names that the os can't handle well.

[pmario]

... don't understand me wrong. ... I'm in favour of having a simple plugin distribution mechanism. ...

I just think that npm isn't the right tool atm. TW plugins can't be used without TW core. ... so they are pretty much useless for existing node devs. .. So they may be more confused, than happy about the stuff. .. just some thoughts.

[danielo515]

Dear @pmario

IMO the main problem here is, that the node directory structure and TW directory structure don't match atm

The entire issue is about finding a solution to that problem.

wrong way! We have TIDDLYWIKI environment variables. see:
I do that too, but it is also cumbersome and very manual . But the way you get the code is the same, clone the repo if it is available.

I just think that npm isn't the right tool atm. TW plugins can't be used without TW core. ... so they are pretty much useless

That is not a problem at all. There are many projects that use npm to distribute their plugins and there is nothing wrong about it. Have you ever used a gulp plugin without gulp ? Have you ever installed a yeoman generator without having yeoman installed ? Typescript type definitions ? PouchDB plugins ? Babel plugins ? Angular extensions ? Just to name a few, and I don't see anyone going crazy about that .

Maybe npm is not the perfect tool for the task, but it is the best we have , with a widespread community and totally free.

[sukima]

I personally would support an effort to add npm compatibility to TiddlyWiki. One of the driving force for many open source projects is plugins. For example Ember.JS would not be where it is today if it were not for its thriving addon community.

I realize the complications involved. Especially the need for a consistent (and enforceable) set of plugin standards. This feature would be a huge lift. I still feel it is worth discussing.

Can we start collecting requirements and technical gotchas. Maybe addressing the smaller details individually will shed more light on the topic.

[danielo515]

Implementing #2835 will help with this. For example using a tiddlywiki command that installs the plugin from npm and then adds the plugin dist folder to the plugins path array.

[sukima]

In the Ember world an addon is an NPM module which includes the following in their package.json:

"keywords": [
  "ember-addon"
],

When the build starts it will search the node_modules for any packages with that in their package.json and know those are addons specific for Ember and _not_ normal npm modules.

Much like TW plugins are different you can use npm to host and distribute but they are inert unless run with the tiddlywiki command which is smart enough to hunt out those modules with the tiddlywiki-plugin keyword.

[danielo515]

@sukima what do you think about my proposal at #2835? Do you see it as a viable approach for this ? What I like about your proposal is that it scans the available plugins at each start

[andypalmer]

I've been experimenting with TiddlyWiki and had the same desire to add plugins and themes to the package.json file. I'm also installing and running TiddlyWiki from the package.json file rather than installing globally (i.e. npm run tiddlywiki -- . --server).
Being able to install the wiki-core and plugins via package.json means that we're describing the infrastructure and the content separately, which I like in a separation of concerns way.

While based in Ruby, the https://github.com/twp/little-plugger mechanism is a nice one for plugin fetching and discovery by the core. There may be an equivalent in node that could help add this functionality to TW