Tiddlywiki5: Reload changed tiddlers from disc

[dufferzafar]

Is there a faster way to load changed tiddlers from disk into the browser without restarting the Node server?

Google Groups post didn't mention any solution but it is 2 years old. Has there been any development on this?

[j-xella]

+1 on this. Useful when TW is on dropbox, for example.

[sukima]

I have mused over this for some time. A few things need to be addressed:

1. You have to watch a directory is is always a troublesome task but there are Node.JS libs available to help.
2. Once a watching process realizes there is a change how do you sync this? Many apps assume you are editing in another browser so they can use livereload to force the browser to refresh the page on a change. But this is problematic in TiddlyWiki's design since you would loose you place on a full reload without permaview always on.
3. Since TiddlyWiki handles changes made in the app how would one synchronize unsaved changes in the browser with now saved changes on disk? The conflict would be quite significant and could lead to increased human error and lost data.

I think this is an excellent place to have some discussion around these problems and hopefully develop a possible solution or consensus on how to approach implementing such a feature.

[tobibeer]

@sukima

1. You have to watch a directory is is always a troublesome task but there are Node.JS libs available to help.

Any particular modules you have in mind / experience with? There appears to be chokidar and variants of it and the underlying fs.watch().

1. How do you sync this? Many apps assume you are editing in another browser so they can use livereload to force the browser to refresh the page on a change.

I think the key here is a "partial page refresh". I don't think reloading the full ui is necessary. The sync adapter is the one to ensure tiddler representations are refreshed in the story(s). So, the key is chaining up the syncer with the importer to come to a managable form of collision management.

1. How would one synchronize unsaved changes in the browser with now saved changes on disk?

1. conflict recognition, i.e. not brute-overwriting
2. conflict handling, i.e. merge tools
3. conflict resolution, i.e. commiting the eventual result of a merge

All of the above presupposes some form of change management and quite possibly a version history as well. So in that sense, a Git-/GithubPlugin (or even using Github for the very purpose of a tiddler store) might at first appear to yield too complex a process. On the other hand, it would provide exactly the type of conflict management that is needed based on a solid revision history. So, the task of a Github-syncer would in-fact be to relay the git status to the frontend... and require the user to be able to handle the interface, while simplifying the process to as little as necessary... and leaving out any of the complex merge scenarios.

[pmario]

Any particular modules you have in mind / experience with?

Some existing libs, only work well for 1 OS category. Those, that work well with unix, sometimes can't cope with windows and vice versa. .. IMO we should have a closer look to existing build frameworks, that allow you to activate a watch mechanism. ...

I'm using webpack -w a lot to create the file-backups AddOn. It works very well on windows and ubuntu. ... But I don't know macOS

[pmario]

it seems webpack uses: https://www.npmjs.com/package/watchpack

[inmysocks]

This is a bit old but the multiuser plugin has a part that does this. It will hopefully eventually be split into its own plugin. In case anyone is interested, the relevant parts in what I made are:

• A web socket server to handle the two-way communication between the browser and the file system
• A new sync adapter type that uses the websockets
• The file system monitor part which is based on nodes fs.watch and sends a message to the sync adapter when there are changes

This can probably be done without websockets, but using them makes it easy to update the page without doing a refresh. The way I have it set up tiddlers change the same way as if you had done the editing in tiddlywiki so you don't need any full page refreshes.

[sukima]

Yup, livereload uses websockets as well. Seems to me the solution that the multi-user plugin created is the correct solution here. Can we get that sync adapter split-out from the plugin? maybe we could bundle it into core in the same what the filesystem plugin is now.

[inmysocks]

I am planning on splitting it into separate plugins and trying to get them added to the core once I get it working well enough. Hopefully that will be soon.

[xeor]

Any progress on the plugin @inmysocks ? Having the files stored in git would mean a near perfect personal note/journal-taking solution.. Willing to donate some money to get this thing rolling.

[inmysocks]

Bob has come a long way since March but I have had very little success splitting it up into different plugins.

What changes from the current version would you want in order to make using git with it easier? I considered making a plugin to go with Bob that would let you access gits functions from within a wiki but I haven't gotten around to it yet.

[xeor]

If a change is happening via Tiddlywiki, it should commit (commit-message not that important) and optional push (to a configurable remote, default to origin)

The other part would be more tricky.. Maybe a git pull on a configurable interval would be enough..

Alternative, this git part could be a whole separate thing, then the only thing needed would be for TiddlyWiki to detect if a file have changed, and make it available in the UI.

To be honest, I think the latter would be the best way to do this. It would then be upto the user to do the sync, either with using tools like https://www.cis.upenn.edu/~bcpierce/unison/, or other syncing tools. Creating a job that does git pull, git push from time to time isn't that hard..

[inmysocks]

What you are describing is exactly what Bob does. If there is a change on the file system than it is immediately changed in the browser, or if there is a change in the browser it is immediately sent to the file system.

I am eventually going to make a git plugin that automatically does the git part but for now I have been doing it manually.

[xeor]

Thanks for the info.. I did not know what Bob was, so I assumed it was a person working on the problem... :) I found https://github.com/OokTech/TW5-Bob/, and it looks very promising.. I'll check it out. Thanks

[linonetwo]

But bob has a bug that prevents me from using it https://github.com/OokTech/TW5-Bob/issues/130

I'm going to make its https://github.com/OokTech/TW5-Bob/blob/master/FileSystem/FileSystemMonitor.js a standalone plugin, so I can have this bidirectional sync before the fix https://github.com/Jermolene/TiddlyWiki5/pull/4630 is merged.

Or maybe I can change the Filesystemadaptor as said in https://github.com/Jermolene/TiddlyWiki5/issues/1617

[saqimtiaz]

@linonetwo I think writing a plugin or running a patched version of the core is your best option until #4630 is properly implemented.

[pmario]

@dufferzafar, @Jermolene ... I think this one can be closed. latest node server version can do it.

[linonetwo]

@pmario Really? I tried changed one of my tid file, but noting changed in the browser.

[pmario]

You need to wait up to 1 minute or so. There is a sync timer somewhere.

[linonetwo]

@pmario will this happen in 5.1.22? Or 5.1.23-pre release?

My 5.1.22 nodejs wiki didn't change. And if this is not programmatically controllable, it won't be a good solution for me, because I want to use this feature in my Electron App.

[saqimtiaz]

As far as I am aware, the node.js server currently does not reload changed tiddlers from disc.

What it does do now, is send changed tiddlers from the node.js server (that is in memory in the server) to clients. Therefore editing a tiddler file has no effect until the server is restarted.

[pmario]

@saqimtiaz ... seems you are right. I did have this example in the head: https://groups.google.com/d/msg/tiddlywiki/PW0_ZCyD5SM/CZZRbw_wAgAJ ... So if you use curl to create new content, it definitely gets synced. I did just try it.

curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New' --data '{
 "tags": "FirstTag [[Second Tag]]",
 "my-custom-field": "Field value",
 "text": "your text comes here!\n\nnext paragraph"
}' -H "X-Requested-With: TiddlyWiki"

The only problems users seem to have is, that they want to modify the title field, which is not possible. One needs to modify the URL eg:

curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New'
curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New 1'
curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New 2'

and so on.

[linonetwo]

I tried to do this:

// on delete
      if (event == 'remove') {

        // can't use $tw.wiki.syncadaptor.deleteTiddler(tiddlerTitle);  because it will try to modify fs, and will failed:

        /* Sync error while processing delete of 'blabla': Error: ENOENT: no such file or directory, unlink '/Users//Desktop/repo/wiki/Meme-of-LinOnetwo/tiddlers/blabla.tid'
        syncer-server-filesystem: Dispatching 'delete' task: blabla 
        Sync error while processing delete of 'blabla': Error: ENOENT: no such file or directory, unlink '/Users//Desktop/repo/wiki/Meme-of-LinOnetwo/tiddlers/blabla.tid' */

        $tw.syncadaptor.wiki.deleteTiddler(tiddlerTitle);
      }

Seems

1. $tw.syncadaptor.wiki.deleteTiddler will try to modify disk, is there a method to only modify wiki, but not the disk?
2. Seems $tw.syncadaptor.wiki.addTiddler(tiddler) will not sync the change to the wiki, I still have to reload wiki in the Browser to see the change, why?

[linonetwo]

This is my current watch-fs plugin:

https://github.com/linonetwo/wiki/blob/6abb5904edfc77a39cfbdf2a8058f3a649be6f3f/Meme-of-LinOnetwo/plugins/linonetwo/watch-fs/FileSystemMonitor.js

It handles create and update properly, but when I call $tw.syncadaptor.wiki.deleteTiddler(tiddlerTitle), there are intervaled error:

syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'

And change is stay in the server, I don't know why it doesn't sync to the wiki automatically.

[linonetwo]

I managed to create a plugin to reload file that changed in the disk:

https://github.com/linonetwo/tiddlywiki-plugins/tree/master/plugins/linonetwo/watch-fs

There are still some limitations, such as

1. can't handle rename in the disk, you can only rename from within the wiki (no such API to tell tw I've renamed a file)
2. I'm now refreshing the wiki using $:/config/SyncPollingInterval as said in https://github.com/Jermolene/TiddlyWiki5/issues/2698#issuecomment-670834997 (no such API to tell the browser to refresh)
3. I haven't tested this with $:/config/FileSystemPaths and https://github.com/Jermolene/TiddlyWiki5/pull/4630 , but I use this feature every day, so I will definitely support it
4. Can't handle if git change the tiddler while you are open its Draft tiddler (might be fixed by https://github.com/Jermolene/TiddlyWiki5/issues/4792 )

Here is the working screenshot about how it works with git:

[pmario]

@linonetwo Hi, Jeremy somewhere mentioned, that he would be interested in "Server sent events". The mechanism is much easier as web-sockets.

Since TW server doesn't use 3rd party libs, the most basic examples I could find was: http://html5doctor.com/server-sent-events/ from 2012. It also links to a github repo: https://github.com/remy/eventsource-h5d/blob/master/app.js

IMO it should be interesting to include this function.

[linonetwo]

@pmario I can see it is actually poll from the client, to see if there is an update in the server, if so, it triggers a pull from the client.

I implement this in my watch-fs plugin:

client-side poll for sync state, if server has something new, it triggers $tw.syncer.syncFromServer():
https://github.com/linonetwo/tiddlywiki-plugins/blob/c494c2edd3e1b3c066e2032dc816943a38e5a9ca/plugins/linonetwo/watch-fs/trigger-sync.html

fs watcher debounces, wait for the disc to be stable:
https://github.com/linonetwo/tiddlywiki-plugins/blob/c494c2edd3e1b3c066e2032dc816943a38e5a9ca/plugins/linonetwo/watch-fs/FileSystemMonitor.js#L117-L129

A simple server to handle the poll:
https://github.com/linonetwo/tiddlywiki-plugins/blob/c494c2edd3e1b3c066e2032dc816943a38e5a9ca/plugins/linonetwo/watch-fs/get-can-sync.js

[hoelzro]

If you are interested in Server-Sent Events, @Arlen22 has an open PR adding that functionality: #4609