Taskwarrior: The future of syncing and taskserver

[jschlatow]

As part of #2338, @pbeckingham mentioned that taskserver is a support nightmare and raised the question for suggestions what to do with it. So, let's start the technical discussion here.

Some suggestions have already been made:

• Remove the support-intensive parts from taskserver and re-use existing libraries for that, like removing the SSL and communication part (e.g. using Pyhon instead).
• Switch to a decentralised sync scheme as done by dstask which uses git.

I think in addition to the question of development efforts, the question of what functionality the TW community would like to see is at least equally if not more important. I therefore invite not only potential contributors and devs to join this discussion but also the users. A good starting point would be to look at existing feature requests.

• #2319 requests for the possibility to sync with multiple servers.
• t.b.c.

[kwoot]

Multiple servers can easily be achieved by using something like Cassandra as a storage backend. Very simple to setup and to cluster multiple servers (I just don't see the usecase for mutiple servers yet).
TW data is not time-sensitive so moments that servers are out of sync is not an issue.

Of course you can also use git for that, just push to multiple servers, but it is more of a hassle imho.

[jschlatow]

@kwoot The multiple server scenario does not aim at availability. The usecase is rather that you have multiple different servers that hold a disjoint subset of your tasks so that they can be shared with different groups/devices. Intuitively, I would not choose to add the complexity of filtering the database into subsets for syncing with different servers as suggested in #2319. I believe the same effect can be achieved by modifying the client to allow mulitple databases to be loaded for the UI. Each database can then be associated with a distinct sync mechanism.

[kwoot]

Wow, I would never have thought of that. And seems to me like a nightmare waiting to happen. But that's just me of course.

[sl4mmy]

Before discussing technical details I suggest we take a few steps back. Since @pbeckingham mentioned his desire to kill taskserver (see: https://github.com/GothenburgBitFactory/taskwarrior/issues/2338#issuecomment-729067633) I think we should question the purpose & goal of taskserver. What value does it provide? What use cases require it?

Personally, I use taskwarrior for a bunch of different projects (both work & personal), and on a bunch of different machines: work laptops, personal laptops, a Linux desktop, a Linode, etc. I basically use it for everything, and I use it in a very simple Getting Things Done-style (i.e. recording everything I need or want to do). Being able to sync all of my tasks across all of those machines is really useful so that I have visibility into everything at all times regardless of which machine I'm on. If I didn't have all of my personal tasks on my work laptop, for example, then I might forget that I need to stop at the store on my way home from work to pick up something one of my kids needs for school the next day.

Being able to filter or sync only a subset of tasks on different machines isn't particularly important for my uses, and I wouldn't use it unless it was really straight-forward to configure & maintain those rules (i.e. without having to re-read the entire man page every 6 months when I have to tweak them).

One killer feature that taskserver presently lacks (I believe?) but would be a huge benefit for my uses would be syncing timewarrior data as well. Sometimes I might spend a little time working on personal projects when I'm at the office, or on weekends I might spend a little time on work projects; being able to accurately track all of that time regardless of which laptop I'm sitting in front of at any moment would be awesome! In that case I could see myself using timewarrior even more than I already do.

What other ways do people use taskserver? What other use cases do people have for sharing tasks across more than one machine? I know somebody recently mentioned in one of the other issues about wanting to setup a cloud-based timeserver instance for a team at work, would love to hear more about that.

[sruffell]

One killer feature that taskserver presently lacks (I believe?) but would be a huge benefit for my uses would be syncing timewarrior data as well. Sometimes I might spend a little time working on personal projects when I'm at the office, or on weekends I might spend a little time on work projects; being able to accurately track all of that time regardless of which laptop I'm sitting in front of at any moment would be awesome! In that case I could see myself using timewarrior even more than I already do.

Just FYI, I do something similar with timewarrior. I've implemented a command to allow one timewarrior instance to import the output of timew export. So to sync machines, I do something like:

 timew export 1w ago | ssh remote-server 'timew import'

Granted, it still needs some more work to handle conflicts, but if there is someone else who would use it I could try to prioritize that:

https://github.com/GothenburgBitFactory/timewarrior/issues/359

[Crisses]

I am using taskwarrior across machines because I'm a dogsitter as well as coder, life-coach, virtual assistant (it may be easier to ask what I don't do). So sometimes I have my laptop, sometimes my desktop, and in a pinch I have a server shell instance and can log in via mobile devices, sync, and see my taskwarrior while on the road or with limited devices with me via ssh to my server.

I suppose I could just ssh and use the one instance from all of my machines & devcies, but having the ability to work offline is handy.

Syncing timewarrior would be handy too, but I suppose, again, I could probably ssh into my server and use timewarrior via ssh. I'm on mac devices, there's a perfectly good shell on my device, though. I don't always have amazing connectivity to the Internet (believe it or not).

The main difference is the number of steps to get to my task list. Am I opening a shell and one command from my task list, or am I opening a shell, logging in, typing a bunch of passwords or decrypting my key, etc. Generally you want to be able to quickly view a task list and update it or add to it. Also the Taskwarrior Pomodoro and other plug-ins rely on local sync'd installations.

[corbolais]

@Crisses interesting.. My daily routine sounds similar.

I'm using multiple devices w/ individual TW instances, tasks sync'ed via taskd. I very much appreciate the SSL enc and certification use. These I do not, must not miss. In fact, it wouldn't be of much use to me switching to Let's Encrypt certs as I will not be able to acquire certs for internal hosts. No much use there.

Whenever I open up a shell instance, task sync is called, the list of open/recurrent tasks updated, sync'ed w/ git, sync'ed to taskd. PROMPT task indicator. I'm using watson instead of timeW. Need to implement some change though as the project description and TW/watson tags have different meanings (shows in watson report). watson sync is achieved by, yes, git. watson report log is plain ascii, merge and go.

In addition I use TW in a localhost git repo to have a fallback, I once lost the TW history for reasons I no longer recall but the solution to that is git (again).

TW config is shared (sync'ed, via git, glad you asked) between hosts via vcsh (so happy that exists, if not I had would've had to implement smthg like it).

TLDR: LE certs are welcome but of no use in my scenario. Do not drop SSL cert support. Great thing taskd is enforcing SSL cert usage.

[brfid]

Taskserver was enough of a headache that I just synced across devices with dropbox (and a third person ical connector until it disappeared). It seems that users that would have had to spend too much time on task server (like myself), they can get by with brute force solutions and not really care (that's me with dropbox at least).

[mehdisadeghi]

FWIW I suggest to look at DecSync. A combination of DecSync and syncthing has been working extremely well for me. DecSync provides a conflict-free data format and syncthing does the synchronization.

[djmitche]

I run freecinc right now, and it is kind of a headache. It leaks memory and crashes a lot, for one thing, but slowly enough that Kubernetes can keep it more-or-less operational.

More importantly, the sync algorithm fails in a number of critical ways. So, for me various replicas stop syncing once or twice a week, or continue syncing but have different sets of tasks (for example a task might be completed on one replica but not another).

Another issue is that taskd has no encryption. I can easily read the task data of everyone using freecinc. I'm a little frustrated that taskd uses TLS for transport (and, it's kind of a headache to get that set up correctly!) but then stores the data in cleartext -- I think it gives users a false sense of security. _Any_ sense of security wrt taskd is false, to be honest.

My personal use-case is simliar to @Crisses -- I have task lists on my various laptops, my VPS, and my phone. I sync lots of data into taskwarrior using bugwarrior. I appreciate that I can have a local copy of my tasks when I don't have access to a server -- for example, my +buy tasks are useful in a store, where internet access might be unavailable. I also appreciate that if any of those replicas fails, my data is not lost.

Issues with taskd drove me to start working on a more formally-grounded sync algorithm, based on operational transforms. That's become TaskChampion, with the sync algorithm described here. This approach has a bunch of advantages, including encryption at rest, backups of both the server and replicas, possibility of a shared backend datastore, and constant space requirements. Unfortunately, the model isn't compatible with the TW data model, which is most of why I dropped TW compatibility from the TaskChampion plan.

I'm writing this to provide information, not to try to convert anyone to TaskChampion -- it's not (yet) a usable tool, anyway! But maybe it spurs some thinking that will help move TW in a fruitful direction..