Taskwarrior: [improvement] decentralize TW ! manage multiple task databases at once

[sucrecacao]

Origin of the problem :thinking:

I like to have multiple .task folder compartmentalized. I will create one for each of my side-project or each one of my courses or what not. I store them locally, at the root of the project's directory. For e.g. my home folder could look like this:

├── side_projects
│   ├── bar
│       ├── …
│       └── .task
│   └── foo
│       ├── …
│       └── .task
└── uni
    ├── alorithm
    │   ├── …
    │   └── .task
    └── linear_alg
        ├── …
        └── .task

For the moment I jump from one todo list to another.The downside is that it would be nice to have control on more that one todo list at the time. That why I would like for taskwarrior to be able of managing multiple .task folder at once.

For example, if I want to manage the tasks for all my university courses, I could run something like ( assuming the same tree as above ):
find uni/ -name '.task' | task
The command above would list all tasks from uni/algorithm/.task and uni/linear_alg/.task.
same with my side project ( but with other command )

cd side_project
 find .  -name '.task' | task

or to manage all my computer tasks:
find $HOME -name '.task' | task

Why not using task warrior "project" attribute ?

They are equivalent in the sense that we use the folder path instead of the task warrior's project attribute. We do:

cd uni/algorithm
task add readXXX

instead of:
task add readXXX project:uni/algorithm

but the first offer way more advantages:

Less keystroke => manage task faster :racehorse: and less finger pain :v: !

I notice that I usually focus on one project at the time and I manage my todo list the same way. In other word, instead of the painfull folling workflow

task add read page ... project:uni/algorithm
task add do exercise ... project:uni/algorithm
task add send mail TA project:uni/algorithm
open uni/alorithm/algoBook.pdf

we could simply do:

cd uni/algorithm
task add read page ...  <- add task to the uni/algorithm/.task folder
task add do exercise ...
task add send mail TA
open algoBook.pdf    <- I'm already is the right folder if I want to start working 

Sharing todo list with other people :people_holding_hands:

Let's say that I work on my foo side project with a friends that also use taskwarrior or inthe.am . Now we can share a todo list in our git repo that only focus on foo's tasks. But on our local machine, we can both append those tasks to our main todo list.

Sharing todo list on different computers :computer: :desktop_computer:

Let's assume that I store all my job related tasks on a particular database ( for e.g. in ~/work/.task) and I host this DB on a github repo so I can sync it with my work computer. When using my work computer I am very happy to pull strictly work related task and not all of my personal life tasks.

and more... :fire:

Have decentralized todo list naturally enhance elegant behavior. For e.g. if the project foo has no reason to exist anymore and I delete it from my computer. This will also delete the contained .task folder. Therefore all the related tasks will disappear at the same time. Awesome ! I don't have to manually delete them.

conclusion

By making taskwarrior modular and decentralised, it will allow user to have more freedom on how they manager their task. Plus, it wont overlap on any existing task warrior feature :)

[Arwalk]

You can override the configuration (taskrc) and data repository using the TASKRC and TASKDATA environment variable respectively.

exemple:

TASKRC=~/uni/algorithm/.taskrc TASKDATA=~/uni/algorithm/.task task list

Using stuff like direnv ( https://github.com/direnv/direnv ) you could get exactly what you are asking already.

[ahillio]

Thanks for sharing those details @Arwalk. I think there's an aspect of this which can not be achieved currently... in the aggregation of multiple task files into a single one....

So if I have separate task files for various projects that I'm working on (example being task data included in a git repo for a software and having numerous projects like that) as well as a personal task file and/or others.... what's desired is to have a way to combine them all together into a single file (temporarily) and then save the changes back to their respective files.

Currently, an individual task file can be loaded, but then you can't see your whole schedule because pieces of it are kept in other task files so you don't know what dates are available/occupied by the work you have scheduled in other files and thus aren't able to easily schedule dates for the currently loaded file.

What's needed (which I don't think exists(?)) is a way to merge these together, and then split them apart so that the changes get saved back to the proper/original files.

Could taskwarrior be improved to accommodate this? Is this something that would be better as an external script/software? I see a lot of value in being able to accommodate this workflow :)

[Arwalk]

Please note that i'm not a taskwarrior developper/contributor, only a user, so my answers may not have a lot of weight or value.
Also note that i can't read properly, because i indeed did not notice the part about "merging temporarily", so once again, my comments aren't worth much more than the value you want to put in it.

Now for more of my opinions.

Most of the functionalities you are asking for are available, except for the split files themselves. Contexts, projects, UDAs and reports already give you a lot of flexibility, and should be enough to get the core of what you are asking for. Again, if you really want to split files, then it's just about configuring your TASKRC and TASKDATA appropriately depending on the project you want to work with. Or to use multiple TASKRCs for multiple TASKDATAs, the sky is the limit.

What i'm afraid of, with this approach, is that it seems to raise a lot more problems than it solve.

What i appreciate a lot, personally, in taskwarrior, is the excellent stateless design that this software adheres to. Each "task "'s result is purely calculated through the input parameters that are TASKRC, TASKDATA, and the command itself. There is no in-between state, and no side effect for the commands. The reports are purely a reflection of the database at the current moment, and the modifying commands apply themselves to the current moment too.

What you ask either requires to break out of this, or to make a lot of work _just_ for this functionality that can already be, albeit a bit awkwardly, available.

I'm just going to explain the kind of problem i can already see to implement what is asked. Let's just talk about the simple case of task ID collision.

Let's say you have a TASKDATA_A database and a TASKDATA_B database. They both have a single task, so both these task have ID 1 if you do task list on the databases separately.

Now, let's imagine your functionality. Let's say we now have the possibility for TASKDATA to give multiple path, to multiple database. And you run task list now, hoping to see both your tasks.

Let's try to see how to manage this ID collision.

First approach that i can see is, just append the IDs. The task on TASKDATA_A is ID 1, the task on TASKDATA_B is 2.

You now see two tasks, with ID 1 and 2. According to your work, task 2 is over, so you do task 2 done. But task ID 2 doesn't really exist, it only exists in the __context__ of you having TASKDATA_A and TASKDATA_B loaded at the same time. You introduced the need for a context to interpret the command, TASKDATA_B is not sufficient to answer this command anymore.

The other obvious approach would then to make an entire rework on task IDs to "namespace" them, to get somethink like "A:1", "B:1". This solves everything, except it makes everything much much more verbose. Also, how do you pick the name of the databases? Let the user decide and save it in the TASKDATA directory? Automatic choice? If so, how? What happens if the user loads the database in another order? How do you deal with it?

Also, just imagine this simple use case. You have an alias called taskboth that loads both databases behind the scene using our multiple database path feature for TASKDATA. You run taskboth list, see that the second task is over. Do you trust yourself enough to not run task 2 done by mistake and completing potentially the wrong task?

I do not know the sources of taskwarrior enough to say if that would be easily possible, but in any case:

• Approach 1 involves reading all the databases, building a temporary representation of these, then running the command on it, hoping that the user never fails to run his commands while keeping in mind this "both database loaded" context.
• Approach 2 involves reworking a lot of stuff to identify by-database everything. It also has an impact on UX (and i'm just evoking the case of IDs here).

Once again, what is asked requires a lot of work, for something that would not be used by many users (imho, always), or can be answered through all these tools that are:

• TASKRC
• TASKDATA
• Contexts
• Reports
• Projects
• Tags
• UDAs

I feel that this feature request, a well-intentioned and nicely expressed feature request, illustrate some lack of mastery of taskwarrior's arcane usages. I do not mean to say that in a demeaning way, i won't pretend i'm a good warrior myself. I suspect you could have exactly what you want using some crazy hooks and some really clever stuff using task export and task import, but at this point, is it really worth it? (except for the fun of doing it).

All of this is purely my opinion as a user and should be taken as is. I never meant to be demeaning to you both, and if i was, please don't hesitate to complain about it.

Thanks for reading my blog.

[Arwalk]

On a side note after my rant, here's how you can manage some stuff that was asked. Using stuff like direnv ( https://github.com/direnv/direnv ) (again, i know), you could do this kind of folder structure:

├──.task                            <--- all my data is here for all my projects and home
├──.taskrc                         <--- root taskrc, might have a "home only stuff" context
├── side_projects
│   ├── .taskrc                   <--- sets a context with all side_projects
│   ├── bar
│       ├── …
│       ├── .taskrc              <--- sets a context with only the bar project
│   └── foo
│       ├── …
│       └── .taskrc
└── uni
    ├── .taskrc                   <--- sets a context for all uni stuff
    ├── algorithm
    │   ├── …
    │   └── .taskrc             <--- or only for algorithm stuff...
    └── linear_alg
        ├── …
        └── .taskrc

Using multiple taskrc files, but a single database, you can filter the context of what you're working on easily. Direnv or similar software can load the TASKRC override for you in your environment when changing directories.

PROs:

• it werks
• Not too complicated to do and maintain considering the "include" directive exists in .taskrc

CONs:

• It's a single database, which might not be really what you want
• Must not forget to put project attribute when adding a new task for it to be properly used by this.
• probably a lot more CONs than i have at this hour.

That's just an idea of what you could do, for the sake of giving ideas.

[sucrecacao]

@Arwalk thank you for your feedback !

Most of the functionalities you are asking for are available, except for the split files themselves.

Split files is the only feature I am requesting ! So no, it's not already available and it's not possible to achieve it with task warrior current tools.

What i appreciate a lot, personally, in taskwarrior, is the excellent stateless design that this software adheres to.

I think my request can be done without breaking the stateless design of task warrior.

Let's try to see how to manage this ID collision.

Yes I agree it will create ID collision, but I don't think it's a problem:
From here ID are not essential, they are basically the line number of where the task is written in your *.data file. TW rely on uuid to identify task, and from what I understand, the ID are just used as a shortcut to the uuid. Therefore I think it's a minor modification in the front end to map ID to UUID, but it won't need to change taskwarrior backend's logic. And from the doc you can see that TW's ID are already used in a mutable way .

Once again, what is asked requires a lot of work,

If you are not part of the developer I don't understand why you are making this assumption

for something that would not be used by many users (imho, always),

I don't think so, this would add a lot of value to task warrior, and add a cleaner way to make it interact with other task management system ( trello, github issue ) which are pretty popular.

or can be answered through all these tools that are:

Again, it can not. TW is made to manage one database at the time. You can change to different database with TASKDATA, but it should always by one and only one.

I suspect you could have exactly what you want using some crazy hooks and some really clever stuff using task export and task import

In did one could implement it with external script manipulating different .task but I think it would be a fragil and hacky solution.

[sucrecacao]

@Arwalk
Interesting idea to use multiple .taskrc with direnv ( direnv looks very cool ! ) but still doesn't help me manage multiple todo list at once :/
I think they is a similar solution implemented here:
https://github.com/GothenburgBitFactory/taskwarrior/issues/549
or this extension:
https://github.com/brutus/taskwarrior-local

[Arwalk]

Yes I agree it will create ID collision, but I don't think it's a problem:

My example with ID collisions was purely from a UX point of view. Though i just realized "approach 1" could be solved by always calculating the IDs with all databases in mind, even when only working with one. (posting at 2AM should be forbidden)

Another example of questions it can raise is when you add a new task. If you're in the "uni" directory, and add a task, in which database should this task go? If you prompt the user, you make the UX less fluid. If you make an assumption, whatever it is, someone will complain that it's not the assumption it wants, so you either make it not usable for someone else, or either have to propose options.

Another question raised: report viewing in general. Should tasks from multiple databases be merged in a single view? If thats the case, some sorting might not make any sense. If you sort by urgency, your work task with a very high urgency will be at the top of your "next" report instead of the thing you should be doing at home right now. Should it be split per database then?

The main goal of my posts are to illustrate how your propositions raises a bit too much questions __yet__. I ask some of these questions to have your ideas on the subject, but keep in mind: everyone uses taskwarrior in his own workflow. If using multiple database is restricted to a single workflow, it kinda loses its charm.

this would add a lot of value to task warrior, and add a cleaner way to make it interact with other task management system ( trello, github issue ) which are pretty popular.

There is the excellent bugwarrior ( https://github.com/ralphbean/bugwarrior ) tool that allows to import tasks from github, trello and many more already. This allows to to pull data from multiple "task sources" directly into a single database, without the need for a split database, which is something i do at work. (i pull from redmine and gitlab issues). Sadly it only pulls data, but doesn't update. But it should not be hard to do some hooks to manage some type of updates.

[sucrecacao]

Another example of questions it can raise is when you add a new task. If you're in the "uni" directory, and add a task, in which database should this task go? If you prompt the user, you make the UX less fluid. If you make an assumption, whatever it is, someone will complain that it's not the assumption it wants, so you either make it not usable for someone else, or either have to propose options.

That's a very good point. Definitly, the user would be asked to specify which database it should go to. But I don't think it would make UX less fluid, because it equivalent to typing project:algorithm for e.g.
Then either we forbit to add task without specifying database, either we add to the database of the current directory ( ~/uni/.task ). We create the new db if needed.

Should tasks from multiple databases be merged in a single view?
yes

If thats the case, some sorting might not make any sense. If you sort by urgency, your work task with a very high urgency will be at the top of your "next" report instead of the thing you should be doing at home right now. Should it be split per database then?

I don't understand what you mean exactly. But I think urgency should be calculated every time from all the tasks it receive as input, regardless if it come from one or multiple database. Since TW doesn't store urgency in the *.data file, I might be already working that way.

I think report should not cause problem. Imagine appending all the pending.data from various DB into one, and then apply report on this file.

everyone uses taskwarrior in his own workflow. If using multiple database is restricted to a single workflow, it kinda loses its charm.

Totally agree. This feature should not overlap on any current workflow/feature, but just add more freedom on how we use TW.

This allows to to pull data from multiple "task sources" directly into a single database, without the need for a split database, which is something i do at work. (i pull from redmine and gitlab issues). Sadly it only pulls data, but doesn't update. But it should not be hard to do some hooks to manage some type of updates.

Yes that exactly the kind of problem my feature would solve. Currently if you want to create a script to 'update' your issue, you need first to sort which one are from redmine, which one are from gitlab, and which one are neither. And if your script go wrong and mess with the wrong data ? It doesn't feel good to me.
Imaging having those pulled issue in a separate databases ~/work/.task_gitlab and ~/work/.task_redmine . Don't you think it would be more elegant to have a script dealing directly with those files ?
You could still have the same workflow as you have today by using a simple alias like so:
alias task="TASKDATA=['~/.task', '~/work/.task_redmine', '~/work/.task_gitlab'] task"

[Arwalk]

I don't understand what you mean exactly. But I think urgency should be calculated every time from all the tasks it receive as input, regardless if it come from one or multiple database. Since TW doesn't store urgency in the *.data file, I might be already working that way

I just mean that your work database is mostly useful in the context of your work, and your personal stuff database is mostly useful at home. Take the exemple in reverse: you're at work, you do "task next", and the first entry is the remainder to do the dishes that you put yesterday night at home. Is it really what you want to see first?

Once again, these are just examples.

Yes that exactly the kind of problem my feature would solve. Currently if you want to create a script to 'update' your issue, you need first to sort which one are from redmine, which one are from gitlab, and which one are neither.

bugwarriors adds UDAs for each type of task sources, so sorting them is trivial. If needed, bugwarrrior's code is extremely hackable, so adding a UDA to sort them more properly would be really easy. But it's not very important, bugwarrior is not the key of what we're talking about here.

You could still have the same workflow as you have today by using a simple alias like so:
alias task="TASKDATA=['~/.task', '~/work/.task_redmine', '~/work/.task_gitlab'] task"

Well, as i said, people have many different workflows.

My workflow at home is

• using a UDA to filter what appears in the "next" report that is made mostly of remainders
• having custom reports tailored to specific usages such as personnal projects, stuff i want to buy and series/anime episode tracker to not forget what was the last episode i saw.

But at work my workflow is a GTD-like workflow.

How can i manage to have both these workflows at the same place with your proposition? That's another subject to add: people might be splitting the databases to manage different workflows.

Sure, what you propose is an addition, and it would not stop me to do theses multiple workflows. But considering that you already manage your own workflow with the relatively minor inconvenience of not having aggregated reports, don't you think it is a bit overkill to ask for this whole new feature?

Also, if it ends up requiring an alias, i'm not sure how i see the argument here, considering that i already proposed to do aliases in the first place. I can return to you the argument: you could do a few aliases like:

alias task_redmine="TASKDATA='~/work/.task_redmine' task"
alias task_gitlab="TASKDATA='~/work/.task_gitlab' task"

and a small bash script to read reports from multiple databases at the same time

TASKDATA='~/work/.task_redmine' task "$@"
TASKDATA='~/work/.task_gitlab' task "$@"

To be fair, this does not merge the tasks in a single report. Once again: you want it, someone else would like it another way. But you could just use the API to build your custom multiple-database-report with merged tasks too.

[sucrecacao]

Take the exemple in reverse: you're at work, you do "task next", and the first entry is the remainder to do the dishes that you put yesterday night at home. Is it really what you want to see first?

I think you didn't understood what I meant. This would not happen. task next would promp the next task from your work database because it is splited.

How can i manage to have both these workflows at the same place with your proposition?

I don't understand why you could not

don't you think it is a bit overkill to ask for this whole new feature?

No. I do believe it's an elegant feature that enhance the current TW philosophy. I was actually surprised TW didn't worked that way.

[Arwalk]

I think you didn't understood what I meant. This would not happen. task next would promp the next task from your work database because it is splited.

I clearly do not see what you have in mind then. You're saying on one hand that report results should be merged, but here you say that it would be splitted.

[sucrecacao]

They would be merged on your personal laptop, but on your work laptop you would only have your work related taskwarrior database

[ahillio]

How's it going @sucrecacao ? Have you seen Taskwiki? It apparently can use multiple task databases https://github.com/tbabej/taskwiki/blob/master/doc/taskwiki.txt#L443.

It'd still be good to have another way (without taskwiki). I wonder if the solution should be an external/separate tool (the script simply merges multiple task databases into a single one (and probably keeps track of original .task file and original UUIDs in a custom UDA so that an on-modify hook script can then keep the original .task files updated)) or if this is something that Taskwarrior itself could accommodate(?).

[sucrecacao]

Nice one !
I think it would be awesome if this feature lived inside TW, but maybe the solution for the moment is an external script. I'll definitely look how taskwiki has implemented it.

[xerus2000]

I haven't read through the entire discussion, but here's my take on that:

For focusing on one project at a time, there's an easy fix - define an alias to task that uses the current working directory, something like:

alias t="TASKDATA=$PWD task"

The sharing and having project files in the same directory as the task database are indeed good points I did not really think about yet.