Orgzly-android: git synchroniztion

[IvanMalison]

The possibility of new synchroniztion methods was mentioned way back in 2015 https://plus.google.com/+Orgzly/posts/S2nBbxVibsB

Did anything become of this? Are there plans to add git synchronization support to orgzly?

[angrybacon]

Google Drive synchronization is wanted too.

[nevenz]

You can open a separate issue for Google Drive sync, thanks!

[ghost]

@IvanMalison

When you say git synchronization, were you just talking about committing to a local repo, or also potentially pulling/pushing as part of the sync from that repo to a remote?

I'm just wondering what the full scope of this feature would be.

I like the idea of having the "git client" handle the conflicts during a replay as part of pull-before-push, perhaps adding a new commit with the conflict left in it verbatim, so that I can resolve it later. Like:

 <<<<<<< HEAD:mergetest
 This is my third line
 =======
 This is a fourth line I am adding
 >>>>>>> 4e2b407f501b68f8588aa645acafffa0224b9b78:mergetest

I think this would be a huge improvement from the limited options of force-load/force-save that Orgzly currently provides.

[jappeace]

Shouldn't this be done by a separate program? Or for example make a library from the core features of: https://github.com/maks/MGit and then use those in this project to provide nice integration.

[mkaito]

I use Resilio for phone-to-computer sync. Orgzly-to-sdcard is the kind of auto sync I want. Git might be a good backend for conflict resolution, but staying in sync automatically will outright prevent many conflicts from ever happening.

El 3 de mayo de 2017 7:15:32 GMT+01:00, Jappie Klooster [e-mail] escribió:

Shouldn't this be done by a separate program? Or for example make a
library from the core features of: https://github.com/maks/MGit and
then use those in this project to provide nice integration.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[angrybacon]

@mkaito That would prevent the user to edit a file that is not up to date ie. no offline mode.

The thing is that different users want different ways of synchronizing Orgzly with their Emacs. Git is one backend of interest but I think it's more of something like Tasker's job to update the repository. Automatic synchronizing like you suggest makes a lot of sense with Google Drive on the other hand (like Google Drive documents), but then again you would need to handle the cases where the user edit a document while offline.

[mkaito]

I think the constant need to manually synchronize are a bigger concern than the occasional lack of connection.

I'm not saying there won't ever be conflicts, just that most happen when you forget to hit the sync button.

El 3 de mayo de 2017 16:04:03 GMT+01:00, Mathieu Marques [e-mail] escribió:

@mkaito That would prevent the user to edit a file that is not up to
date ie. no offline mode.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[angrybacon]

@mkaito Well then pick another sync backend than Git :-)

[mkaito]

I just said that using git would make merges a lot more painless when conflicts do happen.

El 3 de mayo de 2017 16:33:56 GMT+01:00, Mathieu Marques [e-mail] escribió:

@mkaito Well then pick another sync backend than Git :-)

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[jappeace]

So I set this up with MGit and then use the file system to synchronize, this works ok-ish although the workflow is a little cumbersome because you have to switch apps to commit and push and pull.
What currently happens is I press synchronize and orgzly writes out to the MGit repo (I assume that before that the changes exist inside the app assigned memory). Then I do git stuff with MGit.

So what is expected of this issue precisely?

• Do you want to auto detect we have a git repo and when you press synchronize automatically commit push and pull just like the github app on windows?
• Do you want to make a MGit like UI inside orgzly?
• Something else?

Personally I'd prefer an auto synchronize thing, and let the advanced stuff be done with an app like MGit.

You thoughts please.

[mkaito]

This issue is about automating what the "synchronize" button does. Nothing else. Git came up as a backend system to use for conflict resolution, not as an additional app that you manually use.

El 9 de mayo de 2017 16:30:20 GMT+01:00, Jappie Klooster [e-mail] escribió:

So I set this up with MGit and then use the file system to synchronize,
this works ok-ish although the workflow is a little cumbersome because
you have to switch apps to commit and push and pull.
What currently happens is I press synchronize and orgzly writes out to
the MGit repo (I assume that before that the changes exist inside the
app assigned memory). Then I do git stuff with MGit.

So what is expected of this issue precisely?

• Do you want to auto detect we have a git repo and when you press
  synchronize automatically commit push and pull just like the github app
  on windows?
• Do you want to make a MGit like UI inside orgzly?
• Something else?

Personally I'd prefer an auto synchronize thing, and let the advanced
stuff be done with an app like MGit.

You thoughts please.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[jappeace]

So when the user presses the synchronize button, git will commit local changes, pull in remote changes and push local changes. If everything goes according to plan, otherwise panic and let the user figure it out?

[mkaito]

We could figure out what specifics work best. Perhaps stash, pull, stash pop, commit and push? Or commit, fetch, rebase?

El 9 de mayo de 2017 17:20:42 GMT+01:00, Jappie Klooster [e-mail] escribió:

So when the user presses the synchronize button, git will commit local
changes, pull in remote changes and push local changes. If everything
goes according to plan, otherwise panic and let the user figure it out?

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[jappeace]

I wouldn't want to do the stash approach, because I think it will prevent conflict detection, and I do believe on a conflict the user should be consulted, rather than choosing for the user.
I honestly don't know what happens if a conflict occurs during a stash apply, does it just overwrite?

Then I think automatic merges are better than rebases, because a merge is non-destructive. At least according to the tutorial I just read on rebases.
We can also not know if upstream is a public branch or not, therefore committing + merging is the correct solution I believe.

[IvanMalison]

Then I think automatic merges are better than rebases

yes, definitely. automatic rebases are a bad idea

[alphapapa]

Then I think automatic merges are better than rebases, because a merge is non-destructive. At least according to the tutorial I just read on rebases.

Merges are non-destructive in the sense that they don't alter commit history, but if a merge doesn't apply cleanly, the merge commit is potentially destructive if the user doesn't resolve the conflicts correctly.

A rebase is destructive in the sense that it alters the commit history of the branch that's rebased, but a rebase that applies cleanly is non-destructive to the data within each commit.

[IvanMalison]

@alphapapa

Merges are non-destructive in the sense that they don't alter commit history, but if a merge doesn't apply cleanly, the merge commit is potentially destructive if the user doesn't resolve the conflicts correctly.

Nope, its never destructive, because EVERYTHING that was there before is still there (i.e. the commits exist in their original context). If the user doesn't resolve conflicts correctly, its easy to go back to the states before the commit and perform the merge again.

A rebase is destructive in the sense that it alters the commit history of the branch that's rebased, but a rebase that applies cleanly is non-destructive to the data within each commit.

Hmmm. define cleanly. If you take the above scenario I mentioned (i.e. a conflict resolution takes place, but it ends up being incorrect, it is actually impossible to recover the original commit in its original context). The conflict resolution isn't even recorded as conflict resolution (as it is with merges), but as part of the existing commits.

If by cleanly you mean "without dropping into conflict resolution", even that is not really true. Git IS NOT a patch oriented VC, which means the "contents" of a commit is actually the entire state of the repository at that time, not something like a patch file that you might apply to a repository. Git can sort of make it seem liek this is what it stores when you look at diffs, but this is not the case.

[alphapapa]

Nope, its never destructive, because EVERYTHING that was there before is still there (i.e. the commits exist in their original context). If the user doesn't resolve conflicts correctly, its easy to go back to the states before the commit and perform the merge again.

Yes, in the context of the entire commit history, it's non-destructive. But in the context of the merge commit itself, it is destructive. And especially in the context of Org files, it's easy to delete data and never notice it's missing.

Resolving conflicts on a smartphone screen is likely to be error-prone. Especially consider cases where the user may be in a low-visibility or unstable environment, like in a moving vehicle, in direct sunlight, etc: it's not difficult to imagine a scenario where a user accidentally taps the wrong hunk, resolves the conflict, and never notices that the wrong one was selected. In such a case, having the commit history isn't going to save him--at least, it's not guaranteed to, because first he has to notice the data is missing.

This has actually happened to me a few times: data seemed to be missing from an Org file, and I was thankfully able to dig it out of a git commit from months earlier. But who knows if that's happened without my noticing it. Even so, finding the needle commit in a haystack of thousands of commits over months or years is not an easy task, even with gitk.

The bottom line for me is that doing git merges on a smartphone or tablet is probably a bad idea in general; and it's an especially bad idea for anyone not already familiar with git merges. If someone wants to implement such a feature for "advanced users," cool, but I think it would be a mistake to promote it to "normal" users.

Hmmm. define cleanly.

A clean rebase is one that git processes automatically, without conflicts. Or, in git terms, a fast-forward.

If by cleanly you mean "without dropping into conflict resolution", even that is not really true. Git IS NOT a patch oriented VC, which means the "contents" of a commit is actually the entire state of the repository at that time, not something like a patch file that you might apply to a repository. Git can sort of make it seem liek this is what it stores when you look at diffs, but this is not the case.

Git is not darcs in that it fundamentally does not operate on patch theory, but it certainly is patch-oriented for some operations. Git was designed to work with patches from the beginning, as it was designed for the kernel and the LKML. It simply generates patches dynamically when needed, or uses them when they're given.

[jappeace]

But in the context of the merge commit itself, it is destructive

So the way I interpreted non-destructive is 'loss of information'. Sure you can fuckup the merge commit badly, but you didn't loose any information (can always go back with a checkout and try again).
From the article I linked:

Merging is nice because it’s a non-destructive operation. The existing branches are not changed in any way.

Resolving conflicts on a smartphone screen is likely to be error-prone.

My idea was to just not even support this. If a conflict occurs, we panick and roll back. We only accept fast forwards and automerges.

The bottom line for me is that doing git merges on a smartphone or tablet is probably a bad idea

I guess you mean the manual stuff, with which I agree, you shouldn't do that on the phone.

Alternatively on a panick you could offer to push local changes to the remote on a branch. This allows user to merge manually on a better suited device.

[IvanMalison]

Git is not darcs in that it fundamentally does not operate on patch theory, but it certainly is patch-oriented for some operations. Git was designed to work with patches from the beginning, as it was designed for the kernel and the LKML. It simply generates patches dynamically when needed, or uses them when they're given.

Sure, that doesn't change my point, which is that when it comes down to it, EVEN cleanly applied rebases ARE destructive, and because of the way org works, I can imagine scenarios where a cleanly applied rebase could have undesireable results (think subheadings etc.), which would make it important to have the exact original context for a change.

The bottom line for me is that doing git merges on a smartphone or tablet is probably a bad idea in general; and it's an especially bad idea for anyone not already familiar with git merges. If someone wants to implement such a feature for "advanced users," cool, but I think it would be a mistake to promote it to "normal" users.

I don't think anyone was advocating this sort of thing. @jappeace and I were simply suggesting that going with merges here would be better than going with rebases.

[alphapapa]

We find ourselves in violent agreement. :)

[aspiers]

Great discussion all! I have one minor correction, and a suggestion. Firstly the correction:

A clean rebase is one that git processes automatically, without conflicts. Or, in git terms, a fast-forward.

Sorry, but that's not quite right. In the git world, fast-forwarding implies that new commits are not being created, but it is possible, and in fact more common, that rebases will happen cleanly without conflicts and yet still create new commits. Actually it doesn't really make much sense to talk about fast-forwards in the context of rebasing; it's usually more applicable to merges and cherry-picks.

Secondly, the suggestion:

I agree that automatic syncing would drastically reduce the frequency of conflicts. But automatic syncing is not mutually incompatible with using git - you can have your cake and eat it! Have any of you guys looked at the git-annex assistant? That's exactly what it does, and it even has an Android version ...

[IvanMalison]

the git-annex assistant?

I don't think git annex is a good solution since it doesnt keep the actual content of the files in the git repository. Since we ARE dealing with text files and not files that are too large to handle a simple automatically sychronized repository shoudl be good enough (basically https://github.com/hbons/SparkleShare)

[IvanMalison]

I'm going to take a swing at this.

One issue that I forsee from looking at the existing sync code is that its not going to be super easy to batch up changes to multiple files in a single commit because there is no "finalize synchronization action". I'm assunming this is probably fine.

[nevenz]

I'm going to take a swing at this.

Nice!

One issue that I forsee from looking at the existing sync code is that its not going to be super easy to batch up changes to multiple files in a single commit because there is no "finalize synchronization action".

Yeah, though doing a single push would be nice.

[IvanMalison]

@nevenz I'm thinking about maybe making a change to the repo api to allow synchronization to be done more safely with git. I want to ensure that synchronization is only ever performed on top of the appropriate revision of a previous version of the file, so I would like the storeBook method to also take as an argument the previous revision. Does this sound resonable?

[IvanMalison]

@nevenz Another issue I'm having is that I would like to allow the user to specify a location on disk in which to store the git repo IN ADDITION to the actual uri that will be used to indentify the repo (which will also be used as the uri that is stored on the book). This doesn't fit in well with the existing system of having exactly 1 uri for each book.

[nevenz]

I'm thinking about maybe making a change to the repo api to allow synchronization to be done more safely with git. I want to ensure that synchronization is only ever performed on top of the appropriate revision of a previous version of the file, so I would like the storeBook method to also take as an argument the previous revision. Does this sound resonable?

So what the plan in the end for conflict resolution? This suggest you'd always call storeBook, even on (what current repos types would call) a conflict?

Another issue I'm having is that I would like to allow the user to specify a location on disk in which to store the git repo IN ADDITION to the actual uri that will be used to indentify the repo

Why, does he care about that directory? I don't think he should, that sounds like "implementation detail" to me. It can be just a checkout in app's cache directory.

Depends on implementation I guess - if we can start with listing all the steps in different scenarios, that would be great.

[tulth]

I think a new repo class for git would be neat to add someday. It's unclear to me where the responsibility for merging master would live

[IvanMalison]

So what the plan in the end for conflict resolution? This suggest you'd always call storeBook, even on (what current repos types would call) a conflict?

@nevenz

No. Here is the rough idea:

Retrieving/syncing from the repository:

• Every time we update/sync a "book" that is connected to a git repository, we store 2 things:
  
  
  
  • the SHA1 of the blob of the associated file
  
  
  • the SHA1 of the commit at HEAD
  
  
• As we're doing the update, we ensure that the previous commit SHA1 is in the repository history that we are syncing with. If it is not, we will display a dialog warning the user about data loss, but allowing them to perform a "force upgrade" if they want to.

Writing data from orgzly to the repository:

• Check if the SHA1 of the book's file blob is the same as it was when we last synced, and that there are no changes in the working tree/index. NOTE: We DO NOT do try to fetch any new changes first.
• If these conditions are met (they always will be provided that no other program messes with the repository) we can simply apply the update that we want to apply and make a new commit. Note that this is much more permissive than using the SHA1 of the commit since it will allow changes to other files or meta data in the repository.
• If the SHA1 of the book's file blob has changed. We make a new temporary branch at the commit at the time of last sync and apply our changes.
• Then we attempt a merge with what was in the repository.
• If there are conflicts in the merge, the user must manually resolve the conflicts. We will give them two options for how to proceed:
  
  
  
  • a) Abort the merge, but keep the commit that was made with the new orgzly changes, and push it to a new branch of the users choosing on the remote. This will allow the user to resolve conflicts on a computer and then push them back to a branch as appropriate. We will need some type of branch selection interface to allow the user to return to their master branch at some point, as with this option orgzly will now be syncing with a different branch.
  
  
  • b) Leave the branch in a conflicted state. It will be up to the user to use some program on their device (MGit) etc. to resolve the conflicts and do all the other git things that need to be done to restore orgzly's ability to sync.
  
  

• Only now will we fetch from the remote repository (assuming that we are now in the happy case of synchornization) we attempt another merge, using the same procedure as above if there are merge conflicts.
• Finally we push to the remote.

This whole procedure is designed to do three things:

a) Only ever safely apply updates from orgzly and vice versa, making data loss impossible
b) reduce, as much as possible, the possibility of merge conflicts
c) Make it as easy as possible to resolve merge conflicts when they do arise

Why, does he care about that directory? I don't think he should, that sounds like "implementation detail" to me. It can be just a checkout in app's cache directory.

Because, some people will want to perform git operations (like conflict resolution) using applications other than orgzly.

[tulth]

I am thinking along the same lines and I am working a (non git) hash based file sync check. See pull request #172

[IvanMalison]

I think a new repo class for git would be neat to add someday. It's unclear to me where the responsibility for merging master would live

@tulth @nevenz Do you have any idea/preferences about how I should go about storing both the SHA1 of the blob and the SHA1 of the commit, I can do some things like come up with some nasty way of encoding that in the revision string, but Ideally I'd like to do something else.

I feel like the repo/rook/versioned rook classes may be slightly overfitted to the existing repo implementations (Dropbox, Directory). Perhaps the best approach would be to not try to fit the details of all the different sync implementations with these classes? I'd love to hear your thoughts on this.

I am thinking along the same lines and I am working a (non git) hash based file sync check. See pull request #172

How do you plan on handling conflict resolution without git? There's not really a ton you can do there right?

[tulth]

How do you plan on handling conflict resolution without git? There's not really a ton you can do there right?

It would be conflict detection and reporting... not resolution.
It detects conflicts by:
test1) book newer than the last repo sync
test2) current calculated hash mismatched the hash stored in db

if !test1 && !test2: nothing to do, already in sync
if !test1 && test2: repo newer, need to read repo and write to db
if test1 && !test2: book newer, need to write book to repo
if test1 && test2: conflict

edit: fixed logic typos

[IvanMalison]

The thing is that different users want different ways of synchronizing Orgzly with their Emacs. Git is one backend of interest but I think it's more of something like Tasker's job to update the repository. Automatic synchronizing like you suggest makes a lot of sense with Google Drive on the other hand (like Google Drive documents), but then again you would need to handle the cases where the user edit a document while offline.

@angrybacon

I strongly disagree here. Having to
a) buy tasker
b) Set up a sync job that involves BOTH orgzly and tasker and mgit
c) Configure the directories and repository, ssh keys appropriately across both

is always going to be more cumbersome and flaky than having in application synchronization that can sync at critical moments (i.e. after a file is touched, when orgzly is opened), display sync status in application (current SHA last time of update)

Personally I'd prefer an auto synchronize thing, and let the advanced stuff be done with an app like MGit.

@jappeace
I really don't understand why you'd want this. By having the git In app, you avoid having to separately go in to mgit to push/pull etc., and you avoid all the cognitive load of the different levels of syncing (copy to working tree, commit, push/pull etc.). Theres just so many place where things can go wrong if you dont have it controlled in one place. Furthermore, by having the git synchorniztion in app, you can do things like record the sha of last synchronization, and make sure that you are only applying changes to the exact file state that you last synchronized with. Without this guarantee, there are a bunch of ways to accidentally clobber/lose information.

[IvanMalison]

@tulth Where exactly are you planning to implement these checks in the code base.

The current Repo methods do not receive the VersionedRook where it is needed. I actually have a WIP branch where I have added a VersionedRook paramter to storeBook, renameBook and retrieveBook. Perhaps I should make a separate pull request with just this change so that we can avoid making conflicting changes to the Repo interface?

EDIT: Hmm, it seems that you are using the CurrentRooksClient

[nevenz]

It would be good to start from existing Repo interface and see what we need for git. We can improve it or change it as much as we want. I'd just like to avoid a rogue implementation.

Currently, the logic for what needs to be synced is the same for all repo types. It's in updateStatus. It was done so storeBook and retrieveBook can be dumb and just do exactly that.

But if we need a repo-specific decision-making, we could perhaps create a new interface for that. A standard implementation could be created, which repos would return by default, but they'd also be free to create their own. Something along those lines.

I'd suggest trying to implement Repo for start and see how far we can go with that. It'll be easy to move the code around later.

I'm not sure I understand the need for all the things @IvanMalison mentioned when describing the steps (thanks for that BTW), though that could be just by Git ignorance. I have to go through all the comments in this issue one more time.

Personally I'd probably go for:

• Pull master
• Commit changed notebooks to local orgzly branch
• Merge orgzly to master (complain if it fails)
• Import all changed notebooks
• Push master

Using blob hash a revision and trying to implement Repo. That would be a great start.

There'd be issues popping up all over the place probably, but it would be easier to see what we need exactly and how to keep it as simple as possible.

BTW, as for the additional configuration needed for Git (directory, credentials, etc.) - it's not really Git-specific, many repo types will have to have more things then just the URL. This data can be stored in DB or properties, for each repo.

One thing to have in mind (after seeing properties in #173) - there could be multiple git repositories defined by user. One private, one public, one for worg for example. ;) So this needs to be handled too.

I'd probably use DB, even though there's a lot of boilerplate code when doing so. But some tables could be shared between different repository types. For example table with user/pass, or for ssh keys etc.

[IvanMalison]

@nevenz

It would be good to start from existing Repo interface and see what we need for git. We can improve it or change it as much as we want. I'd just like to avoid a rogue implementation.

Yeah, that is what I'm in the process of doing. I started by writing the git synchronizer which will be used by this repo class.

But if we need a repo-specific decision-making, we could perhaps create a new interface for that. A standard implementation could be created, which repos would return by default, but they'd also be free to create their own. Something along those lines.

The main things we need for git are:

a) a way to store directory in addition to git uri
b) a way to store revision hash in addition to blob hash
c) the ability to sync back changes after uploading (since a merge could have occurred)
d) A solution for pushes that avoids pushing after every single file change (although this could be delayed for now)

Personally I'd probably go for:

Pull master
Commit changed notebooks to local orgzly branch
Merge orgzly to master (complain if it fails)
Import all changed notebooks
Push master
Using blob hash a revision and trying to implement Repo. That would be a great start.

This is basically what is being done, but we are also storing blob hashes to avoid merges in more cases.

Also, this sync scheme is MUCH more different than the existing one - notice how many things happen at the start of synchronization and then at the end of synchronization.

Repo simply would not work for the procedure you are describing here (how do you return the versioned rook for the first file you've imported when you havent made a commit yet for ex.)

One thing to have in mind (after seeing properties in #173) - there could be multiple git repositories defined by user. One private, one public, one for worg for example. ;) So this needs to be handled too.

The properties I added were for git author, and ssh key, not for directory location. Do you think people will want to add multiple ssh keys and have different git authors per repo. Perhaps that is something we can do eventually, but I don't think it's something we need for v1.

[nevenz]

a way to store revision hash

This is the part I didn't get - what do we need this again? And can we do without it for now?

Also, this sync scheme is MUCH more different than the existing one - notice how many things happen at the start of synchronization and then at the end of synchronization.

We can start with having onStart and onEnd kind of methods in Repo.

Repo simply would not work for the procedure you are describing here (how do you return the versioned rook for the first file you've imported when you havent made a commit yet for ex.)

But revision is available when you add file to index, which should be enough to create VersionRook. Unless I'm misunderstanding something.

Anyway, I do realize there will be need for some major changes, I'm just kinda trying to push this to its limit, then go back from there.

The properties I added were for git author, and ssh key, not for directory location. Do you think people will want to add multiple ssh keys and have different git authors per repo. Perhaps that is something we can do eventually, but I don't think it's something we need for v1.

Yeah, we'll definitely need everything stored separately per repo.

I'm fine with one step at the time of course, but the problem can be not breaking user's setup later. For example if we start by storing email as git_email in preferences, what do we do in v2 when we support multiple repos. It's why it might be necessary to leave some room for that right from the start. Which might end up being very close to completely supporting it.

We can have a SharedPreferences per repo, something like repo.ID.xml?

[tulth]

Right now I have the repo grab a global app context to get the content provider.
I feel like this is ok because the sqldb and files are both global resources on the filesystem... no difference really.

[IvanMalison]

This is the part I didn't get - what do we need this again?

Lets say that the state of the repo has been changed such that the sha that we have stored for some book is not the sha of that file in the git repository. If we don't store the original revision that we got the file state from, we basically have no way to continue, because we need to apply our update on top of a comitt which has the appropriate state for the file.

You might argue that "The git repo should never get in to this state.", but I would like this repository sync to be as robust as possible.

Another reason to keep the repo string around is that when we are syncing back from the repository, we want to make sure that the last synced revision is a parent of the commit that we are syncing.

We could get by with ONLY the repository sha (i.e. without the file contents sha), as we could always just use this technique of restoring to an old state, but this would require almost every commit to be a merge commit (and would cause more issues with rebasing).

I'm fine with one step at the time of course, but the problem can be not breaking user's setup later. For example if we start by storing email as git_email in preferences, what do we do in v2 when we support multiple repos. It's why it might be necessary to leave some room for that right from the start. Which might end up being very close to completely supporting it.

We can have a SharedPreferences per repo, something like repo.ID.xml?

I'm not super familiar with android programming -- how do you create such templated xml files?

Would we have to make a new view to configure all of this for each git synced repository? Where should this live in the settings menu?

One of the sort of thorny issues is that these settings are not actually per book, but per repository, so they could be associated with multiple books.

[IvanMalison]

@nevenz Another issue we need to deal with is the fact that there is an assumption of uniqueness of uris in the CurrentRooksClient. This will not apply to git repositories as my plan is to store relative (to the root of the git repository) uris in the rook.

Would you prefer that absolute paths were stored here? That just ends up not being very tidy with the existing interfaces.

[IvanMalison]

@nevenz As far as storing the file/commit sha, I guess that perhaps the best thing to do is to store the commit sha, because the file sha can always be retrieved from that...

[IvanMalison]

@nevenz Can you explain why there is inconsistency in the interface for retrieveBook and storeBook?

Why does storeBook take a fileName, but retrieveBook takes a uri?

[IvanMalison]

@nevenz Don't we theoretically need to have rename work both ways? What happens when there is an upstream rename?

Ideally, I think there should actually be no rename at all...

[nevenz]

I'm not super familiar with android programming -- how do you create
such templated xml files?

Check getStateSharedPreferences. It creates states.xml. dropboxToken
is using it for example.

Would we have to make a new view to configure all of this for each git
synced repository?

Yes, like DirectoryRepoFragment for example.

One of the sort of thorny issues is that these settings are not
actually per book, but per repository, so they could be associated
with multiple books.

Right, I was suggesting repo..xml. And we'd have
gitAuthor(repoId, name).

It would be a generic storage for repo implementations which they could
use.

@nevenz Another issue we need to deal with is the fact that there is
an assumption of uniqueness of uris in the CurrentRooksClient. This
will not apply to git repositories as my plan is to store relative (to
the root of the git repository) uris in the rook.

I was planning to remove current_versioned_rooks. If you check where
CurrentRooksClient is being used - it's in tests only. book_syncs is
used though. Is that OK for this implementation?

Would you prefer that absolute paths were stored here? That just ends
up not being very tidy with the existing interfaces.

uri in Rook (so in turn, VersionedRook) is absolute, so if we want to
keep using the same interface, it will have to remain absolute I think.

@nevenz As far as storing the file/commit sha, I guess that perhaps
the best thing to do is to store the commit sha, because the file sha
can always be retrieved from that...

Well if we're keeping the common check for changes, we'll have to have
file's revision. Also, revision is visible to the user currently, it
can be useful for at least debugging.

@nevenz Can you explain why there is inconsistency in the interface
for retrieveBook and storeBook?

Why does storeBook take a fileName, but retrieveBook takes a uri?

Yeah, it's messy. I remember changing that interface few times for
various reasons.

I guess we should prefer relative names here, since we have repo's uri
already.

@nevenz Don't we theoretically need to have rename work both ways? What
happens when there is an upstream rename?

It's not detected currently. Renamed book would be imported, existing
one in app would be stored, so you'd end up with two. It just hasn't
been implemented yet. Same with deleting of remote book.

[IvanMalison]

Check getStateSharedPreferences. It creates states.xml. dropboxToken
is using it for example.

@nevenz
Yeah but dropboxToken is not per repo afaict, why have ssh keys/authors etc be per repo in that case?

[IvanMalison]

@nevenz We are going to need a way to dispatch repo creation on more than just uri scheme, since git repos can come in a bunch of different flavors.

How would you like to approach this?

[nevenz]

Yeah but dropboxToken is not per repo afaict, why have ssh
keys/authors etc be per repo in that case?

It should haven been - Dropbox doesn't support more then one account
easily, unless you're a Business user, so I didn't bother (I think).
Not to mention is was the first repo I've implemented and was writing
the app just for myself.

Do you think it'll be an issue, or you're just asking?

We are going to need a way to dispatch repo creation on more than just
uri scheme, since git repos can come in a bunch of different flavors.

Hmm yeah. I guess type of the repo will have to be saved together with
uri in repos table. Then (in worst case) queried in getFromUri. (And
ideally got earlier together with uri.)

If you don't want to mess with that, just try to figure it out from the
url, which should work in most cases -- check for .git prefix first,
then do schema checks. And I'll see what can be done to avoid that
guessing.

[IvanMalison]

@jgkamat @Endi1 @ebpa @a13 @pawel-zet @tbuschmann @jappeace @gheoan @arrcher @btimofeev @andrewsuzuki @realloc
I'd love some help testing #173. Its still pretty raw atm, but it should mostly work.

[neiljerram]

I'm super-happy to have found that this discussion and work is happening here! Thanks to @IvanMalison for working on this, and to @nevenz and everyone else for contributing to the discussion.

I'm new to Android development, but will try to make my way round to being able to test #173 ... For now, I just wanted (1) to comment on:

Lets say that the state of the repo has been changed such that the sha that we have stored for some book is not the sha of that file in the git repository. If we don't store the original revision that we got the file state from, we basically have no way to continue, because we need to apply our update on top of a comitt which has the appropriate state for the file.

That sounds equivalent to orgzly having its own Git branch in the git repo on the phone. I'm inclined to doubt that it will be a good thing to reinvent this, or that there is a good reason to distrust the local repo state.

And (2) to provide the script that I'm currently using to do this kind of thing by hand, in case it is useful:

# -*- shell-script -*-

# Ensure that we're on the phone's branch, and get its SHA-1.
git checkout phone
local_phone=`git rev-parse HEAD`

# Commit any new changes on the phone.
git add -A .
git commit -m "Changes made on phone"
echo INFO: Committed phone changes to local Git

# Update tracking branches from the Git server.
git fetch --all

# Determine whether the phone branch on the server has been modified
# since it was last pushed from here.
remote_phone=`git rev-parse origin/phone`
if [ $local_phone != $remote_phone ]; then
    # Phone branch on the server has been modified.  We need to merge
    # that into our own phone branch.  This should always succeed,
    # because the phone branch should only be modified elsewhere (than
    # on the phone) when there has been a conflict, and in that case
    # the user should take care not to make any further changes on the
    # phone until the conflict has been resolved.  The conflict should
    # have been resolved by adding commits, on the phone branch, to
    # the last point that was pushed from this phone, and so this
    # merge now should be a simple fast forward.
    if git merge origin/phone; then
    echo INFO: Merged phone branch conflict resolution from Git server
    else
    echo ERROR: Could not merge phone branch conflict resolution from Git server
    git merge --abort
    exit 1
    fi
fi

# Push the current phone branch to the server.
git push
echo INFO: Synced phone branch to Git server

# Try to merge any changes that there may be in the master branch -
# which is where any other devices make their changes.
if git merge origin/master; then
    git push
    echo INFO: Merged non-phone changes into phone branch

    # That worked, so also merge the phone branch back into master.
    git branch -D master
    git checkout master
    if git merge phone; then
        git push
        echo INFO: Merged phone branch back into master
    else
        echo ERROR: could not merge back into master
        git merge --abort
    fi
    git checkout phone
    git branch -D master
else
    # The phone and master branches are in conflict.  This is a pretty
    # normal scenario when updates are being made on multiple devices.
    # We abort the merge and flag the situation, and the user should
    # resolve it by:
    # - probably switching to a more capable device (laptop), for
    #   convenience
    # - doing the same merge on that device (git checkout phone; git
    #   merge master)
    # - manually resolving the conflicts
    # - committing and pushing the merge with those resolutions
    # - coming back to this phone, and running this script again,
    #   before making any further changes on the phone.
    echo WARNING: Conflict between phone and master branches
    git merge --abort
fi

[neiljerram]

(But - to be clear - I'd far rather that this was properly integrated into Orgzly, and so I can throw away that script!)

[nevenz]

PR is merged, but functionality is disabled until issues are fixed.

[mtekman]

It looks like the last attempt at this ended a week ago. Will there be any official moves towards getting git synchronization working in the near future?

[IvanMalison]

That sounds equivalent to orgzly having its own Git branch in the git repo on the phone. I'm inclined to doubt that it will be a good thing to reinvent this, or that there is a good reason to distrust the local repo state.

@neiljerram

I distrust stateful things that are managed by multiple sources of truth, which would be exactly what simply having a branch in the git repository would be. It would be too easy for someone to pull in changes to the git repository with some other application and have orgzly happily apply its changes in a totally incorrect manner. We're not really reinventing the wheel here at all because we are using gits OWN way of tracking this properly which is with SHAs, not with branches, which are ephemeral and therefore not a good candidate.

@mtekman I haven't made any attempts to finish it recently has someone else?

[nevenz]

I haven't made any attempts to finish it recently has someone else?

I recently tried with latest jgit version, but it tried creating (temporary?) symbolic link while cloning, which was failing due to fs not supporting them. I didn't dig into it more deeply yet.

[neiljerram]

@IvanMalison

We're not really reinventing the wheel here at all because we are using gits OWN way of tracking this properly which is with SHAs, not with branches, which are ephemeral and therefore not a good candidate.

OK; I've recently noticed that stashes do something like this too, i.e. use non-branch SHAs to track certain content. Just one worry: can you be confident that that content won't get GC'd?

[IvanMalison]

@neiljerram git is a content address store, and the hashes we are storing are hashes of the file content, not of the commit.

In any case, so long as the commit is merged into some branch that is either a part of a remote or local branch, it (and all of the content hashes that are descendents of it) will never get gced. The only way that would NOT be the case is if the user is somehow doing destructive history editing, which is something only advanced users should be doing anyway. You should think of orgzly as a sort of push remote, and as soon as anything is published to orgzly you can never do any sort of destructive history editing like cherry-picking or rebasing. All of this sort of goes without saying IMO, and you would really need to know your way around git pretty well to make things break.

[barak]

For synchronization to a git repo, how about this?

• Do merges and pulls and pushes as long as there is no conflict.

• If there is a conflict _back out the merge_, and push to a temporary remote branch, thus allowing the conflict to be resolved remotely.

• Include an option to push local changes to a temporary remote branch, and back changes out locally, thus allowing them to be applied/resolved remotely. This would allow Orgzly to ff along the remote branch if desired, as I suspect it's a pretty common use case to want to pull in some remote change and just stash any local changes if necessary.

The basic idea is that the Orgzly App is for jotting stuff down and viewing the .org files. Major operations like resolving conflicts are outside its competence, and should be performed using more appropriate tools (ahem, emacs, magit, ahem.) And git makes this easy by allowing such changes to be pushed into a new branch and forgotten about locally.

[benmoran]

@barak I'm basically doing this now, syncing orgzly to Git externally with Termux. A small shell script implements logic like you suggest. I can trigger it manually from Termux widget, but I also run it in the background periodically with https://github.com/termux/termux-api/pull/235 .

[barak]

@benmoran Cool. I guess I'm suggesting that this be the default behavior, to make it by default hard to lose work and easy to get all the info needed to merge in one place. Underlying it is, I guess, the assumption that people who want to use git as their external sync location know how to use git externally, but might not be set up to use it on android.

On the other hand, if they're really sophisticated they know what to do remotely before telling orgzly to do a force push. So just supporting force push wouldn't be the end of the world...

[neiljerram]

@barak @benmoran It seems there are a few of us using this approach now. (I posted my script a few comments above.) For me, at least, what would still be interesting would be to make the synchronization more automated than I currently have it - or perhaps I should say, more generally, to agree on and further automate the overall workflow.

Currently I use this manual process to synchronize between the phone and my Git master repo:

1. In Orgzly, sync (all notebooks).
2. In Termux: cd storage/shared/Org2/ and bash osync (that being the name of my script)
3. In Orgzly, sync again (all notebooks).

If (2) reports conflicts, then I add in

• (2a) On laptop, resolve conflicts between master and phone branches, and push resolved master and phone back to master repo.
• (2b) In Termux, bash osync again.

What I'd like next would be something on the phone that did this process automatically, probably periodically, for as long as there were no conflicts to resolve; and if conflicts were detected, to emit a notification to tell me that. I wonder if it's possible to achieve that without requiring any further changes to Orgzly itself?

[benmoran]

What I'd like next would be something on the phone that did this process automatically, probably periodically, for as long as there were no conflicts to resolve; and if conflicts were detected, to emit a notification to tell me that. I wonder if it's possible to achieve that without requiring any further changes to Orgzly itself?

@neiljerram that periodic background running is what my patch to to termux-api is for - you can schedule a script to run roughly every 30 minutes (perhaps restricting to while the phone is charging/on wifi, etc.) I've not had any feedback on that yet so it's not easy to install without rebuilding Termux though.

I think people use Tasker for this sort of thing - I haven't tried. Or you can use crond in Termux, but I believe that might require the phone to be kept awake and drain the battery.

[neiljerram]

@benmoran What about the Orgzly syncs, though? (Before and after the sync between phone storage and master repo.) Is there a way of automating those too? Or are you using an alternative workflow that doesn't require that?

[benmoran]

@neiljerram I have all Orgzly's "Settings -> Auto-sync (experimental, local repositories only)" enabled so it syncs to the local file when I create or edit a note, or when I foreground the app. So far I think it's doing the right thing and I haven't needed to tell Orgzly to sync manually.

[neiljerram]

@benmoran Ah, awesome, thank you!

[IvanMalison]

The basic idea is that the Orgzly App is for jotting stuff down and viewing the .org files. Major operations like resolving conflicts are outside its competence, and should be performed using more appropriate tools (ahem, emacs, magit, ahem.) And git makes this easy by allowing such changes to be pushed into a new branch and forgotten about locally.

@barak
The approach you are describing IS, esssentially, the approach of #173. Where did you get the idea that it is any different.

[barak]

@IvanMalison Mea culpa, I'll give that a shot.

[amberin]

Anyone who is interested in this should take a look at #491. Any sort of feedback is welcome.

[lfxgroove]

I'm taking a stab at this, will let you know once i have a PR to review.

[schlagenhauf]

As far as i understand the implementation was merged back into master. What do i need to do to get these changes? The current release does not seem to include these changes already.

(Big thanks for the PR btw!)

[IvanMalison]

@schlagenhauf They are behind a flag right that hides any ui that enables them.

There are a list of remaining items that need to be fixed in https://github.com/orgzly/orgzly-android/pull/543#issue-278409449

I'm not sure if @lfxgroove was planning to continue working on them or not. I've sort of had the intention of trying to do them, but I don't really do android development regularly, and leaving emacs for android studio is painful for me (plus the work is mostly bitch work), so I haven't really found time to do it.

[caleb-allen]

I mentioned in #173 that I would like to build this out. I'm an android dev and would need this feature to find value from orgzly.

I'll poke around and see what the implementation looks like as it stands.

[kmicklas]

They are behind a flag right that hides any ui that enables them.

@IvanMalison Where is this flag and how does one enable it?

[IvanMalison]

@kmicklas It's a compilation flag

It's set here, but obviously you will need to build yourself to enable it:

https://github.com/orgzly/orgzly-android/blob/ee0f50c940e111bbc19c2fbcd18959c78ca318eb/app/build.gradle#L31

[devurandom]

I am not sure what the current state is, but I would already be happy, if I could press the "sync" button in Orgzly, it would sync the data to the file system, MGit would open with the changes staged and a pre-filled commit message was waiting for me to press "OK".

[lfxgroove]

Hi, sorry for leaving this in a half-finished state, life got in the way a bit :) @caleb-allen have you resumed work on this or should i resume where i left off?

[caleb-allen]

@lfxgroove I worked through it for a few hours, but ended up with not much to show. The way the interfaces are set up don't seem to work well with git, at least the way I understand it. Feel free to continue work on it

[IvanMalison]

@caleb-allen The git integration work is done though. I think all that remains is UI cleanup and stuff.

[nevenz]

The git integration work is done though. I think all that remains is UI cleanup and stuff.

There is https://github.com/orgzly/orgzly-android/pull/543#issuecomment-508098676 left, I think, from my last try. I can finish any UI stuff, no one needs to spend time on that.

[lfxgroove]

I'll take a stab at the force pull/push mentioned over in #543.

[nevenz]

I'll take a stab at the force pull/push mentioned over in #543.

Thanks! Let me know if you need any help. I haven't updated the implementation for the new RepoWithProps for example.

[lfxgroove]

I don't think I'll have the time and motivation to finish this up unfortunately, if anyone wants to resume the work on the latest PR it's available in #643.