Gadgetbridge: Pebble: support sending timeline pins

[ashimokawa]

[roidelapluie]

Would be nice to have calendaring support.

[ashimokawa]

Yes, that would be the fist use case of timeline pins.

[roidelapluie]

You are awesome :)

[wngr]

Is there any news on this issue?

[danielegobbetti]

@wngr no news, if there were you would see mentions on this page. The progress we made was that we are now (probably) aware of the complexity of this feature, one example: timeline pins do not "expire" by themselves, even after they disappear from the "past" timeline on the watch. This means we have to add a local cache of sent pins (in gadgetbridge, on the android device) in order to delete them when they are expired (or if the corresponding original information disappears).

Unfortunately sending timeline pins to the pebble is only the last step in a fairly long development that will happen under the hood.

To this end, any help is deeply appreciated.

[ashimokawa]

I guess we would have to periodically compare Android's calendar events to our (to be created) local cache and see if events have been removed or new have been added. Then remove the removed ones from the Pebble and add the new ones.

This will be done, but please do not ask for ETA

[danielegobbetti]

I believe we need to expand our database model a bit, namely:

• a table for devices with unique ID(mac?), device type, ...
• a table for the events with: unique ID (we get it from the calendar provider), device_id (reference to the above table), status [clean=sent to device / dirty = locally updated], id_on_device, data

this needs to be extended and formalized, but what do you think of the idea?

[tecufanujacu]

I saw that with the latest update Gadgetbridge can now send sunrise and sunset events to timeline, so are we now able to manage the timeline pins to use the calendar in the new timeline?

[IzzySoft]

@ashimokawa

I guess we would have to periodically compare Android's calendar events to our (to be created) local cache and see if events have been removed or new have been added.

That might just be part of the deal: how many people would "delete" items from their calendars regularly? I'd say that would require a setting like "auto_purge after X days" (setting that to 0 would only remove "removed items", while a larger value would purge them after that many days no-matter-what.

@tecufanujacu AFAIK only sunrise/sunset for now.

[tecufanujacu]

@IzzySoft
Oh yes, I know that for now are been added only sunrise/sunset but in the past @danielegobbetti told me that we are not able to manage the new timeline pins of the firmware 3.x so I'm happy to hear that this is not more a problem because at this point we are able to add support for the calendar and if we will have an addon as spoken in issue #302 we can also have the support for the weather and Gadgetbridge will be almost perfect.

[IzzySoft]

@tecufanujacu Might take a little more time – but yes, that's what I'm hoping for as well :)

[ashimokawa]

Regarding this feature, we will first merge the db-refactoring branch, since we need a database for calendar sync.
We know how to send timeline pins since forever, that's not the problem. The problem is keeping this in sync without being able to ask the pebble for anything stored on it.

Sunrise/sunset is only an exercise. It works without database because of some clever logic with rotating ids, but that won't work with calendars.

For people who use/used the official app:
How do the sunrise/sunset pins look there?
It there any whether information on it?
I have never seen it. Screenshots welcome ;)

[stepardo]

Is anybody actively working on this ticket? I would like to offer my help.

A few questions:

• The original pebble app seems to sync the timeline if you migrate your pebble between mutliple (android) devices. If it cannot determine which pins are there on the watch, how can it sync? So the question is, is there a way to determine the timeline pins from the watch that we do not know about, or does the pebble app send the timeline pins (or their IDs) to the pebble server (even though they said on reddit that calendar events are never sent to their servers).
• What would happen if we do not delete timeline pins? Or asked differently: how do you know the watch does not delete them once they are no longer visible?

Update

Looks like timeline pins are simply added to a database of the blobdb service.

Maybe we could just wipe the timeline blobdb and install _all_ events fresh?

[danielegobbetti]

About expired pins and their automatic deletion I just tested in the past (I guess it was firmware 3.10) by syncing time to the pebble in the future (one week) and finding the past pins disappeared. But when I synced time back to normal, the pins were there again.

About deleting everything that is certainly a possibility but it's not the most efficient way :)

We also reflected on having a fixed set of pin ids and make them rotate. This is already implemented for the sunrise/sunset pins and works fairly well. The problem with this approach is that you potentially end up renewing far more pins than needed.

The best approach we found is to have a 1-1 mapping of event ids from android events and timeline pins (this is perfectly feasible, as event ids are 64 bit long) and a local cache that marks the event as "dirty" when something changes in the event content provider.

Currently our database is undergoing a big refactoring with the switch to an orm, until that work is done I fear it makes little sense for you to work on this. Unless you have a different idea that doesn't need a cache in the db :)

[stepardo]

I think having a 1-1 mapping sounds the most reasonable.

I am wondering why there is no query action for blobdb. Since each item must have an ID, this could be used by the host to determine the PINs on the device and other things such as the list of installed apps/watchfaces.

So my offer for help still holds. Please contact me when you need me.

[ashimokawa]

@stepardo
Thank you for your kind offer, as @danielegobbetti this is currently not a good time to implement that because our database crap^H^H^H^H structure is unclear^H^H^H^H^H^H in transition.

Anyway we would need to have a database and to quote myself

I guess we would have to periodically compare Android's calendar events to our (to be created) local cache and see if events have been removed or new have been added. Then remove the removed ones from the Pebble and add the new ones.

Also it would be better if we improve our blobdb code to actually map the success/error response to the previously sent commands, so that we can make sure events have been sent to the watch before marking them as synced.

And yes, it is unfortunate that we cannot query the blobdb, it would make the app manager also much better...

[sayash]

I'm not a Java / Pebble dev, but hope I can contribute somehow. I love this project, and it is the only reason I bought a Pebble! Thanks!

Based on the discussion on this thread, is this a useful summary of how the tentative code should behave?

• 2 new fields to each calendar event -
  
  
  
  • ID is the hash of an events' complete metadata (this should take care of edits / modifications)
  
  
  • NEXT-ACTION, an empty string
  
  
• Say primary collection called G in GadgetBridge
• Periodically updating list of calendar events in collection A for Android
• for events in set A intersect G, do nothing
• for events in set A - G, set next-action to ADD
• for events in set G - A, set next-action to DROP
• For each ADD, attempt upload to gadget, clear next action on success, do nothing otherwise
• For each DROP, attempt remove from gadget, clear next action on success, do nothing otherwise

I'll see if I can pick up enough Java to write this...

[danielegobbetti]

We made some brainstorming in the recent past and the raw approach you outline is correct.
The devil, however, is in the details :-) but we found a strategy to deal for instance with multiple pebbles (or other wearables) and deleted events and so forth.
So for the time being familiarize yourself with the codebase and we will share our findings soon (tomorrow in the morning CET ;-) )
Thanks for the heads up!

[ashimokawa]

OK I try to summarize what @danielegobbetti and me were brainstorming:

We plan to have two components
1. An Android Calendar <-> Gadgetbridge Database syncer
- Syncs X days in past and Y days in future
- will set a device_status field for each calendar entry and each pebble to "not_synced", "needs_delete", "needs_update" or "synced" (an enum)
- if status is "not synced" and the calendar entry gets deleted or is out of reach (>X days) it gets deleted from the database
- if an entry gets deleted and status is "synced" or "needs_update" we set it to "needs_delete"
1. A Gadgetbridge <-> Pebble syncer
- will go through the table for the connected pebble and create entries on the pebble that are set to "not_synced", update entries set to "needs_update" and delete entries that are set to "needs_delete"
- will set "needs_update" to "synced" after syncing and delete entries that got deleted.

@danielegobbetti, feel free to correct, IIRC you had the idea of having two status fields and a clearer separation between the two components.

[ec1oud]

It would be nice if any companion app could push pins too. E.g. a traffic app could push pins for nearby bus/train/metro departures. These would be short-lived though. It wouldn't make sense to put them onto the Android calendar.

[danielegobbetti]

Just adding this here as a note for when the development will start: if event reminders are sent to the watch, then we should check whether an android notification is created for a reminder that has been synced previously, otherwise the user will get two notifications (one from the watch, one from the android device)

[borghal]

Any news on this? Calendar events in the timeline would be great!

[chaseadam]

IMO this the last remaining critical feature of my Pebble which is missing. How can I help?

[stickybyte]

This feature would be great ... As Linux Developer, maybe I also can help?

[danielegobbetti]

@chaseadam @stickybyte since the feature has yet to be modeled, a feedback on our current idea (outlined here ) would be appreciated.
Do not forget that Gadgetbridge is a multi-device software when giving feedback ;-)

[chaseadam]

The concept looks good at least for pebble. Do the other devices have this timeline concept or other calendar/event concept?
Might be useful to separate a full timeline database and the calendar "sync". This way we have a generic way of managing the timeline (like the pebble web api provides? I am not that familiar with it). Could be useful for other timeline entries (I.e.weather)

[Avamander]

In addition to syncing the timeline, it would be nice if watchapps and watchfaces also had access to timeline/calendar events to some extent.

[chaseadam]

I believe the f step is getting the timeline. As for the watchface, I haven't made one, but if it has access to a timeline, then it should be able to use it. If not, then it would require additional work which would likely build on top of the work to get the timeline functional.

The concept is good, any advice on how/where to start? I recall reading that work already started on some database support?

[danielegobbetti]

@chaseadam @Avamander I believe feature-parity is priority over extending the available features.

FYI putting pins on the timeline is already supported (sunset and sunrise are just other types of pins).

The concept is good, any advice on how/where to start? I recall reading that work already started on some database support?

Having a POC that is able to interface with the android calendar provider would be the most welcome contribution ATM. It could work against an in-memory structure (instead of the DB) for a start, and should be able to identify changes and deletions in events and flag the in-memory copy accordingly.

[SolidTux]

I tried to implement the basic structure in an additional receiver class, but the data is cached in a hashtable and not yet writter to the database.

But I'm not really experienced in Java and Android development, so it might be rather ugly code.

@ashimokawa @danielegobbetti What is your opinion?

https://github.com/SolidTux/Gadgetbridge/commit/4b0fd313620cf8850c8db04eb272012d7c2ecb77

By the way ... is there any way to delete all timeline pins without knowing their IDs? And is there a limit on timeline pins?

[ghost]

I'm not sure whether there is a limit, but I would reckon that pins up to 48h in the future make sense but not necessarily past that.

On my Pebble (still using the official app) the timeline goes to the end of the day after tomorrow, which I find more than enough ;) Maybe that's a good value to use (as it would be the same as the original)?

[SolidTux]

Sorry, should have formulated my question clearer ... I wanted to know if there is a limit on the number of timeline pins.

[danielegobbetti]

@SolidTux thanks! Will have a look at your code over the weekend.

By the way ... is there any way to delete all timeline pins without knowing their IDs?

Unfortunately not (that we are aware of / that is officialy documented), that's why we need to keep a mapping in the app.

@Bebef yes 48 hours is the limit for what the watch shows, but I believe a few more hours / days would make sense in case you are not connected to the phone. My idea is to have an upper limit of events/pins (e.g. 50) and of days (e.g. 7), whichever comes first.

I am not aware of a limit of the total number of pins.

[SolidTux]

What I am concerned of is that, if you use the watch on multiple devices, there might be pins left which are not deleted and they could accumulate and that might cause problems over time.

Maybe it would be good to have a limited number of rotating IDs like with the sunset/sunrise thing?

[danielegobbetti]

If you take a look at our brainstorming we thought to keep a two-fold mapping (android calendar - gadgetbrige database ; gadgetbrige database - specific device) for this very reason.

The problems with rotating IDs in this use case are (IIRC):

• calendar events could be updated and swap position,
• we cannot update a PIN, we must delete and recreate it

[ashimokawa]

We can delete all pins without knowing anything. But that will also delete sunrise pins etc

[danielegobbetti]

@SolidTux sorry the fix of #643 had precedence over this, but I will review your code in the next days! :)

[SolidTux]

@danielegobbetti No problem, you all spend so much of your precious time for this project, thank you!

[ashimokawa]

@SolidTux
Would you mind doing a PR in branch feature-calendarsync? I would like to experiment with it in a that newly created branch and maybe do the database stuff.

[stepardo]

If you need inspiration, have a look here, this should cover the basics.
https://github.com/JanBobolz/agendawatchfaceAndroid

It is MIT licensed, so it should be fine to have a look into.

I did not have a look into the "basic calendar sync" merge yet. Could be that you already figured everythin out on your own.

Keep up the good work!

[danielegobbetti]

@stepardo thanks for the pointer but by a quick glance it seems that it refers to an additional Pebble app. Our aim is to show the android calendars on the pebble timeline, anyway for inspiration more code is better than less code! :+1:

[stepardo]

@danielegobbetti Sure. I got that. I just thought it would be cool for you to have a working reference implementation of all that android calendar stuff in order to get calendar events into gadgetbridge. The agenda watchface is just that: a watchface that displays calendar events. It is meaningless once we have timeline support.

The bigger question is the gadgetbridge database and how to get these calendar events to the pebbles. And to be able to switch pebbles (people might have more than one pebble); keep track of which events are synced to which pebble etc. pp.

Again, thanks for all your great work!

[ashimokawa]

This feature is currently actively worked on in branch feature-calendarsync and coming along quite nicely.
You should however not try this branch unless you are willing to completely wipe your GB data before and after testing (database schema still not stable in that branch)

[ashimokawa]

This is merged in master and will be in the next release
Support is basic but working. We sync the android calendar events to the timeline with name, start and end time (no other information yet)

I am closing this now as we can say, we support sending timeline pins (we could before with sunrise and sunset but that was only a test)

[Avamander]

But there's no API for app devs or watchapp devs to use to push timeline pins? Calendar pins are only a part of it.

[ashimokawa]

@Avamander

That will be dealt with after the javascript branch has been merged

[Avamander]

That deals with the watchapp JS part, but not the companion app part, right? And what about server-based timeline pins, any ideas how to restore that functionality?

[danielegobbetti]

:notes: I don't have digital. I don't have diddly squat. It's not having what you want. It's wanting what you've got. :notes:

I'm sure that as soon as a contributor has that itches to scratch, there will be a PR addressing your use cases. For the time being let's enjoy calendaring support (and not forget that our call for ideas in #528 went desert).

[roidelapluie]

Congratulations & Thanks!!

[Avamander]

It's all really awesome indeed, advertised it in a few places even. Just asking for information about restoring the full functionality of Pebbles.

[IzzySoft]

@Avamander to me, it's already more than the original functionality (I never used the official app even back when it was "officially run"). And think again of your question of "server side features", considering GB has not (and will not have) the INTERNET permission. Feel free to write a companion app :smiley_cat:

[Avamander]

What about adding a special build type to GB to add networking functionality for those who wish it?

[danielegobbetti]

Just asking for information about restoring the full functionality of Pebbles.

We actually have more functionality than the official app thanks to 810ba5419be080170c351936cd72a0e368a10d17 , but I digress :-)

What about adding a special build type to GB to add networking functionality for those who wish it?

https://github.com/Freeyourgadget/Gadgetbridge/issues/302 is what we are going to do, so no need to have separate build types.