Gadgetbridge: Automatic export of health data

[Avamander]

I'd really love if GadgetBridge allowed me to make it automatically export it's database to somewhere on the system so I could use software like Syncthing to get a copy to my PC and other devices.

• [ ] Exporting
  
  
  
  • [ ] After a number of changes
  
  
  • [ ] Timed
  
  
  • [ ] Intent
  
  
• [ ] Ability to specify the output location

[jpbarraca]

Or a private MQTT broker?

[delijati]

There seams to be already a exportDB functionality!?

[Avamander]

That's where the "automatic" part comes into play. It's rather cumbersome to sync it right now.

[cpfeiffer]

Totally agree, we need an automatic export, be it CSV, DB or something else.

[Avamander]

Ugh, lost my data today because I forgot that the data folder on SD card is wiped on uninstall and pressed "Uninstall app? Yes." in Android Studio.

EDIT: Totally not your fault though, just that let me be the "first" and the last.

[cpfeiffer]

I'm very sorry that this happened. It's a sure sign that we need to get a solution for this soon, rather than later.

[nickvergessen]

A collegue and me both are working on Nextcloud. I wonder if it would be possible to integrate it somehow. E.g. let users specify their server and the data is then saved on the server, rather than the phone. Just as an idea, maybe when you implement the feature you can also keep this idea in mind, so it's easier to implement something like this at some point.

[IzzySoft]

@nickvergessen replace "rather than" by "in addition to" (think: Backup), and it sounds interesting to me as well. Especially if you come up with an app for oC/Nc making use of those data :smiley_cat:

[nickvergessen]

I'm not good with UI/JS stuff, but I'm surly willing to help with the API works in the background.

[danielegobbetti]

@nickvergessen I tried to reach out to Nextcloud GmbH back in december without much success. I certainly would love to see this happen. Especially if we manage to enable synchronization on the android device and let the nextcloud app taking care of the offsite sync.

[ghost]

Why that cloud bridge? Not everyone wants to store that data in the cloud. Just save it on the device w/ a configurable path and let other apps sync it to the cloud.

[IzzySoft]

@mankoell good point I was taking for granted (as GB has no INTERNET permission). So, read (as per the issue's title): „automatic export of health data to a place where it can be sync'd from“ :innocent:

[schiessle]

This sounds great, I can also help on the Nextcloud side. As @nickvergessen I'm also not a UI/JS expert but showing some diagrams, etc should be possible as a starting points and then hopefully other people will join.

[schiessle]

I think it would be possible to just sync the data to a folder on Nextcloud and a app could pick the data up from there. The Notes app does something similar. But in case of Notes we sync plain text files, while in this case the data dump would be less useful in general. Also it could be hard to detect the changes in the dumb to update the Nextcloud db accordingly. Therefore I would prefer a API end-point where you can sync the data directly on a regular basis. It would be optional anyway because people would need to configure their Nextcloud first within Gadgetbridge so I don't see a issue regarding privacy here.

[jpbarraca]

An alternative would be to develop a simple helper bridge providing internet connectivity to GB. The standard GB would have no INTERNET permission and this helper, which would be a different installation, could provide that functionality to users (read the DB directly and sync it continuously with an external service).

[Avamander]

Basically, an official helper would be needed, I think, in the end to reduce reliance on 3rd party modules, be it for calendar, weather or other functionality like app stores (Rebble store). But as far as syncing is concerned, the syncing to internet should be done by specialised apps not GB.

[IzzySoft]

@schiessle You still miss the fact that GB does not (and never will) have the INTERNET permission. So from GB's side, sync can only be done locally on-device – from where a sync app (oC/Nc/FolderSync/…) can pick up data and sync them to the cloud instance. For example, data could be exported as JSON, XML, CSV or the like, which can be easily sync'd (and even diff'd). If possible, I wouldn't make yet another app required for that – as folks who use cloud storage in most cases already have a sync app anyway.

[cpfeiffer]

I think these are indeed two topics:

• using Nextcloud for backup
• using Nextcloud as a health/fitness/data tracking service (See #49)

The former could indeed by handled by exporting the database and preferences and letting the Nextcloud client sync it.

The latter would indeed need an API for incrementally synchronizing data, which would probably be handled by a Gadgetbridge companion app.

[cpfeiffer]

For using Nextcloud as a data tracking server, we need to come up with a data format. The openmhealth guys have defined lots of schemas at http://www.openmhealth.org/schemas/
There's also the APIs of Google Fit, Apple Health, and the like.

At the moment we can provide the following data:
1) user metadata like

• user name
• age
• height
• weight

2) device metadata like

• model
• hardware revision
• firmware revision

3) software metadata

• Gadgetbridge version

4) actual data, all timestamped

• heart rate values
• steps
• distance walked
• burned calories
• "intensity value"
• recognized activity (e.g. kind of sleep, running, etc.)
• user specified activity

This is just what we have at the moment, we intend to add more as time permits and devices are available. We would also like to support gardening/plant growing gadgets, for example, so these would certainly produce some very different data :-)

[nickvergessen]

Okay, so I started to work on this yesterday evening a bit.
Currently I have a skeleton nextcloud app, where you can select/import the file you export in the android app.

I would then continue with importing all the data into the nextcloud database or should we just operate on the file directly, since we only read and don't write?

Who would be willing to do the magic with JS and UI? because now I would need the info, how you want to get the data from the backend? just in table rows, and 24hour sets is enough?

[schiessle]

@nickvergessen having it in the database makes it probably easier to display various graphs, etc because then we can make sure to create a db structure which is most suitable for it. Also if you export from your mobile phone and import it into Nextcloud you will have multiple files over time.

Personally I would still prefer to have a app on the Android side which optionally syncs the data regularly to a user given endpoint.

[nickvergessen]

I continued a bit with the app:
https://github.com/nickv-nextcloud/gadgetbridge

You upload the database export to your nextcloud and select the file in the UI.
It then loads the devices and when you select a MiBand it does a simple ChartJS operation on the last day.
However I had a hard time to identify which sample data means what.

@cpfeiffer you have any idea who did the android activity and could help with that?

[ashimokawa]

@nickvergessen

You mean the rawKind column?

these are device specific, and *SampleProvider classes could help. In your case:

MibandSampleProvider.java

Miband2SampleProvider.java

However, especially with the Mi Band2, there seem to be some unknown values, especially in later firmwares.

[danielegobbetti]

As discussed a few days ago at the SFSCon in Bolzano I created a new repository for a companion java app that would be responsible for syncing with nextcloud.

I created a project for the first version/MVP where I summarized the outcome of the discussion. Please check https://github.com/Freeyourgadget/nextcloud-client/projects

Feedback welcome!

/cc @schiessle @nickvergessen

[Avamander]

@danielegobbetti What about directly writing the backup to disk and leaving syncing to any cloud up to the user?

[danielegobbetti]

Nextcloud devs prefer not to have a file/folder to appear to the user since nothing worthwhile can be done with it. I think what you are proposing makes perfect sense but it's orthogonal to what we agreed to achieve with the new repo/app.

[IzzySoft]

@danielegobbetti Might not hurt to have both, and leave it to the user which one to enable. Whoever uses Nextcloud will have to use that service – everyone else will probably either want nothing or the other option (depending on demand).

[schiessle]

@Avamander @IzzySoft keep in mind that it is not (only) about backup but we want to store and visualize the data on Nextcloud. Sure if you "just" want to have a backup, uploading it somewhere would be enough

[Avamander]

@schiessle Yeah, but that doesn't fix this issue.

[maufl]

I'm looking into it right now #930

[cpfeiffer]

930 was merged, hooray for @maufl!

[maufl]

@Avamander currently it is only exported after a configurable amount of time. Exporting based on an Intent should be easy (we currently schedule a periodic broadcast Intent to start the export). Exporting based on changes might be more difficult. Does this solve your problem?

[nickvergessen]

I'm trying that at the moment and would have 2 requests:

1. Immediate export on enable (so you dont have to wait 1 hour until the first export happens)
2. Would be nice to have a "Export now" button in the Debug options if possible.

[hj91]

I would like to have a mqtt plugin wherein i can send my amazfit data to my private server for analysis.

[farverio]

@danielegobbetti Hoping to see what the status of this and the nextcloud-client is? Read through all issues on this repo but couldn't see what next steps or bottlenecks might be.

[farverio]

@ashimokawa Noticed you might still be actively maintaining this project. Any thoughts? How do you persist logs beyond the life of your phone?

[ccgx]

Would it be possible to add the option to get a timestamped filename (like backup20190222) of the automatic export? Would help greatly with syncing with nextcloud...

[iicurtis]

Would it be possible to register an Intent in Android from an external app and have it sync directly rather than through a file? This way the app can respond and update as soon as Gadgetbridge gets an update? This would go a long way to #248.

I'm thinking something more like how PasswordStore integrates with OpenKeychain. We could possibly create an OpenIntents method for this similar to what was used in OpenKeychain. I haven't looked into the Gadgetbridge side yet, so I'm not sure what fields we want to transfer and how it should be serialized.

[geekpete]

Live MQTT streaming would be quite an interesting capability.
Even batched MQTT exports would be really awesome, allowing the data to be ingested into whatever other tooling afterwards.
I do realise the lack of the permission would prevent this, so what about a possibly secondary shipping app that performs only this action and works with Gadgetbridge to do this?