Gadgetbridge: Integrate with self-hostable fitness tracking service

[cpfeiffer]

The android application is probably not suited to record and keep, manage and visualize all fitness and sleep data, and what not.

Online services like Google Fit or Apple Health are obviously out of the question.

After digging a little I found these projects (in order of viability):

• https://wger.de (python, django, quite active, but tailored towards workout)
• https://fluxtream.org/ https://github.com/fluxtream/fluxtream-app (service for collecting and managing all kinds of data, maybe a bit heavy for our needs, but still would be nice to support)
• http://www.openmhealth.org/ health oriented service, https://github.com/openmhealth
• http://giudici.us/work/communicating_health/ (research project, probably inactive, code is here: https://github.com/theguice/wearable-health)
• https://github.com/jdjkelly/quant (ruby on rails, dormant for almost a year, but very much what we want)
  https://bitbucket.org/dalareo)
• https://github.com/z1lc/qs (consists mostly of charts, unfortunately Google Charts)
• https://github.com/selfhub/selfhub (no demo site?)wonder how self-hostable this is without a dedicated cloud provider like AWS, Azure, ...)
• http://openfitapi.com/ (Drupal 7 extension, but dead since December 2012, code still available at

[chrisvella]

The other option would be to create something ourselves. I don't see why a html5 page wouldn't do the job. For the graphs http://www.chartjs.org/ looks really good.
I'm not sure how to we would get the values from the app as js variables. If that is worked out I may be able to provide assistance in creating the interface for the tool.

Pushing the data via git could be interesting. We could create a github page template that generates the graphs based on the contents of the file. Then anyone could easily host it for free, and easily replicate the setup by forking our repo.

If one of @cpfeiffer options is selected, making a openshift compatible quickstart would help adoption.

[ashimokawa]

@chrisvella
chartjs is what morpheuz uses (http://www.morpheuz.net)
Looks good to me also.

But I really disagree with @cpfeiffer here. I think an Android application IS suited to "record and keep, manage and visualize all fitness and sleep data, and what not."

We could take our data and render it locally in a webview without internet access if we really want to base this on html5.

If we add an "export to some place in the internet" option I would prefer writing a second app which complements GadgetBridge and has internet access to act as a forwarder between Gadgetbridge and the internet. Because adding internet access to Gadgetbridge is something I would like to avoid.

[chrisvella]

@ashimokawa I agree with you completely.
Interested to hear your thoughts @cpfeiffer

[cpfeiffer]

Here's my thoughts:
1) A mobile/handheld device is not suited for keeping/archiving data (be it fitness, sleep or any other data).
Reasons:
a) devices get lost, stolen, broken and backups are so unsexy that few people have them. So for those people that want to keep that data for more than a few days, an _external_ location is better suited.
b) some people may even have multiple mobile devices and track their activities with one or another, but want to have their data accumulated and consolidated in one place.
c) the fitness guys and gals may want to share their data, have competitions and whatnot, maybe they want to annotate their data with notes, images, etc. I would rather have that in a separate (web-) application than Gadgetbridge (or a helper app).

2) I completely agree with you that synchronizing with or exporting to a [self-hostable] web service should be optional in such a way that the required Android permissions (i.e. INTERNET) are optional, too. With current Android situation, this means that the export/sync support needs to be in a separately installable app -- which is perfectly fine for me.

3) Having some local rendering support in Gadgetbridge is needed as well, of course. Just to have a look at "how well did I sleep last night", or to track fitness activity over a short time, say a few months or so.

[chrisvella]

It sounds like a html/js implementation of this would be a good compromise? That way we could put it on a self hosted service AND have a local view.

[cpfeiffer]

If you ask me, yes, sharing code between the website and the local view would be beneficial. I think @ashimokawa prefers to do the charts himself though, because of much better performance. Let's see how it works out.

An html/js solution would definitely be a good thing, though, both server side and client side, e.g. for more sophisticated graphs.

[ashimokawa]

Actually using the android canvas is not more difficult than using html5 canvas.
But performance and - much more important memory to me - memory footprint is far better. The chromium based WebView since Android 4.4 likes to eat up a huge chunk of my 512MB.

Another problem I see is license of the js code we would have to include.

I am not against sharing js code if someone comes up with a good implementation and I can see the benefits and he/she explains how to deal with licenses of imported js code.

[cpfeiffer]

Alternatively, we might use something like https://github.com/dodocat/MPAndroidChart or https://code.google.com/p/achartengine/ or https://github.com/blackfizz/EazeGraph although we would lose the benefit of sharing code with the hosted version.

But we certainly need a db schema before deciding on this.

[danielegobbetti]

About data visualization we could also add something like this:
https://github.com/dlazaro66/WheelIndicatorView (NB: code quality / license unchecked!)
for the google-fit lovers (if any ;-) )

Btw for inspiration this community contains a lot of projects:
https://plus.google.com/communities/100609058582053363304
It's on google plus but can be accessed also without being registered/logged in.

[cpfeiffer]

Yeah, circles are hip :-)

[cpfeiffer]

FWIW, I've got a prototype displaying sleep and activity data with MPAndroidChart.

[danielegobbetti]

Cool, looking forward to that ! :)

[1337sup3rh4x0r]

me too!
Are you guys planning on rolling out released apks more frequently in the future or would it make sense for "interested noobs" like me to look into how to build from this source?

[ashimokawa]

Yes, definitely. The next release will be out in a few days. And from then on we should release more often like it was the case in the beginning.
The "problem" is that we tried to tackle something big which is not yet in a releasable state but at the same time we had important bug fixes we want to have released.
My proposal is to release 0.4.0 on the weekend. That release will contain new and incomplete features which will improve and stabilize during the 0.4.x series.
0.2.0, 0.3.0 were the same. New features but stabilization took a few point releases.

[1337sup3rh4x0r]

No need for stability from the start in this early stage. I'll be happy to fiddle around with anything you give me and report bugs. Really looking forward to wear the miband to track steps and sleep with the confidence that the data stays where it should!

[chrisvella]

If CSV export was developed, we could explore something like: https://rohitrox.github.io/js_csv/
Users would be able to "host" the fitness tracking service on static hosts like github OR have it run offline with very little fuss.

Not sure how possible this is, but it would be super cool if users could fork a repo with the fitness tracker (pretty sure github lets you host static sites pretty easily). Then configure from the app a destination for the CSV file (could we have it committed straight to the forked repo?). Then the fitness tracker would automatically load the file.

@ashimokawa take a look at this: http://www.androidpolice.com/2015/06/06/android-m-will-never-ask-users-for-permission-to-use-the-internet-and-thats-probably-okay/

[cpfeiffer]

@chrisvella Yes, that would be pretty cool!

[rolandgeider]

Hi! I just found this issue by accident, I'm the main developer of wger. While my app's focus are workouts, I'd be interested what other data you guys would be logging and graphing, perhaps I can get some ideas ;)

[cpfeiffer]

Hi Roland, cool to have you here :-)
we haven't made any progress or even decision on this issue yet, because we have lots to do.

Those activity trackers are primarily used for walking, AFAIU (being a non-walker myself -- I prefer other sports). Those devices typically count 'steps', but have some support for other sports activities. I don't know whether we'll ever be able to support other activities, though because that would probably mean interpreting raw sensor data which would involve much more time and knowhow than we currently have.

Another interesting type of 'activity' is 'sleep'. The devices track your daily sleep, not only in hours, but also devided into light sleep and deep sleep.

So, to summarize: our primary needs of supported activities are:

1. walking
2. sleeping

Everything else is future stuff, unless someone comes up and helps implementing it.

PS: sorry that I summarized wger with such few words, I was actually amazed at what wger already provides

[rolandgeider]

Hey!

Thanks for the kind words, but my project still has a long way to go! :) I had already thought (shortly) about adding some kind of sleeping log, but was unsure on how to do it or whether it would be useful at all if the data has to be entered manually. Should I implement this, I'll let you know

[cpfeiffer]

But the workout data has to be entered manually as well, no?
We could provide daily automatically generated sleep and steps data :-)

I'm very interested!

[rolandgeider]

Yes they have to be entered by hand, but those are more tangible, since you enter them as you go or right after the workout. Sleep is more difficult without a gadget (how many hours did I really sleep? and the light and deep phases are out of the question).

Another possibility would be to implement on my side only REST API endpoints (i.e. without a GUI) for the sleep and perhaps even the step data.

[cpfeiffer]

I certainly agree that one wouldn't want to add such data by hand -- I wouldn't want to do it with other activity data either :-)

And yes, an API endpoint would be enough for users to upload their data.

Is wger somehow customizable in what types of activities shall be available? While we can interface with many different systems, I think we should also promote a default web service which is focused on what Gadgetbridge supports; which is sleep and walking (steps) at the moment.

[janvonde]

I discovered this project yesterday and it is exactly what I was looking for for a long time. Using a cool gadget like the Mi Band collecting personal information but without the need to upload it to any company.

I would really love to have a simple self hosted solution where I can push my data. I am thinking about a small PHP script that receives the data via POST and stores it on a sqlite/mysql database. You could secure the transport if you serve the website via HTTPS. You could secure your data if you'd put a basic authentication in front of it.

The app would need to support the following:

• configure optional full URL to the self hosted PHP script
• accept valid https connections
• support basic authentication
• send data via POST to this URL

The data could be send a JSON object for example.

I am willing to help the project, I have very limited PHP scripting skills but I could setup a sample site and script that does exactly this.

[chrisvella]

@janvonde FYI gadgetbridge supports exporting as a sqlite file. If you plan on developing something, it might be an idea to work on importing an sqlite file for now.

I had a look into a html5 way of working with gadgetbridge exporting. I have tested it out, sql.js is able to import gadgetbridge files. https://github.com/kripken/sql.js

[cpfeiffer]

@janvonde Yes, that would be awesome. Do you only want to get hold of the data and later see what you can do with it? Or do you want to have statistics, graphs and possibly adding of additional data in the php script?

As @chrisvella pointed out, right now you can export the entire sqlite database and do all kinds of things with it (see the 'Debug' activity). Maybe a CSV export would be more helpful than the entire database, though (see #67).

In any case -- help in that area is more than welcome :-)

[cpfeiffer]

Updated the order of the researched projects.

[janvonde]

@cpfeiffer I want to receive the data on my own server, store it in a database and see how I can visualize it there. I only want to visualize existing data that I receive, no manual input or other things are in my head so far. Simple and stupid but secure and hopefully some nice graphs ;)

[cpfeiffer]

@janvonde That's the perfect start. Maybe other data will come later, e.g. annotating what you've done some day, others might want to record their weight (loss), etc. But yeah, please start small :-)

[janvonde]

How can I export data as sqlite?

[cpfeiffer]

Use the debug activity and press export database. The db will then be the gadgetbridge directory on your SD card.

[janvonde]

Is this a released feature already available at f-droid?

[cpfeiffer]

No, apparently it's not released yet, sorry. It will be in 0.4.4.

[janvonde]

Thanks for the information @cpfeiffer . I will keen an eye on this issue. Once 0.4.4 is released I will look at the database. In the meantime I set up a vhost with https and basic authentication already. I put three lines of php code to make sure I am proper receiving JSON objects via post. I will look now for a basic user interface. Once I've seen the database structure including some example data I can make a propose on how the JSON object could look like. After you've seen the first result (don't expect something to big or beautiful) you can decide if you would like to have the code somewhere here as well. But I will publish it anyway 8-)

[cpfeiffer]

Maybe you want to have a look at how the other systems listed in this issue do it, i.e. how the API looks like. Being somehow compatible to those, maybe even to Google Fit might be a good thing. I would rather see our data adapted to the needs of the other systems than have other systems directly use the values as found in our database (which might change at some point).

After you've seen the first result (don't expect something to big or beautiful) you can decide if you would like to have the code somewhere here as well. But I will publish it anyway 8-)

Promised? :-)

[rolandgeider]

Sorry for the late reply!

None of the logging in wger is customazible in the sense that each type has its own specific table, REST endpoint, etc. and adding a new type is not just making some configurations. While I'm not sure I can add them in the near future, I'd like to know a bit about the data structures. Is each step logged individually or are they aggregated every $time_interval? Do they also have geographic information? Or some kind of intensity/acceleration? (You can also just point me to a model file)

A different thing I wanted to mention here, since I keep reading about HTML and graphs: this [1] JS library that sits on top of D3js (the gold standard on JS graph libraries) and focuses on making time based graphs easy, which is what is being logged here. I plan on using it in wger and I'm just waiting for it to support one additional feature I need. Compared to manually using D3 it's really like day and night.

[1] http://metricsgraphicsjs.org/

[cpfeiffer]

@rolandgeider No problem, we all have lots to do...

These are good questions! The devices play a role in this. The Mi Band provides one sample of activity data per minute and both step count and intensity.

The pebble does not provide this out of the box at all, you need a custom watch app that does this and so it's up to the watch app developer. With the morpheuz watch app it is one sample aggregated per 20min, in the near future we will get one sample per 10min.

Geographic information is not available, but could be added by Gadgetbridge if wanted (you would need to carry both the mobile and the device, though).

I haven't yet looked at what other activity tracking sites have as data model.

Thanks a lot for the pointer to metricgraphics, this looks awesome!

[simonvanderveldt]

I discovered this project yesterday and it is exactly what I was looking for for a long time. Using a cool gadget like the Mi Band collecting personal information but without the need to upload it to any company.

I would really love to have a simple self hosted solution where I can push my data.

Wanted to say thanks for the cool app as well and add my +1 for gadgetbridge to function as a simple bridge ;) as proposed by @janvonde

It would only have to support the following:

• Configurable endpoint
• JSON formatted data
• Push only new data

But I really disagree with @cpfeiffer here. I think an Android application IS suited to "record and keep, manage and visualize all fitness and sleep data, and what not."

Also wanted to say I really disagree with this ;) Keep the functionality of the app focussed, i.e. make it the best bridge app possible.
If visualisation is needed simply create a separate app for that or delegate it to a self-hosted web app.

Also @cpfeiffer open mHealth seems like a good option.
I'll probably try to get open mHealth, fluxtream and quant running in august to see how they compare.

[cpfeiffer]

@simonvanderveldt Thanks for the pointer to open mHealth, I added it to the list. Looks good to me.

And please let us know about your findings with the services you testdrive, this will help us a lot!

[animalillo]

This would be a lovely feature, so you can check your progress from your pc, and no need to take the phone for everything, plus bigger screen, better and bigger graphs... endless possibilities.

I love this app, but right now is missing a lot of what i expect from the xiaomi band, based on what the official app offers.

I'm a professional php developer, a bit busy right now, but wouldnt mind making a rest interface for the data storage based on json.

Maybe by the end of this month or during the next one i take a look at the android database and try and make a php interface for the app.

But for making something would be good to know what to expect and where is the project going.

Also, regarding the internet permissions, I dont mind either way, as the app (or both apps) are OpenSource, it could be bundled on the same app, or included in an external optional api app, i dont really mind this point.

[cpfeiffer]

I love this app, but right now is missing a lot of what i expect from the xiaomi band, based on what the official app offers.

We know and we're working on it. It's just.. a lot of work, so it takes time. But we have most of the boilerplate code in place, so (visible) progress in accelerating, and it's quite easy to add new features.

I'm a professional php developer, a bit busy right now, but wouldnt mind making a rest interface for the data storage based on json.

Excellent, exactly what we need :-) We can create a git repository for it and give you all permissions, if that would be OK with you. Maybe you even want to team up with @janvonde.

But for making something would be good to know what to expect and where is the project going.

See the list of issues at https://github.com/Freeyourgadget/Gadgetbridge/issues/ for an overview. It doesn't mean that we can really do all of that, but we would like to if we find the resources.

For a start, we want to support

• tracking sleep activity
• tracking sports activity
• notifications on the device

We're already support much of this, with some missing pieces and missing polish.

[simonvanderveldt]

And please let us know about your findings with the services you testdrive, this will help us a lot!

@cpfeiffer I've looked at a couple of the items in the list:

• Open mHealth: Very loosely coupled components + lack of documentation made this a difficult one to get going. But, the major issue is there's no integrated dashboard/way to view your stats. I'll post a question on their mailing list about this to see if they have some suggestions.
• fluxtream: looked at the setup instructions and then decided that that's way too difficult for 90% of the people
• Found a new project Reportr and tried it out. Was relatively easy to setup, but the charts are fairly limited and for some reason it only showed counts for measurements instead of actual measurements for me.

So, unfortunately nothing sticks out as being a good solution :(

I'm using InfluxDB together with Grafana for the (time series) data from my Home Automation system and guess I'll just use that for my Gadgetbridge/Quantified Self needs as well...

[cpfeiffer]

@simonvanderveldt Thanks for reporting back, much appreciated.

InfluxDB and Grafana look indeed nice. if you have something to share, even if just a report how you set this up or a screenshot, please do :-)

[simonvanderveldt]

I tried quant today. Again basically no documentation on how to build/run it, but managed to make it work. It needs a postgres db to store all measurements, which makes it a bit of a hassle, but doable.

In general it covers most of what people will need, but the views/dashboards are limited to how they come out of the box, there is no way to create your own dashboard from the UI.
Included views/dashboards are list below with behind them the values that are shown in cards on the dashboard:

• Weight (weight, fat%, lean mass, date, calculates BMI)
• Meals (Calories, Fat, Carbs, Protein, Description)
• Sleeps (from and to, calculates duration, earliest start last 7 days, recording streak and health sleeps count for last 14 days)
• Journal Entries (Happiness, Feelings, Strategies)
• An overview containing all of the above cards

All of the views contain the above mentioned cards + a tabular view of all the data. There are no charts or further historical views of the data.

So what's missing is a dedicated steps/activity view + charts and history.
Also I haven't looked at how to get the data in automatically (though you can enter data via web UI which are POSTS to the API, so that shouldn't be a problem).

Out of the options I've tried I'd say quant is the best option, but it will need some work to make it usable (adding history/charts + steps/activity).

I personally don't like being limited to predefined dashboards, so I'll stick with my current idea of Influxdb + Grafana.

@cpfeiffer Once I manage to get some data in from Gadgetbridge I'll post an update here with info + a screenshot

[danielegobbetti]

A new (to me) service: https://github.com/openmhealth/shimmer.

Nevermind, it was already in the list on the first post. Sorry for the noise.

[bigretromike]

Hey,

Is there a roadmap for this ? or is it - in day far far away ?
Why is "Online services like Google Fit or Apple Health are obviously out of the question." ?
I see that alot of people that will force to use gadgetbridge and still want the Google Fit sync will self-host anything that will be only as link to google_fit_botnet.
Why not make it posible to use it directly ?

[cpfeiffer]

@bigretromike Good point, we should also support syncing to proprietary services (in whatever way). Could you please open a separate ticket for this?

[danielegobbetti]

ups I just closed an issue about this. I'll reopen it

[danielegobbetti]

This javascript port of sqlite seems to work well:
http://kripken.github.io/sql.js/GUI/

In fact I just loaded my database (freshly exported from GB) and run queries on it. So perhaps someone is willing to implement a nice dashboard/gui on top of our database ?