Talk-android: Feature request: support notifications for f-droid version

Created on 24 Jul 2018  ·  196Comments  ·  Source: nextcloud/talk-android

In #58 it was asked to release an version to F-droid , thanks that this happend.
Also there were a huge discussion related to #75, why not to use gcm/fcm.
To summarize, there are users which have concerns to use Google- infrastructer in anyway.

75 was closed because of the relation to #58.

But now we do have a gcm/fcm free version of talk so I would like to ask to support notifications in this version.
My proposal would be to handle it like every other open source messenger like telegram, signal or riot by keeping it in foreground through a notification.

Thanks in advance! And please keep up the great work.

enhancement help wanted 🔔 notifications

Most helpful comment

@mikelupe we're working on this regardless of the customer or not, we'll do this for the community - it's just taking a bit of time.

All 196 comments

+1

You do understand this will kill through battery and still be super NOT reliable?

Well it is still worlds better than NO notifications.
Riot (F-Droid version) for example works quite well even with 20 sec checks. The app places a notification icon and it worked for me on Android 6 and 8.1. It was reliable. Now i'm not arguing in some setups this could create issues, but you could take a look how Riot works, it seems quite all right.

I used riot in the passed and there was not that big battery drain same with signal what I'm still using with for example 21mAh over the last 8 h.

I do know that it is probably not that reliable as gcm/fcm but I think for people who live completely without google it is not the biggest downside.

For example signal gives a warning to the user when it is set up without gcm.

I would be really happy if this could be done because then I could drop signal

Kind regards

Maybe in addition the question if there is a technical debt which makes an implementation similar to riot, signal, telegram ... impossible?

@nr458h I used the fdroid Riot for a half a year or so and it was very stable. I used even a faster, 20 sec refresh time. The overall battery time improved only marginally after i installed the google play version.

Maybe one last post to convince you :)

It came to my mind that NC advertises metadata-free communication, so on nextcloud.com/talk it says:

"

Unique protection

Nextcloud Talk goes further than other encrypted communication technologies by keeping metadata private.

While public solutions like Whatsapp or Telegram might encrypt content, they and whomever monitors them still can see who you communicate with and when. With Nextcloud Talk meta data never leaves your server."

With this in mind it could also be an option for the gplay version!

Not metadata-free, just that it's all on your server. Push is the only way to wake up a device. All content in a push message is encrypted with YOUR OWN DEVICE-SPECIFIC KEY which can be verified.

@mario I think people realise the downsides of not using GCM, but as gradinaruvasile said, it's still better than not getting notifications at all (or being forced to install MicroG for one app). And other apps have been doing this for quite a while now. Not as the primary or ideal way of delivering notifications, but as an alternative method for app versions delivered outside of the Playstore and for people using custom ROMs without Gapps.

If that's not a priority for you right now, it's totally fine, but people know what they're asking for.

+1, @DudleyDursley summarized it perfectly.

A little repetition for newcomers (original link): the only thing I wish I knew is how WhatsApp does it as it works perfectly and shows no "permanent notification" at all (maybe a hidden notification? I know the icon can be hidden but whole notification?) without any noticeable battery drain. Facebook Messenger works the same at least for calls (I receive calls immediately, messages often slip by without notification..).
I am running LineageOS 14.1 (Android 7.1.2) without any Google licensed apps.

Edit: added original link and also a correction: today even the Messenger works great for message notifications

+1, @DudleyDursley summarized it perfectly.

Only workaround with f-droid I know is using something like the https://delta.chat Email-Messenger, to ask each other to call back.

The situation could be improved of course:

And there are at least two ways to solve the push issue properly:

  • https://github.com/nextcloud/talk-android/issues/58#issuecomment-410278560 Proof-of-concept to test IMAP-push notifications.
  • Add a https-push "protocol" to the nextcloud server, let it just keep a websocket connection open over time, sending just enough keep-alive messages that NAT routers don't close it as a stalled connection (14 min.). Then nexcloud-apps and other app-servers could use the nextcloud server as a push server, and mobile apps could use a free push client library (e.g. https://gitlab.com/foss-push/planning/wikis/home) that implements a service that uses the required tricks to remain running in the background and would allow the apps to subscribe to nextcloud's push notifications.

In case you haven't seen it: Battery consumption in deltachat is down to 1% (seems less than what gcm would use if it were installed) and reliability is good since v0.18.2

@testbird, it's fine!
Is there any overhead of IMAP-push notifications compared to GCM?

Don't have any comparison, but the Email-chat spec that deltachat uses sure leaves room for some Email-notify spec that could minimize the overhead.

+1

I think they should have been upfront about the fdroid version not having notofications, I have been trying to solve the problem since a month... This is how Telegram F-droid seems to solve it:

screenshot_brave_20180812-080638

The app is kinda useless without notification.

We are VERY upfront about not supporting notifications if you install from F-Droid, in addition to me responding to queries and issues on all possible mediums (twitter, forums, github).

You're free to implement another service which the app will connect to, Nextcloud connector and a patch for the Android app which I'm happy to review and merge if it's good enough. But like the above post says (from Telegram) it's nowhere near anything I'd call good UX.

Thanks for the info mate. Notification support is written on the NCTalk page on F-Droid, I find it misleading...

screenshot_f-droid_20180813-120155

@Bubu can you pls check why was F-Droid README not updated despite me updating the fastline files? Do we maybe need one stable release before it happens?

Do we maybe need one stable release before it happens?

Yes. Not sure if that's intended behaviour on f-droid side but that's what the code currently does :woman_shrugging: .

Not wanting to add oil to the fire, but adding a +1. My hope was that nextcloud talk could replace telegram for some of my use cases but now that it doesn't have notifications for my setup I think i need to be a little more patient / implement it myself. 🙂
I like the way telegram handles notifications, sad that Google blocked that in 8.

This bothers me.

The Nextcloud ecosystem should not have any Google dependencies whatsoever.

If you are going to depend on features from the Google ecosystem then might as well not bother using Nextcloud at all in the first place.

So really the trichotomy is:

  • notifications without Google, or
  • no notifications at all; or
  • not part of Nextcloud.

I think there's no reason to not implement notifications without GCM.
(except maybe it's "easier" to just give up and use the google libraries)

Conversations (XMPP) does it, K-9 Mail (IMAP) does it.
Yes, I exclude them from the battery optimization, however they use almost no battery, especially when compared to the google bloatware,

Long lived connections with no traffic don't really drain the battery, so it should be simple to just implement a notification based on websocket.

https://en.wikipedia.org/wiki/WebSocket
https://stackoverflow.com/questions/12054412/are-websockets-adapted-to-very-long-lived-connections

This blog post on f-droid could be interesting.

@hoellen - wow

Sorry people, but I've written quite a few times why it's not as easy as you make it out to be.

I have personally invested both my own free time and time of Nextcloud (the company) to make Nc Talk available on F-Droid and so have the guys behind F-Droid (thanks @Bubu, a lot!!!). I would be the first one to implement notifications there if it was technically possible.

Sure, there are workarounds, but not in cases we need - near real-time call notification.

@mario
I've tested the F-Droid version again and at least the text chat seems to be able to poll the server for new messages. Would it be possible to do that on a semi-regular interval and produce a notification?

@mario, first a big thanks to you and everyone making it possible to have nc talk available in F-Droid.

While I would agree that it is not as easy as just including the google libraries (google likes to make it easy for developers to say "screw it" and to just use their closed-source stuff),
and that it might require more work than is currently available (after all, the focus might not be on those pesky f-droid users),
I have to strongly disagree on you that it would be not technically possible.
As mentioned earlier, all it would require is a long-lived connection (be it websockets or something else) through which the notification can be sent.
BTW, the same channel could also be used to get notified on file changes, not only on mobile but also on the desktop which would be a lot better than the 30 second polling that it currently used.

"All it would require" is a very bold statement given that Google and/or various OEM implementations would kill the background service doing this relatively often, even if you ignore battery optimisations (which is bad UX anyway).

(Btw. I'm one of the pesky F-Droid users :P)

@mario, basically google is trying to shit on all apps not using their libraries by saying it is in the name of battery optimization. These attacks on the open nature of AOSP were starting early:
https://arstechnica.com/gadgets/2018/07/googles-iron-grip-on-android-controlling-open-source-by-any-means-necessary/

At the moment it is still possible to do though (see K-9 mail, see conversations).
And regarding bad UX, having a small dialog pop up asking you to disable optimizations with an OK button just like conversations does shouldn't be too complicated for the average F-Droid users (Normal users will just use the playstore version anyways)
Yes, I know about some OEM implementation that are doing even more crap regarding killing background apps, but as mentioned there are two valid options

  1. giving up (and as mentioned, I fully understand if there are no resources for this on your side)
  2. trying (let's say if it works at least in lineageos then it should be good...)

Hm as nextcloud clients always have their own server (obviously) it might be legit to use it directly. ;-)
I mean even without putting the code into a shared library to collaborate on something quite useful like https://gitlab.com/foss-push/planning

See in this article
https://f-droid.org/en/2018/09/03/replacing-gcm-in-tutanota.html
that "tutanota" already implemented (stand-alone) an extension for the server side events (SSE) of the HTTP spec, to allow pushing events for multiple accounts over one connection.

SSE => https://en.wikipedia.org/wiki/Server-sent_events
https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

@mpfau @charlag @bedhub
A note concerning the tutanota gcm article:

  • A permanent notification would only need to be active while the screen is off (i.e. would never have to be visible to the user, see https://github.com/deltachat/deltachat-android)

Wikipedia points to this PHP library for Server Side Event support: https://github.com/hoaproject/Eventsource

@mario Do the recent Websocket-related commits have something to do with this?

No, sorry.

On Wed, 17 Oct 2018, 15:29 Laszlo Kertesz, notifications@github.com wrote:

@mario https://github.com/mario Do the recent Websocket-related commits
have something to do with this?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/talk-android/issues/257#issuecomment-430628527,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWshRG59GbuPhbx6bAPfdCQBMHPg7-ks5ulzDFgaJpZM4VcfIC
.

If you run Signal without Google Play Services installed, then Signal defaults to a "constant connection" and the device still receives timely notifications.

I believe this is a good example of an app which works both with and without Google Play Services.

I am commenting on this issue to point that out, as well as to express that this is heavily desired by fans and serious users of Nextcloud. For instance, in my association, we set out to self-host an F-Droid repo with the Nextcloud apps that we all use, but we have a hard policy of not distributing any closed software, which means we only are distributing the F-Droid build of Nextcloud Talk. Until we can receive notifications, this build of the app is all but useless to us. We are reduced to just using it for scheduled group meetings.

Another reason for google-independent notification

328

Sorry people, but I've written quite a few times why it's not as easy as you make it out to be.

What are the specific hurdles to overcome?

I don't know, if not collaborating with others to create a shared mobile push client (https://gitlab.com/foss-push/planning), existing server-side-events libraries for php/java/c and exiting code examples (see posts above) may ease integrating fully fledged push messaging in the nextcloud server, and the clients.

But maybe the current client-server connection can work as it is, even if it is not optimized for idling? At least in the beginning for experimenting?

The android client would need to emit a permanent notification with display-off events (and remove it with display-on events), to keep the app running permanently in the background (only listening on the idle connection), even when the phone is in the doze state. (Allows incoming calls to wake up the phone.)
For a working code example for that look in this pull request: https://github.com/deltachat/deltachat-android/pull/389

Adding the permanent running option should be independent from whether in the end the SSE http extension may be the better protocol for maintaining and reestablishing mobile connections on the fly.

Would it be a lot of effort to create a F-Droid repository for the Play Store version of Nextcloud Talk? Then I could get that version from F-Droid and use microG for notifications.

@qoheniac that I can look into eventually, but soon there won't be a need for that as Google is Open sourcing the Firebase library. :)

@qoheniac that I can look into eventually, but soon there won't be a need for that as Google is Open sourcing the Firebase library. :)

And this means that we will have to install the push server and an option in Nextcloud to use it?

I guess it means that including Firebase wouldn't hurt F-Droid's rules anymore so that you could then use FCM with the version downloaded from F-Droid just like now with the Play Store version. You would still rely on microG Service Core (or Google Play Services).

@gradinaruvasile no, what this means is that FDroid will be able to build a regular flavour of the Talk app and those with MicroG will have Push functionality even when app was installed from FDroid.

So @qoheniac was right :)

It is absolutely not easy to implement a GCM free solution but there are also success stories, see this report :

replacing gcm in tutanota

@cron0mat this was already mentioned by @testbird above in https://github.com/nextcloud/talk-android/issues/257#issuecomment-423903883 :)

Another issue apart from missing notifications is also apparently that after ~10 minutes, you are dropped from an ongoing call. It just happened to me twice directly after another.

@mario I know that this is extra work – Signal, Riot and Telegram all support this via a permanent notification. We are at a stage where we need all these early adopters and people from the free & open source software community, where we come from. :) Also, Nextcloud’s main premise is to provide a great alternative to Google services, so we shouldn’t rely on them.

Another issue apart from missing notifications is also apparently that after ~10 minutes, you are dropped from an ongoing call. It just happened to me twice directly after another.

That doesn't really sound like it's related to the no-push support. At least it really shouldn't be.

This will require the following changes:

  • support for the user to automatically whitelist the app as "Ignore battery optimizations" on F-Droid build
  • changes to the mobile app to support setting websocket server url manually
  • changes to the notification app to POST notifications to the websocket server instead of the push proxy
  • a websocket server (to be self-hosted as I'm not sure if we'll host this?) written in whatever, but probably Go

This ALL takes a significant amount of time. Happy to do it as soon as I get it, or someone volunteers to do it in which case I'm more than happy to mentor :)

cc @nickvergessen just so he's in the loop.

10 minute drop is not related to push at all.

This will require the following changes:

  • ...ignore battery optimizations...
  • ...configurable websocket server url...
  • ...relying on websocket server...
  • ...implementing websocket server...

Wasn't it the main advantage of using the HTTP standard's "server side events" (instead of relying on yet another centralized websocket server park) that it could be integrated and provided directly by the nextcloud webserver installations?

The "eventsource" PHP library for providing HTTP server side events or maybe the "Tutanota-extensions" may be able to turn a good part of the above requirements into "given"s.

See https://hoa-project.net/En/Literature/Hack/Eventsource.html
(and above https://github.com/nextcloud/talk-android/issues/257#issuecomment-423903883)

Please read above where I said it could be self-hosted.

I don't care about the exact implementation, WebSocket server was mentioned as a technical means to accomplish something. That being said, a WebSocket server would be infinitely more scalable.

But it all depends on what we need to accomplish. First we get the time and contributors, then we get into the technical nitty-gritty :)

But yea.. discussion for another day.

FWIW, why is that site claiming the Hoa library (containing "eventsource") is already "trusted by owncloud", anyone? https://hoa-project.net/En/

The API was part of ownCloud/Nextcloud but has been removed.

I'll be working on one solution for this issue through prototype fund, beginning in March: https://prototypefund.de/project/open-source-push-service-fuer-android-apps/

Finding a suitable technology/protocol for this will be part of the project.

But as there'll be a server and a client library part to this neither the nextcloud server nor the apps should care about the chosen implementation all that much.

Congratulations for getting the project @Bubu ! :+1:

Hooray, https://gitlab.com/foss-push/planning/wikis/home
A more universal solution, that is usefull for more than just a single application like talk-android, is of course even better!

as there'll be a server and a client library part to this neither the nextcloud sevrer nor the apps should care about the chosen implementation all that much.

Jep, and the abstraction also allows to support alternative protocols, for example to allow operation without having to setup any additional server by simply using basic e-mail accounts (imap servers), while still allowing, e.g. to upload and run a dedicated HTML-SSE based (php, rust, crystal whatever) app on a standard (web)server space, if higher messaging loads require separate servers and they are worth the extra maintenance effort.

Another big advantage is also that the decentral (mobile) clients can itself be responsible for the credentials, instead of the servers (i.e. the device itself can authenticate and allow others to wake up the device through allowed protocols; dedicated servers may proxy auth requests and filter messages, while imap could still also work based on plain store and forward operations).

Hello, in the meanwhile you could have a look at gotify project: https://gotify.net/
It is a SIMPLE, SELFHOSTED, FOSS, CROSSPLAFORM and BATTERY SAVING push service. (Brilliant for my private use!)
Maybe the nextcloud server could a) integrate gotify or b) use gotify push service as interface for push massages to talk app.
On my Android 8.1 Motorola i have not really any battery problems with gotify app.
Of course, a general, common used alternative like "foss-push" should be prefered, but this can take month/years.

Hello, what is the status of this topic please ?

Two solutions have been proposed here: FOSS-PUSH and GOTIFY.
It seems that Nextcloud-talk developers were informed about the FOSS-PUSH project a little over a year ago: https://gitlab.com/foss-push/planning/issues/2

Which one would you like to implement into Talk for F-Droid, and what could be the agenda ?

Thank you very much for your feedback.

None.
1) Foss-push seems to be not doing anything?
2) We're working on an alternative solution at the moment and you'll be notified as soon as it's ready.

It's a shame though that everybody seems to push push push, yet nobody wants to contribute :(

@Mario, all respect and credits given for this great app and for making it available on f-droid. It's sad no one wants to at least partly finance the development of this feature.

But I don't think that all of those manifesting the need for this feature are as well developers (or big companies with large budgets). At work we (some of us NC fans) already lost the challenge "nctalk vs telegram vs signal vs Teams vs whatever" because I couldn't bring in the big argument "fully works with notifications w/o dependencies to google services etc.". I had a small timeframe when evaluating that.

Hopefully one of your big customers will find usecases for this feature.

Anyway, again - thank you for your great work.

@mikelupe we're working on this regardless of the customer or not, we'll do this for the community - it's just taking a bit of time.

GCM/FCM is just XMPP. So I think we could reproduce whole process in Nextcloud by adding XMPP part to Nextcloud Server and to "Service" part of mobile app. We need to do what other FOSS Android XMPP Clients did.

I'll start to dig into it in free time, I'm also very interested in this feature. We could bring Nextcloud Cloud Messaging instead of Firebase Cloud Messaging, why not? :)

EDIT: I found out that was problem with my firmware update for Cat S61. Notifications didn't work for Nextcloud, Slack, and other important apps. Finally, after few adb logging, updating and several factory resets notification system went back to work :)

Hello the last answer makes me doubt if this issue is solved or not, and how.
@thecyberd3m0n : could you please tell me if you can have notifications on Talk Android version from F-Droid just by upgrading your phone firmware (or maybe the solution has been integrated into the app by the team) ?
Thank you !

I'm learned to use Google Play version, but I'll check it. It works now in Google Play version

I'm learned to use Google Play version, but I'll check it. It works now in Google Play version

OK, I see, thank you very much for your feedback ! So we're still at the starting point.

I really hope push notifications will be implemented in the F-Droid version some time soon.
But as @mario mentioned above, it seems that metadata is stored encrypted on the server running Nextcloud. If I now use the Play Store version with GCM, would I provide any metadata or other data to Google, and could they analyze it?

Yeah, well they know that something is pushed to that device. They also know which developer certificate is allowed to push to that push token.

Hello,
I have a little feedback from me.

I use pihole and I have blacklist mtalk.google.com + alt*.mtalk.google.com => I blocked all notifications in nextcloud talk... :/

I read : https://github.com/nextcloud/talk-android/issues/497 and anothers issues with notifications

Ok, no "data" (text and co) are send "directly" because all are encrypted.

I search on my privacy app and ding ding => fairemail 👍
On FAQ :
https://github.com/M66B/FairEmail/blob/master/FAQ.md

Some people suggested to use Firebase Cloud Messaging (FCM) [...]. The first is not going to happen and the last would have significant privacy implications.

I have delinked all dependency on GAFAM, oooh not, FCM on apps :/

This comment is for add positive encouragement to migrate FCM to privacy notification :)

Have Fun, Good Day at All ;)
Eric

Edit : add a beautifull fork on signal app :
https://langis.cloudfrancois.fr/
;)

@mario Are you aware of the project OpenPush? As far as I understood it is still not completely functional, yet it seems interesting.

You can find further info here: http://bubu1.eu/openpush/

This is meant just as a support for your activity. HTH

@Spartachetto I know @Bubu :)

Hello. Now that Nextcloud 17 has become Nextcloud Hub, and that my last reminder was done around July 2019, I would like to re-ask again if there is finally an alternative to GCM please?
Thank you

@mario, there were news about OpenPush at the FOSDEM 2020

vid: https://ftp.fau.de/fosdem/2020/UA2.220/dip_openpush.webm
pdf: https://bubu1.eu/openpush/openpush_fosdem20.pdf

Looks promising to me!

Someone should get ahold of Huawei :). Heard they are working on a google play alternative and will need push notifications for all their apps... but in all seriousness, great work @mario and @Bubu. I can’t image trying to start from the ground and do what google probably had a whole team and a massive budget to do. push notifications have never worked well for me on android or iPhone and I love it. Because Nextcloud and you guys are awesome!

@mario, there were news about OpenPush at the FOSDEM 2020

vid: https://ftp.fau.de/fosdem/2020/UA2.220/dip_openpush.webm
pdf: https://bubu1.eu/openpush/openpush_fosdem20.pdf

Looks promising to me!

Yes and F-Droid itself is promoting this solution mentioning Nextcloud Talk : https://f-droid.org/en/2020/02/03/openpush-talk.html

Linked to https://github.com/nextcloud/talk-android/issues/257#issuecomment-467889135

For anyone looking for a quick fix, I wrote a python script, that fetches messages through the talk api and send them to a gotify server. It is very rudimentary, but works for me. It still doesn't handle calls tough. If anyone wants to have a look, you can find it here:
https://github.com/mrchainman/Talkify

It's a shame though that everybody seems to push push push, yet nobody wants to contribute :(

As you can see, this is not only about _pushing_. People here (and I really thank them) are trying to help : they propose some ways to progress. Here, @mrchainman has shown how it was possible to implement a FOSS notification program (he even propose his code) which works on Talk. His solution is a response to a proposal from exactly 1 year ago, totally ignored.

I hope "Talkify" will be taken into account, at least for chat notifications.

Hi there!

  1. Foss-push seems to be not doing anything?

Why that? Thought you were aware that @Bubu implemented "openpush" https://f-droid.org/en/2020/02/03/openpush-talk.html

Though, it seems to require that its own push server implementation is running somewhere, and is not (yet?) able to make use of arbitrary IMAP or XMPP accounts from existing providers as transport protocol.

  1. We're working on an alternative solution at the moment and you'll be notified as soon as it's ready.

Cool, for example, with something like an HTML 5 Server Side Events (SSE) queue in Nextcloud, Nextcloud could also become an alternative transport protocol for OpenPush. :-)

Hi there!

  1. Foss-push seems to be not doing anything?

Why that? Thought you were aware that @Bubu implemented "openpush" https://f-droid.org/en/2020/02/03/openpush-talk.html

In his defense, his commentary was written last year, so before the work of @Bubu.

The real problem here is that:

  • the text/video notification request on F-Droid version was issued almost 2 years ago,
  • for the last 2 years, we have been told that it is too complicated, that we have to go through the Play Store version to have this functionality
  • then we are told that we are being too insistent without helping the project, while leaving the hope that a solution is being deployed,
  • but almost a year later, there's no more official news.

I'm starting to think that the real problem is a Nextcloud business model problem: to satisfy our demand (a basic messaging functionality), we have to use Google Cloud Messaging. During the 2 years we're negotiating here, Google is still receiving metadata of every chat from almost every Talk user in the world.

In this context of Covid-19, I am forced to go through Viber to talk to my close family who is stranded abroad. Because the choice of video-call platform is usually the easiest one to use. And that's all I was afraid of: knowing that important exchanges with my family are recorded on the server of a private company. I relied a lot on Talk, but I can't tell my family to open the application every minute to check if I am calling them.

Anyway. Here's another _"pusher"_ that does a great job of integrating Jitsi into the Talk application (see also his demo video). Combined with Talkify seen above, we can hope to see an interesting alternative to our problem.

It sounds like trolling, but let me rectify this thought: at this point in the post, it's April 2020, this is no troll, just real disappointing fact.

In his defense, his commentary was written last year, so before the work of @Bubu.

Thank you for dropping a note.

I am sorry, yes, at that time @Bubu had still only announced here, that he got funded in an older message.

Looking forward, though, my guess would be that a solution might be available in this direction:

If Bubu's OpenPush (mobile) Pushclient code could work with a push-message queue that gets implemented in the Nextcloud server (like HTML 5 Server Side Events (SSE) or similar). Then Nextcloud-talk can benefit from using the shared OpenPush Pushclient code.
And it would already provide a complete, self-hostable solution for Nextcloud-talk.

In a further step, the OpenPush Pusherver code could provide some sort of "library-only" mode, that does not start its own server process, but helps non-Nextcloud Apps and App-Servers to register/add messages to a Nextcloud server's push-message queue. This lib feature could allow other Apps to work with self-hostable push messaging based on Nextcloud, and other OpenPush supported transports.

So easy to forget... remember push could also work without first needing any Nextcloud (server) support for queuing etc.

Similar to Nextcloud's two-factor-authentication method "notification":

Simply send out an email that contains a URL/link/token that is registered with the local nextcloud-talk app. Emails can be received by any mobile email-app that the user has already installed right away.

All the advanced things, like autocrypt, automatic filtering, etc. can come by composing correspondingly formatted emails, for which already existing tools and scripts can be used: https://autocrypt.org/dev-status.html These include mobile email (-messenger) clients that already fully support receiving IMAP-push notifications even during suspend.

EDIT: i.e. even existing, independent email-apps that can wake up the phone by keeping only a single connection to the IMAP server, thus building upon the IMAP protocol's own direct push capability.

Very strange problem, relly don't see whay nextcloud talk can't check how resolve this problem for example like Telegram.

@mario Is there anything the community can do to help with the solution that's being worked on already? Is there a repo people can contribute to? Would funding a bounty (even one with a large goal) help? Would it help if someone looked into OpenPush further? Is there something else the community could do to help?

Maybe we can take some of the pent up demand for this feature from the community and channel it into something that helps you out. ;)

hmm. Is it not possible to "simply" implement a automatic pull? i can do it manually in the app - just swipe down.
and it is not possible to do this in a configured interval?

i got most of the poins here. battery savings, its not 'instant' etc. pp.

but i personaly dont care. i just want to know what is going on - and i can live with that 'downsides'.
if i missed an call the participant have to whait at least maybe 10 min or so and i will call back.

if i just would get any kind of notification...

By now, XMPP/jabber clients (conversations.im or the improved pix-art messenger) from f-droid properly support audio and video calls with call-notifications.

With https://quicksy.im even using phone number addressing.

Simplified self-hosting is possible, for example with https://yunohost.org or a freedombox (implementation not finished yet).

@testbird please focus on the topic. XMPP is not related to this request, beside that all clients mentioned do support notification without gms

I just thought I'd add my 2 cents to this "issue".

So if I correctly understand the notifications flow from Nextcloud to the Talk application, it goes like this;

Nextcloud --> Notifications* --> FCM <-(p)-> play services --> NC Talk --> android notification

So assuming that that is correct, the biggest part of the server-side infrastructure is already implemented in the form of Nextcloud Notifications. The missing piece is, of course, the ability to feed information into a persistent connection for a mobile (or non-mobile) device (i.e. withOUT using the FCM <--> play services path), however that is little more than trivial when taking the EventSource/SSE approach, which at the simplest level amounts to a php script just setting the Content-Type header to "text/event-stream" and feeding out notifications as they arrive.

The client side is also fairly easy, although IMO should not be implemented in Talk, but rather a Nextcloud Notifications service package or as part of the main Nextcloud (files) application, which can delegate notification data to other applications as applicable. This is done by grabbing your choice of SSE library, such as https://github.com/biowink/oksse and setting up a connection in a loop. When a message comes in, it executes a parser and delegates the data.

I think that by-and-large, the foreground service / persistent notification approach to keeping the service alive, and disabling battery optimizations to keep it communicating are acceptable, if not pretty. For building it into a system image, the same approach that google uses can be applied to avoid the uglyness.

And also obvious (hopefully), there should be a bit of housecleaning implemented in the process.

A real easy to follow server-side example can be found here;
https://www.w3schools.com/html/html5_serversentevents.asp

<?php
header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');

$time = date('r');
echo "data: The server time is: {$time}\n\n";
flush();
?>

Imagine sticking those last three lines in a loop with a 1 minute pause between iterations. That would make it push a time update to the client once every minute for as long as the connection remains open.

Would be really great to have this implemented!
I'm using LineageOS and I'm glad of using no Google-Services. It would be a good compromise to have a lower battery runtime like in other Apps.
Please add it because without Push-Notification Talk isn't useable.

Still waiting for this feature. Without it the APP is mostly useless for many people. And thats a pitty as the APP works like a charm when you are lucky and can catch a call. When people go the extra mile to run there own cloud infrastructure they perhaps don't like there communications metadata delivered to Google.

Wow, did not know this issue holds for 3 years already

Recently started searching for nexloud subscription providers but realized notifications do not work w/o google apps

Significant amount of users consider nexcloud talk as a private alternative to other messengers. Thus avoiding google apps is a natural requirement

Would be happy to donate (pretty sure same for others in this old thread) to help bring notifications w/o google cloud messaging

I'm beginning a project to implement a non-google dependent notification system. Its not going to become part of the nextcloud talk application, but should at least be good enough that messages and notifications won't be missed. This is going to involve an android application/service that registers for push messages and generates appropriate notifications for the user. When this application/service receives a notification, it will launch the applicable nextcloud application (i.e. talk) on click.

The main obstacle for google-free push notifications is the fact that there is no mechanism within nextcloud for subscribing to notifications. The nextcloud web-application POLLS the server, and push messages are sent to a nextcloud-owned server running closed source software that redirects messages to google. So I guess when nextcloud project participants talk about increasing battery drain, they're referring to the need to quickly poll the server in order to check for high priority notifications like an incoming call.

However, there is infrastructure in place to allow nextcloud to connect up to other "push proxy's". This means that you can create your own push proxy to receive notifications from nextcloud, to which clients can connect persistently and wait for messages to be pushed using, for example, html5 server sent events.

The nextcloud documentation regarding this capability is not the most clear, but I have managed to get a proof of concept working using basic commandline tools like 'nc' and 'curl'. With curl, I can create a push registration with the nextcloud server, which calls back to nc listening on a port as specified, so I am able to both register for, and receive pushed messages from nextcloud without any involvement from google or nextcloud-owned servers -- the push message comes straight from my nextcloud instance to nc.

This is where I'm working; https://gitlab.com/Nextcloud-Push
There are currently 2 projects, one is for the push proxy, and the other for the notification client service. Both projects have a wiki with a single page and not yet anything more. I will be adding to both projects as I build it out.

https://gitlab.com/Nextcloud-Push/direct-push-proxy/-/wikis/home
https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier/-/wikis/home

Edit: apparently I neglected to set the first of those two projects to public, its rectified now.

@ASerbinski Working with common lib like https://f-droid.org/en/2020/02/03/openpush-talk.html ?

@ASerbinski Working with common lib like https://f-droid.org/en/2020/02/03/openpush-talk.html

No, definitely not.

No, definitely not.

Why not? Just curious.

Your independent client application also sounds a lot like Gotify (https://gotify.net/) - just wondering if you could save yourself a lot of work and build on Gotify. (Users who already have Gotify set up would also benefit from reusing that existing connection instead of creating another one to drain the battery.)

Because I don't feel like creating more work for myself and adding unnecessary and likely unreliable dependencies.
Gotify looks interesting, but it isn't useful because it is still necessary to write the receiver for messages from nextcloud. By far the biggest part of the project is just dealing with Nextcloud's security, the actual message sending is trivial. That and the fact that it is crazy to deploy an additional server with its own bugs and security holes listening on its own port to accomplish something that you can write in 20 lines of php on an existing deployment of apache or whatever web server you're running. And for that matter, even the gotify client application would not suffice, since the client still needs to authenticate with the nextcloud server and register with it for push messages.

As far as openpush goes, the project's website is really not inspiring. Especially the first line that includes the words "aims to create a", which means that he hasn't yet and maybe never will. Its a non-starter.

So, believe it or not, I already have some really good progress.

Wiki page for apache configurations;
https://gitlab.com/Nextcloud-Push/direct-push-proxy/-/wikis/Apache-configuration

And added a couple of files to the server repository;
https://gitlab.com/Nextcloud-Push/direct-push-proxy

Obviously this isn't "production ready" yet, but you have to start somewhere, right?

What it does right now (in a whopping 34 lines of code!!);
1) It can be registered for and receive pushed messages from nextcloud,
2) It can relay pushed messages to anything that connects for SSE.

What it doesn't do;
1) It does NOT authenticate the user (can be htpasswd for now just to keep baddies out),
2) It does NOT yet support multiple client connections.

It will actually be pretty easy to handle multiple client connections; I'm using a message queue for communication between the backend (push receiver) and frontend (client connections) with a key generated using ftok of the backend php file -- obviously one file means one key, one message queue, and therefore only one client. In order to receive messages, the client has to do two things; (1) register with nextcloud, (2) connect to the SSE frontend. Nextcloud will supply details from (1) to the backend, the client will supply the same details during connect (2) to the SSE frontend, therefore we can generate an appropriate key to be used to communicate from the backend to the frontend.

So next steps;
Backend line 3, frontend line 8: Instead of ftok, parse $data/$_POST and calculate a suitable key. That should be all that's needed for handling multiple clients, then its on to the Android client.

@ASerbinski Sounds like good progress.

If the existing android open push client code is decent enough (also @Bubu) what do you think about using that client to make it register for SSE?

By now there seem to exist several protocol clients in F-Droid: https://search.f-droid.org/?q=push&lang=en

@ASerbinski Sounds like good progress.

If the existing android open push client code is decent enough (also @Bubu) what do you think about using that client to make it register for SSE?

Stop suggesting that. I've already explained that I will not be using it.

Just got into the office, and good news. I've had curl connected to my SSE front end all night long with no data transferred on the connection, submitted a test message on nextcloud, and it instantly pushed out of curl. My nextcloud instance is running in a datacenter located in a different city, so the test isn't tainted by having a super reliable local connection.

The code has now grown to 58 lines and supports multiple clients.
Hash collisions (i.e., two clients trying to use the same IPC key) are... not impossible, but not a security problem, since the payload is encrypted with the recipient's public key.

Time to move on to the Android side of things.

Nevermind, that f-droid search actually deos not list an open-push client, only others that went through the build and publish thing already.

Strange, now unifiedpush.org seems gone from the f-droid listing, maybe separate repository instead of project repository builds, now.

@testbird: please stop already. I don't know why you are so fixated on the idea of using some pointless 3rd party server software and framework to do something that is very basic and already built in to every web server and browser.

Let me spell this out as plainly as I can for you: the entire PURPOSE of SSE is to drive messages from server to client. SSE is supported by ALL major web servers, browsers, and supporting libraries. It is a fundamental component of HTML5. The ONLY complicated part of this project is to interface with Nextcloud -- the message passing is TRIVIAL.

Mainly agree with your explanations. Was not only thinking of the code, but mainly what you can accomplish on the client side.

If I understand correctly, the unifiedpush.org client lib does not have to require third-party servers. It allows the app user to configure one of the supported alternative services, and the non-free version currently automatically falls directly back to FCM. But with SSE support in the unifiedpush library, it could first fall back to using SSE if available.

=> It can thus provide a simple migration path for different app versions out there that allows for seamless introduction of direct SSE push, even for the apps that (still have to) support their non-free version!

That's why I'd guess your SSE proof-of-concept could go way beyond android and nextcloud, with SSE client support in something like a unifiedpush library (beyond nextcloud, beyond android). HTML5 is kind of a destined default. A multi-platform migration path that's build-in makes it more likely to be adopted.

That's more reasonable, but like I said, the issue isn't making the SSE connection or displaying notifications, which are the limits of a "unified" approach. If that's all there was to it, I'd have had it done in 10 minutes. The issue is in negotiating with the Nextcloud server so that it sends messages to the right location, and in interpreting the messages received from Nextcloud.

On first connection, we need to make a special log-in to generate an application-specific password and receive the user-public-key from the server. On subsequent logins, we need to first register the push proxy to the nextcloud server by logging in with the previously generated application-specific password and feeding it a unique device identifier and the device-public-key. THEN we establish a connection to the SSE frontend while providing the identifiers needed by the frontend to make it possible for the backend to pass messages to it and start receiving messages at the client. The messages we receive are encrypted using the device-public-key and signed with the user-private-key. We then must decrypt the message using the device-private-key and authenticate it with the user-public-key. Then finally, we must interpret the message and deal with it appropriately.

That all has to be implemented specifically for Nextcloud at the client.

The way forward is this; integrate something functionally similar into nextcloud (server) itself. On the client side, I am convinced that the ideal place to end up is for there to be a specialized push receiver available as a separate package, however, it isn't ideal for the notifications to be delivered by the push receiver package, rather for messages to be dispatched from it to the specific applications to generate their own notifications.

So lets look at "unified push":
http://unifiedpush.org/developers/intro/

From what I can tell, it doesn't actually do anything useful at all.
It just breaks your work into two parts (two apk's) to make it confusing to the user, forces you into dependency on their library, and increases the development time by forcing you to learn their library.

So now instead of writing an Android service that establishes a connection to a server to receive messages and loads notifications when they arrive, you now write an Android service that establishes a connection to a server to receive messages which transfers them through openpush to an Android RECEIVER that loads notifications. So more work, more points of failure, and for what advantage?

the issue is in negotiating with the Nextcloud server so that it sends messages to the right location, and in interpreting the messages received from Nextcloud.

Can't help but somehow that sounds a bit backwards to me. Much more like you're describing the creation of a third party (proxy/push) server. Instead of allowing client apps (eg.nextcloud) to re-use the session for direct SSE to their app-server just to get the wake-up message, instead of a legacy third-party push service. Every app already needs to do the hand-shaking with its application server (eg. the nextcloud server), and as http web-app itself, is can also handle SSE. Actually nextcloud already used to include a SSE capable php lib.

So I had pictured SSE subscription support to go into nextcloud server (using the same credentials). And new client app versions could start using SSE directly. However, the problem being the missing SSE support in the stock releases. So switching out the client-side push-library, and possibly a (your) temporary "third-party" proxy, could possibly be a viable way to allow building working apps.

Let me add, it could perfectly be that I have not understood you or that unifiedpush very well.

An advantage I do see that a lib could have, though, is that there may be some things necessary to do, to prevent the listening connection from getting terminated by the OS (or auto re-initiating it) that could be useful for different apps.

So to summarize:
There are a few options to make push "work".

Option 1 would be to do everything "yourself", using SSE or whatever within the Nextcloud app.
Upside: quick to implement.
Downside: Nextcloud needs to be running all the time which will probably not be the case under memory pressure etc..

Option 2 would be to use something like openpush which would be a separate library basically like FCM but where you can specify your own push server which in turn just wakes up Nextcloud if a message is pending at the nextcloud server.
Upside: multiple apps can use it, might mean that it is less likely to be evicted from memory.
Downside: openpush does not yet seem to be fully "ready"...

Option 3 stuff like unifiedpush which if i understand it correctly requires a bit more configuration on the unifiedpush app while the openpush approach would be a universal one where the app itself basically tells the pushserver who can send push messages etc or did I misunderstand that?

My preference would definitely be something like Option 2 but I guess anything would be better than nothing :D

You are right, but that is how Nextcloud is implemented, and is therefore the set of restrictions to which we are limited. The ideal future will be to have this (or at least the concept of this) integrated into Nextcloud server. It would make things a heck of a lot easier by eliminating the need to establish a separate connection, or to have the separate level of message encryption that is used to protect the messages from the proxy (SSL on HTTPS would suffice).

Open push does not manage the connection at all. The connection is managed by what they refer to as a "distributor", which you need to implement yourself. All that open push is, is a library that manages the message passing between the distributor application and your actual application -- both of which are up to you to implement.

So to summarize:
There are a few options to make push "work".

To clarify, the issue with things being killed under memory pressure applies equally to all three options. There are a few tricks to dealing with that issue;
1) Disable battery optimization,
2) Persistent notification (foreground service),
3) Install as a system application.

I'd imagine that for most people aiming to obtain push without GCM/FCM, its probably not too big of a problem to make system level changes. The goal, for most, would be to eliminate gservices altogether. Although I imagine that there are also those who are just getting sick of the fact that talk notifications are unlikely to be delivered even if you have the playstore/fcm version. Examining my database right now, I see only one push registration, and its one I created for testing my proxy -- there should be at least 10 registrations. MicroG on my phone claims that it is registered. And its not just Android phones in that expectation -- 2 are apple. The genuine fcm registrations do appear from time to time, but only briefly.

For an additional clarification, SSE can apply to any of the three options, depending on how it is implemented.

Option 1 would be to do everything "yourself", using SSE or whatever within the Nextcloud app.
Upside: quick to implement.
Downside: Nextcloud needs to be running all the time which will probably not be the case under memory pressure etc..

There are multiple nextcloud applications, which would benefit from a certain level of coordination. Not a small job here, and not something I'm willing to approach. If somebody wants to take on the job of bringing this into nextcloud applications, then they should feel free.

Option 2 would be to use something like openpush which would be a separate library basically like FCM but where you can specify your own push server which in turn just wakes up Nextcloud if a message is pending at the nextcloud server.
Upside: multiple apps can use it, might mean that it is less likely to be evicted from memory.
Downside: openpush does not yet seem to be fully "ready"...

This is more or less what I'm doing, except a lot more efficiently by not introducing an unnecessary server and non-standardized protocol in the middle of it. Openpush itself is basically just trying to implement its own version of SSE.

Option 3 stuff like unifiedpush which if i understand it correctly requires a bit more configuration on the unifiedpush app while the openpush approach would be a universal one where the app itself basically tells the pushserver who can send push messages etc or did I misunderstand that?

There is no "unified push app". Its just a library that you build in to your own program in order to make unified push's author feel good.

And its not so much that it requires more configuration, its more that it doesn't do anything. You still have to write a service that opens a persistent connection to your server and takes an action when a message is received -- i.e., you still implement and use SSE. The only difference is that instead of taking the objective action (like creating a notification), you have to take the action of converting the message into open push and sending it up. You then have to create another application (receiver) to receive the message from unified push and take the objective action. Its hard to grasp the limitations, because it sounds like its describing a client/server infrastructure, except that its more like this picture;

-------------------------
|     Android           |
|     -------           |
|                       |
|     Service <----------------> Internet <-------> Server (data source)
|        ^              |
|        | unified push |
|        v              |
|     Receiver          |
|   (notification)      |
-------------------------

With that picture, I hope you can grasp just how pointless it is. The message has already been received by your phone before unified push even comes into play. All it is, is a wrapper around IPC. This kind of makes sense from the point of view of having more than one receiver for dispatching messages to more than one application (i.e., one in nextcloud files and one in nextcloud talk), but its such a trivial task that you are much better off just using the IPC directly instead of relying on a 3rd party library.

Thanks @github-k8n and @ASerbinski for your write-ups about the issue!

@p1gp1g could you explain if people understood you correctly here, and what your design goals are with unifiedpush?

Ideally, I think we'd like something that is able to make stock apps work through a custom push connection. Either by using a custom proxy-server, or much better direct connection to the app-server, by using or adding support for HTML5-SSE connections to the app-server. (The latter ideally working without any configuration need on the client, by letting it prefer the direct connection before any legacy push service fallback).

Hey!

So unifiedpush is specifications and libraries giving a simple interface* devs can use to get notifications. Applications doesn't look how notifications are transported, the user choose that. < that's the main idea

To get notifications, users need to install a distributor which will handle the connection with a server to receive notifications. (That's a bit how FCM notifications works today, you have GServices, the "distributor", connected to a google server. With unifiedpush users choose both.)

Today, unifiedpush users maily use gotify (it needs the unifiedpush branch to work atm), but some uses FCM (google) with the UP-FCM distributor (which will be soon on play store) and others use noProvider2Push (which needs a way to get a static IP - reachable with a vpn).

I was waiting to get some free time to write or to ask for a nextcloud distributor ! Since there are a lot of persons self hosting, it would be a first place for a distributor. It should be very simple to do, most of the code from noProvider2Push can be used for the android app.

Another thing is : it helps a lot devs who want to do gplay/fdroid flavours. It only needs to declare an embedding distributor. See how fedilab did it. That's a good illustration, a few lines diff : https://framagit.org/tom79/fedilab/-/tree/develop/app/src/playstore/java/app/fedilab/android/services

The gplay flavour use FCM by default. If the user have another distributor on the phone, it will ask which way the user want the notifications to get delivered.

Moving from FCM to unifiedpush isn't that hard (and is invisible for users using FCM). For instance, I've started to write a PR for element (https://github.com/vector-im/element-android/pull/2993), it still need the fallback to the background connection for fdroid flavor when there is no distrib. I don't know if it will be accepted, but I'm using it :)

If you want some words from devs who used it the fluffychat team speak a bit about this here: https://matrix.org/blog/2021/02/05/this-week-in-matrix-2021-02-05 (the video) and fedilab wanted to do something but I don't know think it is done yet.

*In addition to 3 functions to register the application, the simple interface is :

val handler = object: MessagingReceiverHandler{
    override fun onMessage(context: Context?, message: String, instance: String) {
        // Called when a new message is received. The String contains the full POST body of the push message
    }

    override fun onNewEndpoint(context: Context?, endpoint: String, instance: String) {
        // Called when a new endpoint be used for sending push messages
    }

    override fun onRegistrationFailed(context: Context?, instance: String) {
        // called when the registration is not possible, eg. no network
    }

    override fun onRegistrationRefused(context: Context?, instance: String) {
        // called when the registration is refused, eg. an application with the same Id and another token is registered
    }

    override fun onUnregistered(context: Context?, instance: String){
        // called when this application is unregistered from receiving push messages
    }
}

PS: a word about the battery, it increase a lot compared to background sync. Fedilab did a background connection and for the same period fedilab dev said he have seen its app consumption from ~30% to ~5%.

Also, the objective is to have only one application on the phone constantly connected to a server to get notifications to all other apps, just like FCM.
-> every app doesn't have to implement a new connection to a server
-> every devs doesn't need a server (to host push gateways as today for FCM)
-> it reduces the number of connection and saves battery.

As @ASerbinski pointed, it wouldn't make sense to use unifiedpush if you have only one app on your phone or you don't care that there is at least one persistent connection per application which needs notifications + disabling battery optimisations on all of them (so android don't kill it - not true for all vendors), and you have enough time to write its own connection + the ability to have a server. It can however be considered if they are not a lot

@github-k8n

basically like FCM but where you can specify your own push server which in turn just wakes up Nextcloud if a message is pending at the nextcloud server.

Your option 2 nearly resumes what unifiedpush is.

Thanks for responding @p1gp1g, so I see some similar parts as in https://gitlab.com/foss-push/planning/-/wikis/home but find the talk about installing different distributors instead of protocol backend support confusing.

The most interesting messaging backends would of course be those that don't require any special (user) setup, i.e. SSE implemented in the app-server or using the users existing imap-push (email) account.

Thanks @p1gp1g
I'm not sure what you know about Nextcloud, but it fills a pretty big place with a high priority. Think of it as a self-hosted total replacement for Google services. Its also fairly easy for any other data sources to feed their notification data to Nextcloud (there are tons of 3rd party applications for Nextcloud, many of which issue their own notifications!). Consequently, I consider Nextcloud to really be the only necessary channel for notifications in much the same way that Goog considers FCM as the only needed channel.

Anyhow, @everyone.... I've got something to share now on the Android side of things;
https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier

There is still some work to do before its ready for productive use. Currently, it is just a single activity (not even a service!).
On startup (every startup....), it generates a public/private key pair and a push token hash, it then prompts the user to select a nextcloud account and authorize its use, makes an API request to the nextcloud server to register for push messages, and connects to the SSE endpoint.

When a message is received, it decrypts the message and loads a notification. Not a bad start, huh?

This application obviously depends on the Nextcloud (files) application being installed, since its using SSO.

TO DO:

  • Obviously, a lot of that work its doing on every startup needs to be saved and reloaded.
  • Move most of everything to a service.
  • Service [re]start on boot complete, 10 minute timer, network change.
  • Persistent notification.
  • Secure the SSE endpoint (technically, baddies can't do anything besides waste system resources by holding a connection open -- they won't get any data from it).
  • A bit of work on the notification, like launch the applicable nextcloud application on touch, grouping, etc.
  • Signature check on received messages.
  • Load previously existing notifications through api request.

@ASerbinski

Think of it as a self-hosted total replacement for Google services.

I'm using nextcloud for quite a while, and I see it that way too. That's why I'd like to have a nextcloud distributor.

Anyhow, @ everyone.... I've got something to share now on the Android side of things;
https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier

And I've seen that, I wanted to look deeper in it (I haven't even looked, very busy right now). Actually adding (and using) those 2 classes should probably make it a distributor for unifiedpush :
https://github.com/NoProvider2Push/android/blob/main/app/src/main/java/org/unifiedpush/distributor/noprovider2push/services/RegisterBroadcastReceiver.kt
https://github.com/NoProvider2Push/android/blob/main/app/src/main/java/org/unifiedpush/distributor/noprovider2push/services/PushNotification.kt

It doesn't mean that the app have to be only a unifiedpush distributor: see gotify which does it aside its other functions.

We can do a test when it can get notifications from the servers ! The only thing needed is an endpoint to send the message as a raw POST body.

@testbird
the "foss push" you mentioned deals with "the server to phone app" part. it doesn't handle the same part. (It means it can also be a distributor.) [edit: Actually I've read a bit fast, they want to implement many server to phone protocol, so that's like a big distributor, unifiedpush is more how this kind of app is connected to other apps]

I did this some time ago :
diagram

Push gateway: to translate application notification to a raw POST body, optional : depends on the application.
Rewrite proxy: to translate the raw POST body to the distributor-server (also named provider), optional : depends on the distrib
Distributor: the application which receive the notification from the server and forward to other apps

This is the path a notification does. The square is up to the user, it can be (gotify-server -> gotify-android), (google servers -> GServices), (nextcloud-server -> nextcloud-android) (? :) ), or anything else !

Since microg is able to support the platform's IPC standard (ie. wake up unmodified apps), wouldn't that actually be the best target for patches to support alternative channels/protocols (nextcloud-sse, imap-push, gotify, ssh pipe, ...)?

@testbird
From my point of view, microg is a nice "hack". I actually use it myself, so I'm not saying anything bad about it.
However, if the end goal is to rid ourselves of big brother's watchful spying altogether, then even microg needs to go. Getting push notifications working on nextcloud is one very major step towards that specific objective.

Microg is a re-implementation of (part of) the GService but it is still connected to the google servers. It means with microg we now know what data is taken and how. It is still better than GService but not enough :)

Patching would say we follow partly implementation dicted by google needs not ours.

I don't know what exactly do you mean with "wake up unmodified apps". There are some ways to wake up apps on android, some of them require a system application, which make the solution unreachable for users who do not use a custom ROM / root their device. It is also possible to wake apps with service intents and broadcast receivers (so apps needs to declare it)

Was thinking of an f-droid variant of microg, with all the gservice stripped out, only supporting the alternative channels there.

I'm not sure if there is much worth to do with the stock images. I had just assumed no, and to be ok with an f-droid compliant microg variant on that platform. It may likely only really go away with the new alternative platforms support for different channels.

Ah, get it, that binary-compatibility would hinder the open-source advantage of portability to all the different platforms.

I consider Nextcloud to really be the only necessary channel for notifications in much the same way that Goog considers FCM as the only needed channel.

Since there are also use cases like vpn usage, ssh sessions, and ubiquitous imap-push, that would not need that, how would you picture push wake-ups with an ideal platform or portable lib?

I don't know if you want to use UP, but if you do, a good way to implement receiving app would be:

  1. if the nextcloud distrib is installed, use it,
  2. if only one distrib is installed, use it (that can include embedded one with fcm on gplay flavor)
  3. if more than one : ask which one to use
  4. else : point to the nextcloud distrib to install.

That way :

  • it is invisible for the large majority of users since they will most probably have the nextcloud distrib installed or they use the gplay flavor
  • it is possible to get notifications with another provider for fdroid flavor before the nextcloud provider is written (using the lib is fast forward)
  • other apps can use the future nextcloud provider (<-- once again, I'd really like that :) )
  • if someone want to use another way to get notifications it works (which is a bit strange in the context of a nextcloud application)

Since there are also use cases like vpn usage, ssh sessions, and ubiquitous imap-push, that would not need that, how would you picture push wake-ups with an ideal platform or portable lib?

Just for your information, (I'm not sure you know) we indeed wake up apps (and it gave hard times for flutter apps ^^')

@testbird Push wake-ups, in terms of what? Having applications respond, or bringing/keeping the CPU online?

The networking stack brings the CPU online when data arrives, and it coordinates up the stack until the callback in the connection-holding application is run. From there, you likely need to dispatch some action, which will be asynchronous to the networking application, therefore when the callback is terminated, the CPU may go to sleep without the final objective being reached.

So for example, if you look at the "onMessage" SSE callback here; https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier/-/blob/be9d3b168fbfb42f6fe55d72aa9d09b58919fb96/app/src/main/java/tk/aprax/pushreceiver/MainActivity.java#L306 -- I call a notification at line 330, but because it is asynchronous, even though the message was received by the callback, the notification won't fire unless the CPU is awake for some time AFTER the callback is terminated.

For that reason, I've added a 5 second wakelock at line 307 -- it keeps the CPU awake plenty long to ensure that the notification fires.

Now if you're talking about doing something to trigger an action in another application, the other application has to implement some means of being triggered. Android has something called a RECEIVER, which is roughly analogous to an interrupt service routine. It is a special function in an application which is registered with the system such that it runs under certain circumstances. So if you fire off a broadcast (interrupt) in one application and have a receiver (ISR) registered for it in another, then the system will fire off the receiver (ISR) and pass in the broadcast data (parameter). From that receiver, the target application can take whatever action is appropriate to react to that broadcast.

And so the overall process works like this;

1) Network data arrives and wakes up the CPU, calls the networking callback,
2) The callback takes a short term wakelock and fires off an asynchronous response that is expected to be completed within the timeout period,
3) Some other process is triggered to complete handling of the data received by the callback.
4) Before the original wakelock expires, the triggered process takes its own wakelock in order to ensure complete processing before the CPU resumes sleep.

Since microg is able to support the platform's IPC standard (ie. wake up unmodified apps)

Ah. That's what is being referred to. There is nothing special about what MicroG does in this regard (nor, for that matter, what gservices does). Same as I explained right above. Network, wakelock, intent, receiver, notification.

@ASerbinski

the issue isn't making the SSE connection or displaying notifications, which are the limits of a "unified" approach. If that's all there was to it, I'd have had it done in 10 minutes. The issue is in negotiating with the Nextcloud server so that it sends messages to the right location, and in interpreting the messages received from Nextcloud.

Hello, I really appreciate your efforts to bring notifications on Nextcloud Talk (no matter you succeed or not). What you discuss here with other members is over my knowledge, but I would like to clarify something for my understanding : you say you would like to use existing browser notification standard to push notifications into Nextcloud Talk (with more or less difficulty to push notifications to the right place).

As you have preferences on HTML5 standards, do you think it is possible to use the same HTML5 logic for other apps ? (I think mainly about Axolotl, it is a Signal client for Ubuntu Touch which lack of push notifications).

Thank you for your feedback.

@prog-amateur2 Based on the link you've provided, the issue there isn't so much a matter of the technical feasibility of delivering notifications, but of the willingness of the people running the signal servers to deliver them. So from a technical standpoint, definitely it could be useful, but if the people running the service are unwilling to provide an endpoint, then there isn't a lot you can do.

Despite the publicity that "signal" has received from Julian Assange et al, FROM A PRIVACY POINT OF VIEW I strongly recommend avoiding their software and services for the following reasons;
1) Their unwillingness to provide a SECURE and PRIVACY RESPECTING channel for delivering notifications -- FCM is the opposite of private.
2) The fact that they are unwilling to even consider a method of switching off direct share on Android (all your "private" contacts from signal end up in the Android direct share menu!!!),
3) The fact that your supposedly secure and private communications are linked to a record in their servers tied to YOUR PHONE NUMBER -- in other words, records that may be made within their "not open for inspection" servers can link your telephone number to other telephone numbers by date and time, which could potentially be used to prove that you have had communication with certain parties, despite the service being marketed as preventing such associations.

If it was one issue that they were at least open to discussing and respectful of, then I could consider it an oversight, but the fact that there are multiple issue that they are totally unwilling to address tells me that this really isn't an organization that anyone should trust. IMO, for all the security that signal provides, you'd ACTUALLY be safer using google hangouts through tor.

@everyone

What I have is almost usable now.
It stores and reuses all credentials, and is converted into a foreground service.
The main things that make it somewhat less than usable are the fact that it doesn't load on boot, and I haven't done anything to account for network changes.

First time starting the activity, it will prompt for a nextcloud account via SSO (you have to have the main nextcloud application installed and working), then it will start the service. Subsequent activity starts will only load the service. If you want to select a different account (currently only supports one account, but that can be changed), you have to wipe through system settings. I'll add a few options to the activity later.

@ASerbinski thank you I appreciate your clear feedback and understand better why it is nearly impossible to find a solution without Openwhisper collaboration.

I agree regarding what you say about Signal. So far, to me, it is better to use Signal than to have Whatsapp/Viber/etc.,
I convinced a very good messaging app is one that you can self host like Nextcloud Talk, or a decentralized like Session or Matrix.

In any case I am amazed by the speed at which you progress, thank you again !

Looks like what I have is closer to usable than I expected. After driving home (so transition from wifi to cellular, a couple of cell disconnects, and on to another wifi, it is surprisingly still receiving messages. And not only that, but I haven't even disabled battery optimizations. I wonder if that's even necessary?

Obviously still needs a boot receiver, which I can pull together tomorrow, then some testers would be nice.

Anyone feel like testing?

Install the "Direct-Push Proxy" https://gitlab.com/Nextcloud-Push/direct-push-proxy into your Apache as per wiki: https://gitlab.com/Nextcloud-Push/direct-push-proxy/-/wikis/Apache-configuration
Sorry, I don't do nginx, so if you do, you'll have to figure out the equivalent configurations on your own for that, however, I'd be happy to add a wiki page if anybody would be so kind as to supply appropriate configuration instructions.

Edit pushclient.php -- there are some variables at the top needed to connect to your nextcloud database in order to verify authorization based on the deviceIdentifier and pushTokenHash.

Add 'allow_local_remote_servers' => true, to your nextcloud configuration.

Build and install "Nextcloud Push Notifier" on your phone: https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier

Once you have it set up, you can hide the permanent notification by long pressing it, selecting the "gear" icon, and unchecking the "Nextcloud Service" switch.

Clicking the notifications will launch Nextcloud Talk (if its installed).

This concludes basic usability features. Further changes will be related to bug fixes, security, and enhancements.

So I see a few positive Smiley's, but has anybody tried it? It would be nice to get some real world feedback.

I would definitely be game for testing, but I'm using nginx, and I'm not certain how to translate the config you provided from Apache to nginx.

I would definitely be game for testing, but I'm using nginx

Same situation here. :( I'd be thrilled to test when it's possible with Nginx

I use nginx and i would try it if i had a working config.
Also, can you provide a pre built package? Setting up the whole android
build is a pain for someone who doesnt do this routinely...

On Sat, Apr 24, 2021, 01:13 Adam Serbinski @.*> wrote:

So I see a few positive Smiley's, but has anybody tried it? It would be
nice to get some real world feedback.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/talk-android/issues/257#issuecomment-825950683,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD7UOU5SJ6NWXDCKRKUL3DLTKHWI7ANCNFSM4FLR6IBA
.

@ASerbinski thanks for your effort - for me it's dito, no routine to compile APKs. Maybe I get around setting up a VM for this, last time I did it is some 6yrs ago.

For the nginx part, like I said, I don't have an nginx anywhere to test on, but that really shouldn't be at all difficult if you've managed to make nextcloud work at all on it. You had to configure it for nextcloud, right?

Try looking at this for converting the port 8008 virtual host;
https://winginx.com/en/htaccess

And this appears to discuss the nginx equivalent of an Alias;
https://serverfault.com/questions/278351/nginx-root-versus-alias-for-serving-single-files

I'm not sure at all about the proxytimeout (or even if its needed in nginx), nor the fcgi flush, but the above should be enough to get anyone started.

I could potentially provide a prebuilt apk, however, I would have hoped that people would _insist_ on building it themselves, as a matter of security. I'm not sure if all of you realize just how vulnerable it makes you to accept a prebuilt apk from someone you don't know. In the least, it could provide access to your user account on your nextcloud instance. The program is simple enough that even an amateur could quickly scan through the source code to make sure that nothing looks fishy, but once compiled, all bets are off.

Much appreciated your hinting towards the risks with unknown binaries. This is one of those reasons I am waiting for an official nextcloud app which usually is provided by them or distributed by f-droid. But maybe I ll make the effort to setup a dev env for compiling.

Thanks a lot for the great effort you are putting into the solution of this topic, from what I understand it looks like you know very well what you are doing and why. Unfortunately I don't have capa to support in testing, but I would like to encourage you to continue your efforts. Maybe the colleagues heavily involved in NC Talk development could support you better, as this is a long-open topic which caused great confusion on my side as a beginner, and created a lot of frustration on NC, until I learned the hard way that the F-Droid version and the Playstore version of NC talk are a) different (!!) and b) that the F-Droid version is kind of useless without notifications. So please keep on going, until it reaches the official F-Droid release.
Btw: Will it in the end be possible to use the F-Droid version with your improvement on notification in parallel to other apps/messengers using the Google notification system? Such that I may switch to the F-Droid version without restriction of functionality?

I've made some updates recently to my notifications project, in particular;

  • Support for multiple nextcloud instances pushing messages to an Android client through a single connection,
  • Support for Redis as an alternative to System V message queues, which offers several advantages including eliminating the possibility of collisions in the 32 bit hashed queue id and eliminating the need to poll the queue (at the server, not the mobile client which NEVER had to poll).

I've been using this project personally for some time now, and have found that I've been able to completely eliminate all use of proprietary messaging services, since frankly, this is a lot more reliable at message delivery.

And for power consumption... immeasurably low. It doesn't even register.

Interesting how reality refutes both claims (excuses?) made by "contributor" @mario ... (see the third post in this thread)

I've also taken some time to look at the new "files high performance backend", also known as "notify_push".
https://github.com/nextcloud/notify_push

What that is, is a push messaging server. Unfortunately, it just doesn't work well for us. I have to say, I'm kind of sad at what they came up with there. They had a real opportunity to build something that would achieve the objectives that they are achieving with it, but failed to account for mobile devices in their implementation. So what they have there is a backend server that communicates using websockets, but is bound on a 1:1 basis with an instance of nextcloud. They also don't do any of the housekeeping needed for maintenance of mobile connections where you're frequently switching networks, or dropping connections, or have short timeouts on routers. What that means is that in theory, it could work for pushing messages to mobile clients, but it is inefficient and, without more work, wouldn't be reliable (I'm not saying that the concept is unreliable, just that they haven't addressed issues relevant to mobile devices).

It also isn't clear what, if anything, they've gained by this approach over push-v2 (what I'm using, and what NC uses to push through google services). To me it seems that push-v2 can do everything that notify_push does, but in addition allows you to share a single push proxy among multiple nextcloud instances and combine all messages destined for a single handset through a single TCP connection, which is a lot more efficient.

My best guess is that the people who built notify_push weren't aware of push-v2 or were under some impression that it is limited in its application to feeding messages through google, which it is not. It would certainly be a nicer approach to complete, and if necessary, extend the solution that already exists rather than building a competing solution.

@nickvergessen ;
The approach I've been using has been working very well, but right now I'm actually in the process of redesigning it a bit. With some digging into Nextcloud server a bit, I can make the server side work as an installable "app", with very little needed configuration changes to the server itself. And even more, even in this form, I can make it support multiple nextcloud instances using a single connection from mobile device to a single instance of Nextcloud.

I.e., mobile application registers for push notifications of several nextcloud accounts/instances to use ONE of those instances as a push proxy and feed all notifications over a single channel.

I believe that this would be an optimal approach. Now given that other nextcloud applications could receive notifications as well, it would make sense for one mobile application to register to receive notifications, and forward them to other applications as needed. Likely the main (files) application should receive the messages, and forward them to talk and other applications as appropriate.

I have a rewrite of the push proxy here;
https://gitlab.com/Nextcloud-Push/direct-push-proxy-v2

The webserver configurations are much simpler for this, since it installs into Nextcloud as a plugin, however, there are two configuration changes that may be required, described in the project README.

Note that this version requires Redis.

I just label issues, I have no expertise with push notifications on non-googleservice ways

@ASerbinski first thanks for your efforts in proposing a solution to this old issue. You seem to have made a big step to help all people with de-googled phones.

I installed direct-push-proxy-v2 on my nextcloud instance, and built and installed the app. The service start without error, but I get no notifications when I send talk messages from another phone (older, SDK 25 phone, without the app).
My phone has Microg installed, but GCM is not connected to Google and I don't want it to. Could this interfere with nextcloud push notifier?
What logs, on logs or phone, would your recommend me to investigate?

Microg will not have any impact on it at all. Nothing to worry about there.

Logs from your phone; logcat | grep NC- (several minutes of these logs would be useful)
Logs from your server; grep ssepush access_log

Go through the logs to make sure that there is no sensitive information before sharing. The android log, in particular, may contain the actual message contents if a message was decoded by Android correctly. It still has some use for debugging.

The logs should be able to identify the issue, however it is most likely one of the two issues discussed in the readme.

I added the "ProxyTimeout 600" and " " to the httpd-vhost.conf file on my Synology server. I'm not absolutely certain this is the good file for that though.
So far, nothing from logcat | grep NC-.
As for the access_log,is it an Apache or a Nextcloud server log? I couldn't find it.

Apache log. Not sure what synology refers to, but for example on EL (redhat/centos/rocky), you'd find it in /var/log/httpd/

Did you press the "[RE]LOAD SERVICE" button after setting everything up? If you did, then it must output logs like that.

I've added several more changes to the repositories;

Authentication: The process of registering to receive push messages now involves an API request to one Nextcloud server designated as the "master" in order to provide authorization for any other Nextcloud instance to use it as a push proxy. These credentials will be used during the notification process and by the push client for authentication.

Simplification: Because we now flag one Nextcloud instance on the client as the "master", we can generate the needed connection URI's and it is no longer necessary to manually input them.

Configuration: Redis configuration was hardcoded, it is now retrieved from the Nextcloud system configuration file at config/config.php. Note that the database number used is going to be 1+ the system Redis database number.

Note that it is now technically possible, and secure, to send 3rd party notifications through a Nextcloud server. The client can't handle it just yet, but it wouldn't be difficult to implement there as well. This would require the following;

1) Android application connect to 3rd party service, provides notification URI and pushTokenHash and receives a deviceIdentifier string (i.e. some unique string, encrypted).
2) Android application adds this deviceIdentifier to the authentication call.
3) 3rd party service pushes notifications to the specified URI including the pushTokenHash and deviceIdentifier.
4) The notifications are sent by the push proxy to the Android application, which can do something with the data.

@lotth
One other "issue" comes to mind;
If you have the chat open in browser or talk-android from the push recipient's perspective, then Nextcloud won't send the notification since it figures you're sitting there watching them.

What I mean by that;
Say you have 2 users, A and B. A is registered to receive push messages, B is sending them. If A has the chat open in a browser when B sends it, then Nextcloud wont push the message, figuring that you don't need it to be pushed since you're sitting there at the browser.

Also note that I can't control this behavior, its part of the nextcloud notification system.

I think this project is in pretty excellent shape now.

  • messages are pushed reliably,
  • messages may be pushed from multiple nextcloud instances,
  • messages may be pushed securely from 3rd party services,
  • battery consumption is negligible,
  • there is now a help/configuration page in Nextcloud's settings.

From nextcloud/apps/
sudo -u apache git clone https://gitlab.com/Nextcloud-Push/direct-push-proxy-v2 ssepush

Now go into your Nextcloud instance, Apps --> Your apps, and scroll down to "SSE Push Notifications" and enable it. Now Settings --> Personal --> Mobile Push Messages, and read.

@ASerbinski thanks for your comment. Actually I find this behavior perfectly normal,

@lotth
One other "issue" comes to mind;
If you have the chat open in browser or talk-android from the push recipient's perspective, then Nextcloud won't send the notification since it figures you're sitting there watching them.

Actually, this behavior is the one you would expect from a reasonably clever app. Receiving notifications even when you have the app open would be annoying. I was expecting this and I made sure the app was not open when I made the tests.

Since then, I succeeded in getting your push working! With the app that I had built, I got a TokenMismatch exception. But when I used the 0.0.2 apk you released, the notifications pop up.

Thank you again! By the way, I like the new packaging for the nextcloud app, with verbose readme on the app configuration page.
As possible improvements do you think it is possible to have more detailed options with the notifications? Especially, for incoming calls it would be great to have an "accept" button in the notification to directly open the call in Talk. In the current version, the notification opens Talk and you have then to open the call.

Now let's hope the app gets packaged officially...

Actually, this behavior is the one you would expect from a reasonably clever app. Receiving notifications even when you have the app open would be annoying. I was expecting this and I made sure the app was not open when I made the tests.

I wasn't actually saying that its a bad feature. Just that it could be confusing if it caught you off guard.

Since then, I succeeded in getting your push working! With the app that I had built, I got a TokenMismatch exception. But when I used the 0.0.2 apk you released, the notifications pop up.

Interesting. Don't suppose you saved the stack trace for the exception? I'd like to know where it came from if possible. Shouldn't be anything special about being compiled by me or you, I suppose its possible that you caught the two repositories while there was an API mismatch. There's been a lot of flux over the last few days ;)

Thank you again! By the way, I like the new packaging for the nextcloud app, with verbose readme on the app configuration page.
As possible improvements do you think it is possible to have more detailed options with the notifications? Especially, for incoming calls it would be great to have an "accept" button in the notification to directly open the call in Talk. In the current version, the notification opens Talk and you have then to open the call.

I agree that this would be ideal, but I'll have to investigate whether it is even possible. For this, I'll have to dive into and understand talk-android a lot better, since it would involve sending a command to answer, as well as the correct account and id. If it doesn't have infrastructure available to send instructions like that, then there won't be anything I can do.

Now let's hope the app gets packaged officially...

This is the song and dance you need to go through to get it "published":
https://nextcloudappstore.readthedocs.io/en/latest/developer.html#publishing-apps-on-the-app-store

I'm not eager to go through that myself. Would be nice if somebody would take that aspect on..... ;)

Interesting how the first comment you had to make after working on something is putting the word "contributor" next to my name in quotes when 99% of the code of Nc Talk is written by me, and how you had to trash other files-related push just because it doesn't work for you :P

Your solution "works", awesome - good job. Except it relies on an alarm every 15 minutes so that means if the foreground service dies for any reason whatsoever, you won't get a notification. Which, suprisingly, will happen, one way or another, and you're promising magic and perfect delivery :P

Sign it with platform keys, preload the app, and you don't even need a foreground service - the problem is that is unfeasible for 99,99% of the world.

Please be more considerate of other peoples' work, thank you.

Interesting how the first comment you had to make after working on something is putting the word "contributor" next to my name in quotes when 99% of the code of Nc Talk is written by me, and how you had to trash other files-related push just because it doesn't work for you :P

@mario Author would be more appropriate then :-) Thank you for all the work on Talk, and for supporting f-droid integration.

I would have thought you would be more supportive about a GCM-free notification solution, because without it f-droid version is basically useless, which wastes some of your achievement. But your past comments on this issue were all negative, about the possible solution being a battery killer and poor UX. It looks like you just wanted to kill the idea from the start.
Lots of f-droid users are certainly ready to use an imperfect solution instead of having none. If it means using a few percent battery and less-than-perfect system integration, I personally can live with it. Compare it with Google services that not only uses significant battery but also uses your own personal data!

Your solution "works", awesome - good job. Except it relies on an alarm every 15 minutes so that means if the foreground service dies for any reason whatsoever, you won't get a notification. Which, suprisingly, will happen, one way or another, and you're promising magic and perfect delivery :P

I understand why you find it unreliable, but doesn't your comment apply to other apps that rely on that kind of trick and are reliable enough to live with? I still need to check long term usability of this app of course. Even if Talk is a really useful app :-) it is not highly critical, so rare service interruptions would be tolerable.

Sign it with platform keys, preload the app, and you don't even need a foreground service - the problem is that is unfeasible for 99,99% of the world.

Please be more considerate of other peoples' work, thank you.

Please understand there is a degree of irritation about this topic. The issue has been opened nearly 3 years ago. Judging by the thumbs up, there seem to be lots of people interested, but there has been a litany of "bad UX, poor solution" etc. from the start.

As you wrote earlier

It's a shame though that everybody seems to push push push, yet nobody wants to contribute :(

ASerbinski at least came up with a solution that is not perfect, but it does the job by producing near real-time notifications. It uses some battery of course, but is not killing it. I only have been using it for about 10 hours and it accounts for 3% battery loss.

I am extremely supporting of GCM-free solution (even now when I'm not working on Talk anymore), but I'm also painfully aware of the Android internals that will prevent this from working properly, especially when system goes into deep sleep (how this manifests itself depends on the vendor, but is a problem regardless).

Again, with the assumption that some users could push apps into appropriate folder on the system partition and have it signed with platform keys, this could work decently enough, but tell me - what percentage of people can/want to do that, even if they use F-Droid?

You might or might not be aware, but I worked with @Bubu to get Talk into F-Droid, we discussed push without GCM over lunch and several other sessions before FOSS Push came to be, and together with a contributor I even co-authored an SSE server that tied directly into the push proxy (one that's running on https://push-notifications.nextcloud.com) without needing any changes on the Nextcloud server side. It worked, but wasn't good enough for the use-case of calling - I just can't tell people to use something when they can miss a call. Sure, Talk is not an "emergency calling" solution, but is still extremely annoying.

I completely understand the iritation, and I welcome solutions, just like I always invited people to contribute. I'm happy that there is something that kind-of-works, and if that's enough for most people - then so be it.

I'll contribute any knowledge I have, including the before-mentioned platform signature and preloading note that might help some people. Equally however, I want to make sure people who contributed or did stuff they didn't neccesarily have to do, including the Files-related push, are appreciated and not bashed.

And I see I failed to answer one of your comments @lotth - various apps try to circmuvent the issues in various ways, all with their own set of problems, so yes - my comment applies to all of them.

In the end, thank you for a very thougtful and neutral comment - much appreciated and goes to show there are still voices of reason around here :)

@mario "contributor" is simply the github tag you carry here. After your comment: It's then really quite unfortunate that you didn't leave any comment here earlier to inform about the progress and findings you made. And please, do also be more considerate of other peoples' work. You have and still seem to dismiss all push solutions, _not_ having presented one.

In so long of a time the only apparent "author" action here was to tag and hide my comment as "off-topic" in which I forgot to mention that --when I finally compared them-- the calls (talking) in two f-droid "XMPP clients that I referred to worked well without gms".

It's a platforms-made problem, so probably only to be solved by alternatives.

Custom roms + F-Droid are only an incomplete and intermediate step (with a small but growing share), but still, this combination seems to allow these things to just work (other issues aside).

@mario Just tried to explain irritation, to make it understandable. From your side I think I understand that you were looking for a general solution (your global push server) and there it unfortunately won't be able to work with every vendor's rom.

Clearly, my thumbs up for your messages here, the work you've done and experience you share.

@mario

Interesting how the first comment you had to make after working on something is putting the word "contributor" next to my name in quotes when 99% of the code of Nc Talk is written by me, and how you had to trash other files-related push just because it doesn't work for you :P

Its not your work I'm critical of, its your comments in this thread, which are downright wrong and need to be countered. The fact that you have been tagged as a contributor it there to imply that your voice carries more weight, which makes it confusing to most people reading this thread when comments you make are invalid.

Your solution "works", awesome - good job. Except it relies on an alarm every 15 minutes so that means if the foreground service dies for any reason whatsoever, you won't get a notification. Which, suprisingly, will happen, one way or another, and you're promising magic and perfect delivery

You're putting the cart before the horse by assuming that I wrote the alarm code to solve a problem I've seen happen. This is not so. I've not seen the service ever die, I wrote the alarm before the system was at all functional just to make sure that it would be restarted always. But like I said, so far it hasn't shown to be needed.

And further, if the process did die for any reason, the notifications would all still be delivered immediately upon reconnection. Like it or not, mobile devices do disconnect. Frequently. Its the nature of wireless.

Sign it with platform keys, preload the app, and you don't even need a foreground service - the problem is that is unfeasible for 99,99% of the world.

What's your point?

Please be more considerate of other peoples' work, thank you.

Like I said, its not your work I'm critical of, its your bad comments.

Anyway, this defensiveness you are showing is clearly not helping your position. I see that others have already called you out on it.

I only have been using it for about 10 hours and it accounts for 3% battery loss.

@lotth Please file a bug report on my issue tracker. That is extremely high and inconsistent with expectations. I'd like to look into what is going on there.

Have fun with your solution - again, it might work for you, it might work
for others, but it is not the solution I was aiming at. Getting a call
notification 10-15 minutes later (even if you havent seen the issue) is
just not good user experience.

Its one thing when There’s a network disconnect, its a whole other thing
when you have network connectivity and you dont get notifications.

Others in the World have attempted this to varying degree of success, but
nobody guarantees Magić Like you do here - so all of my comments still
stand.

FYI, there are alternatives inside Android you could explore to
significantly lower the chances of your app dying which will work at least
on A13 and possibly even A14 before being completely killed off from AOSP -
so do explore that instead od trying to prove a Random person on the
internet (Like me) wrong xD

On Saturday, May 15, 2021, Adam Serbinski @.*> wrote:

@mario https://github.com/mario

Interesting how the first comment you had to make after working on
something is putting the word "contributor" next to my name in quotes when
99% of the code of Nc Talk is written by me, and how you had to trash other
files-related push just because it doesn't work for you :P

Its not your work I'm critical of, its your comments in this thread, which
are downright wrong and need to be countered. The fact that you have been
tagged as a contributor it there to imply that your voice carries more
weight, which makes it confusing to most people reading this thread when
comments you make are invalid.

Your solution "works", awesome - good job. Except it relies on an alarm
every 15 minutes so that means if the foreground service dies for any
reason whatsoever, you won't get a notification. Which, suprisingly, will
happen, one way or another, and you're promising magic and perfect delivery

You're putting the cart before the horse by assuming that I wrote the
alarm code to solve a problem I've seen happen. This is not so. I've not
seen the service ever die, I wrote the alarm before the system was at all
functional just to make sure that it would be restarted always. But like I
said, so far it hasn't shown to be needed.

And further, if the process did die for any reason, the notifications
would all still be delivered immediately upon reconnection. Like it or not,
mobile devices do disconnect. Frequently. Its the nature of wireless.

Sign it with platform keys, preload the app, and you don't even need a
foreground service - the problem is that is unfeasible for 99,99% of the
world.

What's your point?

Please be more considerate of other peoples' work, thank you.

Like I said, its not your work I'm critical of, its your bad comments.

Anyway, this defensiveness you are showing is clearly not helping your
position. I see that others have already called you out on it.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/talk-android/issues/257#issuecomment-841641590,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABNMUYPMY4Q7MHGF6VGC3TNZJJPANCNFSM4FLR6IBA
.

@mario
Let me make this very clear...
Notifications are INSTANTANEOUS.
They are not delayed, not by 15 minutes, not by 15 seconds.

And despite the size of your ego, I really don't care about proving you wrong. I just care about making something that works, which I have.

They are until the service dies, I do not doubt that at all. Again, I am
not saying your solution is BAD and I am happy somebody Stepped up, but it
is not the Magic you seem to want to make it.

On Saturday, May 15, 2021, Adam Serbinski @.*> wrote:

@mario https://github.com/mario
Let me make this very clear...
Notifications are INSTANTANEOUS.
They are not delayed, not by 15 minutes, not by 15 seconds.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/talk-android/issues/257#issuecomment-841648352,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABNMQ5HCVT34CNZVDV6QDTNZQDHANCNFSM4FLR6IBA
.

You are assuming that the service would die, which is unlikely. Again, try to understand that the alarm isn't there to fix a problem that exists, it is there just to make sure that if the whole world explodes, somebody survives. The techniques I've used, which others have also used very successfully, make it unlikely for the the service to ever die unless the device is under EXTREME memory pressure, under which circumstances, google services also would be killed, assuming they were present.

And speaking of google services, its always been my observation that messages passed through google stand a frustratingly high chance of being lost or delayed. They aren't the example of absolute reliability that you make them out to be.

As far as the word "magic" goes, if I used that word, it was obviously meant with sarcasm.

I am _extremely_ supporting of GCM-free solution (even now when I'm not working on Talk anymore), but I'm also painfully aware of the Android internals that will prevent this from working properly, especially when system goes into deep sleep (how this manifests itself depends on the vendor, but is a problem regardless).

You might or might not be aware, but I worked with @Bubu to get Talk into F-Droid, we discussed push without GCM over lunch and several other sessions before FOSS Push came to be, and together with a contributor I even co-authored an SSE server that tied directly into the push proxy (one that's running on https://push-notifications.nextcloud.com) without needing any changes on the Nextcloud server side. It _worked_, but wasn't good enough for the use-case of calling - I just can't tell people to use something when they can miss a call. Sure, Talk is not an "emergency calling" solution, but is still extremely annoying.

I was aware on your work with @Bubu (and thank you for that, not all developers support integration of their work into f-droid), but not about a GCM-free solution. Now I understand your comments better.
But aren't you too ambitious? What I mean by that is that, as an f-droid user, I prefer to that kind of solution which makes f-droid's Talk usable now, than waiting maybe several more years for the perfect solution - if there is one.

I only have been using it for about 10 hours and it accounts for 3% battery loss.

@lotth Please file a bug report on my issue tracker. That is extremely high and inconsistent with expectations. I'd like to look into what is going on there.

Done https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier/-/issues/5

Please do not put words in my mouth if I've never spoken them :)

CM/FCM are awful, but they're the best we have on a platform that is open source in principle, but whose most interesting components are closed source. That being said, lots of what is wrong with GCM/FCM lies in the Android optimizations for the battery (despite them having privileged handling) and poor handling of "edge cases", at least part of which will hopefully be resolved once the client part of FCM is open sourced (promised, but still pending).

Again, I'll repeat myself and put this story to rest - I'm happy you've done it, I'm happy it works for you and I'm sure it'll be useful to others. "Unlikely" is a very odd term to use when you have no real-life data - in principle that is correct, reality is often very very different, especially with the various vendors doing their own optimizations (Huawei, Xiaomi, etc folks - helllo! :))

Ego or not, I've done GCM/FCM notifications for two major Nc Android apps, I was at the meeting where we designed encryption for their content, and I've spent significant time on trying to find a solution that would work for everyone that doesn't have Google Services - the solution is simply not as simple as you make it out to be. I wish it was.

Have fun and, once again, thank you for your contribution! :)

I am _extremely_ supporting of GCM-free solution (even now when I'm not working on Talk anymore), but I'm also painfully aware of the Android internals that will prevent this from working properly, especially when system goes into deep sleep (how this manifests itself depends on the vendor, but is a problem regardless).

You might or might not be aware, but I worked with @Bubu to get Talk into F-Droid, we discussed push without GCM over lunch and several other sessions before FOSS Push came to be, and together with a contributor I even co-authored an SSE server that tied directly into the push proxy (one that's running on https://push-notifications.nextcloud.com) without needing any changes on the Nextcloud server side. It _worked_, but wasn't good enough for the use-case of calling - I just can't tell people to use something when they can miss a call. Sure, Talk is not an "emergency calling" solution, but is still extremely annoying.

I was aware on your work with @Bubu (and thank you for that, not all developers support integration of their work into f-droid), but not about a GCM-free solution. Now I understand your comments better.
But aren't you too ambitious? What I mean by that is that, as an f-droid user, I prefer to that kind of solution which makes f-droid's Talk usable now, than waiting maybe several more years for the perfect solution - if there is one.

Hey hey @lotth ,

In most cases I would agree, but for an app that wants to do calls I just feel this isn't it. Again, it's me - your opinion might be different and I respect that. I have always invited people to find solutions that I might have missed or couldn't do, whatever the case might be.

Like I said to @ASerbinski, if he only wanted to listen, there are several ways better than Foreground service even that would make the service less likely to die. That is without going into custom ROM, platform keys and preloading the app which would give it maximum priority and which is (long-term) probably the way to go - by aactively promoting a given solution like @Bubu was working on to be integrated in "all" popular Free ROMs out there.

The rest:

To each their own, I've done the best I could with the resources I've had (me working on my own on an app that any competitor has tens or hundreds of devs on), and I've tried to serve the Free community the best I could. I'm sorry if it wasn't enough - I'll keep doing open source because of the people I've met and the people I still have to meet who are significantly more considerate and who won't just reply to parts of comments that suit them :P

Maybe instead of focusing on me, you should focus on the fact that the Push proxy is nowhere to be seen as a repository and see where that rabbit hole takes you xD

Ok, so look, as talk-android has employed the use of FCM, that must therefore be the bar for performance. If you concede that performance of FCM is "awful", then I hope you are accepting of the _possibility_ that what I have working now is at least as good as, if not better than FCM (and data that I have accumulated on it shows that it is, in fact, _significantly_ more reliable than FCM), and if that is the case, the minimum requirements for this project have been accomplished.

If you have a better idea than foreground service, I'd be happy to hear of your ideas, but for the moment, this is the mechanism offered by the platform to accomplish this objective.

Now as far as that supposedly integrated solution you are promoting goes, I've already built in a mechanisms to allow nextcloud to serve as a notifications hub for 3rd party services, rendering that other project obsolete. Is there room to grow? Of course, there always is. Especially on the Android side of things where currently all it does it displays a notification.

FCM is awful, and I accept the possibility that your solution is good enough for some people, like I said :) I wouldn't go as far as saying that this is better than FCM as it was not tested on the scale that FCM was AND the fact that your service is significantly more likely (even if "unlikely") to die compared to Google Play Services / FCM.

Nextcloud shouldn't be a notifications hub for 3rd party services, no matter how one wants or doesn't want that. You can't force everyone to use Nextcloud if they want to be free from Google services, nor should that be the intention. There's a reason why @Bubu, while incredibly active here, wanted an all encompassing solution outside the Nc ecosystem.

All due respect for your work in the past @mario , here some of my thoughts

and I accept the possibility that your solution is good enough for some people

It would be good enough for me, and I think for many others as well

as it was not tested on the scale that FCM was AND the fact that your service is significantly more likely (even if "unlikely") to die compared to Google Play Services / FCM.

I guess there is only one way of trying to find out, even if it would be only approximately: Roll it out

You can't force everyone to use Nextcloud if they want to be free from Google services, nor should that be the intention [..]

I don't think the purpose of @ASerbinski 's efforts here is to create some kind of Nextcloud monopoly around this. But after all, this is the nextcloud git and the nextcloud users need some kind of solution

@mikelupe well, he did mention that this solution makes any effort of all-encompassing solution (e.g. one worked on by Bubu) obsolete. Maybe I misunderstood, but the wording was quite clear there.

@mikelupe to add to my comment above, I'm happy that it's good enough for you and that it'll serve for people, but I think I've said that enough times now ;)

@mario @ASerbinski please stick together guys, it would be a waste of "creativity" not to do so

I admit that I don't have too much knowledge in this topic and that's probably why I don't understand how Tutanota could create their own open source push service that works flawlessly on my Google-free Android phone. Could someone enlighten me why their solution does not work for Nextcloud?

@mikelupe well, he did mention that this solution makes any effort of all-encompassing solution (e.g. one worked on by Bubu) obsolete. Maybe I misunderstood, but the wording was quite clear there.

At this point, I'm starting to realize that you are intentionally misunderstanding. The purpose of the bubu project was to provide a channel for 3rd party applications to push notifications to a phone, and what I've created does that, which makes it pointless to incorporate the bubu project.

@mikelupe well, he did mention that this solution makes any effort of all-encompassing solution (e.g. one worked on by Bubu) obsolete. Maybe I misunderstood, but the wording was quite clear there.

At this point, I'm starting to realize that you are intentionally misunderstanding. The purpose of the bubu project was to provide a channel for 3rd party applications to push notifications to a phone, and what I've created does that, which makes it pointless to incorporate the bubu project.

Yup, sure. Let's end it :)

I admit that I don't have too much knowledge in this topic and that's probably why I don't understand how Tutanota could create their own open source push service that works flawlessly on my Google-free Android phone. Could someone enlighten me why their solution does not work for Nextcloud?

Crucial sentence there :)

"We leveraged Android Job mechanism to launch our service periodically (at least every 15 minutes),"

That might be good enough, or might not - depends on your use-case.

@0x47
From the descriptions in your link, they're using a very similar approach as what in using. The main difference is that I'm placing a higher priority on not dying. They mention a potential disconnect, but that they bring it back to life. I'm doing the same thing, but the chance of my process dying is much lower.

The main reasons not to use their implementation is that doing so just makes the job bigger, and creates dependencies on them to support their code. Since SSE is a function of the web server, there is no great justification to rely on other peoples wrapper libraries.

I've made another update on the server-side repository. Admin settings are now split into the admin section, and I've added a setting for configuring the keep-alive interval. For people with a proxy who are unable to configure the proxy timeout, the keep-alive interval can be set to within the timeout in order to prevent it from timing out, or for people who find that the keep-alive interval can be stretched out, it can also be set higher.

In general, the power consumption will be higher with a smaller interval, since it will be communicating with the phone more frequently. I'm not sure how much power, if any, can be saved by stretching the interval out longer, especially since at 5 minutes it doesn't even register. The phone has to wake up every few minutes anyway, and Android is smart enough to fire off waiting timers when the opportunity presents.

After a few hours with the interval set to 55 seconds, it does register power consumption (1%), but that won't be nearly as bad as having it run a full reconnect every minute.

In general, people using mod-php (rather than php-fpm), not using any proxies, and with only a small number of push clients, should be able to just install the server-side into nextcloud and call it good.

@ASerbinski Thank you for your work on getting some kind of notification support. I've been following this thread for a while now and your latest way of just installing the server side app and the apk convinced me to test the solution out.

I've been testing it for a couple of days and so far it has been very positive. Even after wifi disconnects and device restarts, the notifications arrive almost instantly. This is on a Samsung S9+ whose ROMs are known for killing off apps to "save" battery. Battery usage is about 1% per 10 hours or so. I'll keep testing though.

The third party services feature looks very promising as well. I was surprised that it basically functions like a stripped down version of Gotify.

Have you thought about contributing this or opening up a discussion in the main Nextcloud Android app repo since your service isn't just limited to notifications from Talk. Maybe you'll get more beta testers. Maybe the NC Android F-droid version can use your code. But then I guess someone has to maintain the server side app via the official app store. I just wish the F-droid NC apps went the Signal way of having a fallback in case Play Services was absent as I'm sure the concerned users would appreciate having some kind of notification support that might not be instant rather than having none.

Hi @accountForIssues !
Thank you for reporting your success. I am happy that it is working well for you.

For the time being, its probably better to keep the testing group reasonably small. I am open to expanding eventually, as it gets shaken out a bit more.

I've made some more small changes. Because of the potential for proxy configuration issues, I've added a facility for the server to directly issue warnings to the client when undesirable behavior is observed.

So far, I'm recording connections made over a 12 hour period, and if this exceeds 144 (meaning that on average, the client reconnects after less than 5 minutes), a warning notification will be displayed at the client indicating the total number of connections made.

Thanks for your work! I can confirm it works on Raspbian (Debian) Buster with nginx and LineageOS 18.1 (Android 11) On a good ol' LG G5 phone.

Hello,

Thank for everybody who work on app/features and work to perfect it ;)

On my side, i have test ssepush => work on server, not a problem for that.
After, i can't install on my phone :
OS : /e/ version 0.10 (yes, i need time to upgrade to 0.16/17 for Q build android)
on Samsung galaxy A3 2016

@EricThi You should consider creating an issue for your case on the source repo. The author would be able to assist you better over there :)

@EricThi ; I may be able to address your situation just by lowering the minSdkVersion, but I don't know what "/e/ version 0.10" means in terms of Android API level -- it sounds like something vendor specific. Please look up your API level and let me know at my issue tracker so I can run a test build for you. API level can be found here; https://developer.android.com/studio/releases/platforms

I only have been using it for about 10 hours and it accounts for 3% battery loss.

@lotth Please file a bug report on my issue tracker. That is extremely high and inconsistent with expectations. I'd like to look into what is going on there.

Done https://gitlab.com/Nextcloud-Push/nextcloud-push-notifier/-/issues/5

Just to let readers of this thread know that my battery consumption originates in my Synology NAS overwriting the http.conf at every Apache restarts, which makes me unable to set the ProxyTimeout value. @ASerbinski designed a workaround to avoid the periodic reconnections, see https://github.com/nextcloud/talk-android/issues/257#issuecomment-842289887 (thank you Adam). More details at the gitlab link above.

Battery loss was 1% in nearly 2 days !

I may be slightly off topic, but I wanted to bring to your attention this app (stub) I found, that displays nextcloud notifications on phones without Google Services. It's only a stub, but it _kinda_ works, despite not using Android-SingleSignOn lib. The biggest problem is it doesn't allow to open the relevant app clicking on the notification. At least, it warns me that there's a new message in Talk, so I don't answer days late as I've done until now...
I hope this can help as a temporary solution.

Meanwhile, I forked NextcloudServices and I'd like to generate an intent to open Talk and the other nextcloud apps on notification tap. Unfortunately, seems that the web url contained in the notification is unable to open the relevant apps, maybe because being every instance self-hosted, there is no way to know when an url is a Nextcloud one. I suspect I should generate an explicit Intent, for every Nextcloud Android app, but I couldn't find a way to translate the url into an intent valid for all apps.

@penguin86
What you found there polls the server and eats battery. It isn't viable.

It often helps if you read backwards a few messages in a thread like this, because you might see something that actually does the job bou are looking for, well.

https://gitlab.com/Nextcloud-Push

Thank you @ASerbinski for pointing out. It is a long discussion and I may have missed some messages.
Unfortunately, Nextcloud-Push requires installing a plugin, and that's possible only if the nextcloud instance sysadmin agrees.
Imho the strength of NextcloudServices is that uses the standard notification api available by default. Sure, it eats battery (about 10% a day in my device), but it allows to receive notifications, so I just wanted to leave a message to help others in my same situation until an official solution is available.

Not necessarily. You can can set up your own nextcloud instance anywhere you like, install the plugin there, and use it as a relay.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

tobiasKaminsky picture tobiasKaminsky  ·  3Comments

ghost picture ghost  ·  6Comments

mario picture mario  ·  6Comments

tobiasKaminsky picture tobiasKaminsky  ·  6Comments

jakobroehrl picture jakobroehrl  ·  4Comments