Telegram-foss: Unreliable Message Delivery since 3.1.3

[vanitasvitae]

Hi
Since the latest update, push messages are wa, less reliable than in earlier versions. I didnt change any settings. Often messages dont appear even when I have the chat open. When Im in a group chat, i often get multiple messages at once delivered.

Im using Paranoid Android 5.1

[vanitasvitae]

*way less reliable

[Bubu]

I had the same Problem. Reinstalling Telegram seems to have fixed this.
But it does reappear when going back one Version to 3.1.2 through fdroid and upgrading from there to 3.1.3 again.

[Bubu]

It seems to happen sometimes in the emulator. I can't reliably reproduce it. And when it did happen there were no suspicious logcat messages. I have no idea how to debug this.

[vanitasvitae]

Is there a way to provide you with logs?

[ghost]

I have this same issue and also seemed to be fixed by reinstalling the app, but it's there again. I also know the official Telegram team has a modified telegram client with log support. I don't know if its code is avaible tho

[geileszeuch]

I noticed this problem heavily since 3.2.2. Sometimes I receive messages heavily delayed in the middle of the night. The strange thing is that although I confirmed that the messages were sent during the day, the timestamp under the message might show (not always) that it was sent at night.
Sometimes messages only appear when I open telegram and at other times I don't receive them although the chat is open.
Also, it happens that my messages are not being received or with huge delay although they have 1 checkmark and are being sent to the official Telegram upstream version. There needs to be a better way of fixing this than reinstalling. Some people have secret chats that they don't want to be deleted, which, I assume, happens when reinstalling.

[Bubu]

Yes, I agree that reinstalling is not a good solution to the Problem.
Could someone maybe try out if clearing the app's cache (should not delete secret chats) and/or data (this again clears secret chats) also solves this problem?

[vanitasvitae]

Clearing cache doesn't help. Still delayed messages.

[vanitasvitae]

Trying with deleted data right now.

[ghost]

Actually in my case it's not just delayed messages but also sound notification sounds a bit later than vibration. They're not simultaneous has used to be.

El 5 de octubre de 2015 12:59:27 CEST, vanitasvitae [email protected] escribió:

Trying with deleted data right now.

---

Reply to this email directly or view it on GitHub:
https://github.com/slp/Telegram-FOSS/issues/36#issuecomment-145495253

[Ecron]

@helfio said:

I have this same issue and also seemed to be fixed by reinstalling the app, but it's there again. I also know the official Telegram team has a modified telegram client with log support. I don't know if its code is avaible tho

Not in the official Android app afaik. They included a debug option in iOS app, pressing several times over Telegram's icon in the "Change number" screen, but there's no option in official Android app. Although I know for sure they will implement it in the future.

It would be a good idea to implement some hidden debug options here in the meanwhile to be able to find out what's the problem with the messages delivery...

[slp]

Late Telegram versions had introduced a lot of changes in the way it manages the connections to its Datacenters. Version 3.2.2 even moved a lot of Java code to a JNI library (I suppose to share the code with other platforms).

On the other hand, old Telegram versions relied a lot on GCM (Google Cloud Messaging) for receiving notifications, which forced me to implement a Background Service class to pool the servers at regular times. At some point, upstream implemented a push based solution to replace GCM, and made it the preferred way to receive notifications, but with the last two versions it seems that we're back there again.

I think I'll be able to provide an experimental APK with a Background Service in a few days. I'll need some people to test it, to confirm that notifications are received with a reasonable delay, even when the phone has been asleep for hours, and that the impact on battery life is not too harsh.

[ghost]

I can test it. I don't know if you could make the apk with different system name (no org.telegram.messenger) so we don't have to uninstall the standard FOSS :)

[Ecron]

Count on me for the testing too.

[geileszeuch]

@slp You're saying that they're degrading their own push service although telegram is configured by default to use it? That sounds very unlikely and if true very stupid. It seems like people on the official client are not complaining about delayed messages, at least I can't notice it on their twitter channel. Is it possible that the replacement of boringssl is somehow involved and caused issues?

What about the last three commits on upstream which were made after the 3.2.2 release tag:
-"Using pipe instead of signal to interrupt epoll_wait"
-"Fixed restore key generation after network becomes available"
-"Fixed release and debug AndroidManifest.xml"

Do they maybe fix our problem? Judging by DrKlos actions on github, I can very well imagine that they were released in the store just not tagged with a new release.

Also, how then to explain the fact that search doesn't work anymore. I cannot imagine that it doesn't work on upstream and remained unnoticed for weeks now. There are no complains on twitter. Plus not only are the messages delayed, but also the vibration and notification sound appear only a few seconds after the phone starts to flash the notification light.

I just hope that something accidently went wrong when merging upstream so we don't need to go back to a background service. Keep up the good work slp.

(Side note: doze on android m might require some changes to get push messages without gcm done, but android m support was only released after 3.2.2.)

[Ecron]

The upstream official app has none of these problems. It gets the messages and notifications in time, and the search is active. Nobody reported any of those issues.

I can't imagine what's the reason :confused:

[mikran]

I changed notification ringtone to 'none' and vibration to 'short' and all notifications dissappeared / are late. I then readded a sound notification and now it seems to work again. And this was about the first thing I did after reinstall.

edit: and messages are delayed again, that didn't fix it as was the case with re-install.

[xavihernandez]

Same here, it's like IDLE connection is not kept.
When I open the application I can read "Waiting for network..." (in EDGE I can see this message as connection is slower and then takes more time to connect) before it shows "Telegram"

[xavihernandez]

@slp hi, any news on this?

[Bubu]

This problem did not occur with the latest update for me.
Does v3.2.6 fix this for everyone?

[ghost]

I suffered this issue and I've just updated a few hours ago. It looks like it's not fixed but I want to test it longer before confirm anything.

[slp]

@helfio Please check in Settings->Apps->Running->Telegram if the "NotificationsService" is present and running.

[ghost]

It's present and running. It shows the same time the app have been opened.

[slp]

@helfio I've just built a package compiled with DEBUG_VERSION=true, which enables the logcat output. You can download it from here: https://sinrega.mooo.com/index.php/s/cTMqwYXLs5Q1rFs

You can use it to obtain some debug info, but as it's signed with a different certificate, you'll need to uninstall the FDroid version, losing your secret chats and preferences in the process.

[ghost]

Definitely the issue it's not solved on my phone. It doesn't get notifications of messages I already read from telegram desktop neither.

I'm going to use this package to try obtain some info.
This debug flag just enable output to system logcat, right?

Edit: this flag enabled the log system of telegram, just as the upstream team does to get info about issues. Just as I mentioned here: https://github.com/slp/Telegram-FOSS/issues/36#issuecomment-145253187

[Bubu]

You said earlier, you solved this by reinstalling the app but then the problem reappeared?
Was that after another update or without any apparent cause?

If it was after updating again you should be able to fix it again with reinstalling. And at least for me the problem did not appear with the latest update (but that might have been luck...).

[ghost]

@Bubu It was after uninstall and install again I think, but I cannot remember.

I just made a test with debug version and indeed the issue it's still there.
This is the Telegram log: (I have a copy of this)

I sent a message at 23:23:02 and switched on the screen at 23:28. Before opened Telegram I checked the notification service was running wich it was, then opened the app and then the notification arrived.

I hope this helps to catch the error.

Edit: same result on mobile data instead of WiFi. Message sent at 00:10, Screen on at 00:18~19, check for notification service (running), open the app and then the notification arrives.
Here is the log:
(I have a copy of this) 17_11_2015_23_10_35(1).txt

[vanitasvitae]

I can confirm that message delivery is still unreliable :/ Message arrives not until telegram is opened.

[Bubu]

I have another report for delivery again not working after the update. Fixed again with reinstalling.
@slp: Any idea why this might happen?

[ghost]

@bubu what version did you have? What version does the trick?
I'm running debug version which has this issue.

[Bubu]

Not me; but updating from 3.2.2 to 3.2.6 broke it.
Uninstalling 3.26 and reinstalling again fixed it.

[ghost]

I reinstalled a few days ago and still not working.

FLOSS version from F-Droid.

BTW I'm in Moto G Falcon (2013) running CM12.1 a nightly from a couple of days ago. Previously it had an older nightly. I don't know if this makes any difference.

El 23 de noviembre de 2015 9:05:54 CET, Marcus Hoffmann [email protected] escribió:

Not me; but updating from 3.2.2 to 3.2.6 broke it.
Uninstalling 3.26 and reinstalling again fixed it.

---

Reply to this email directly or view it on GitHub:
https://github.com/slp/Telegram-FOSS/issues/36#issuecomment-158873376

[xavihernandez]

Just for info, I repeat that when I'm in EDGE (and then connection is slower I can see written "Waiting for network..." "Connecting..." and "Updating..." message. Seems to prove that connection is lost when application is in background. This is still present in v3.2.6.

[tmelikoff]

I have recently reinstalled my android device and not able anymore to install 3.1.2 from f-droid. Is there any way to find the 3.1.2 apk ( org.telegram.messenger_586.apk ) ?

[Akronix]

@tmelikoff I think this issue is not about that, please open a new specific one. Anyway, why do you want an out-of-date version??

[tmelikoff]

@Akronix you are correct about opening a specific issue but my request is more related to this one. It's not really an issue tho. I reckon the unreliable message delivery as in per this issue subject. I need an outdated version because message delivery is unreliable in versions > 3.1.2

[vanitasvitae]

@tmelikoff Have you tried to check fdroid archive repo in FDroid?

[slp]

@helfio Thanks for your logs.

Looking at them, I'm starting to think that this issue, among others that for some are solved after reinstalling/upgrading, is related to a configuration file disguised as a data file that Telegram creates. Said file, depending on its location, will or will not be removed when uninstalling the app.

Could you please look for a file (using adb or some app) on your phone named "tgnet.dat"? If you find it, uninstall Telegram-FOSS, remove all occurrences of that file, and reinstall it.

I think this could help fixing most of the problems that appeared on the last few releases.

[ghost]

@slp Thank you for your work :)

I've found the file on
/data/data/org.telegram.messenger/files/tgnet.dat
I'm going to proceed as you say and I'll report in a few hours after some testing.
Stay tuned!

[Bubu]

Is this a problem unique to the fdroid version or could this happen with the upstream version too?

[ghost]

Well, just uninstalled and the folder /data/data/org.telegram.messenger it's gone.

[xavihernandez]

@helfio Hi, could you tell me if it has fixed the undelivered messages until application is opened ?

TIA

@slp thanks for your workaround

[ghost]

@xavihernandez It's not fixed.

[xavihernandez]

@helfio thanks for your feedback!!

[Bubu]

@slp While merging the lastest upstream version I just noticed the BUILD_VERSION constant (here) was not updated since 3.1.2. And it is referenced in some user config clearing code and the connection init code. Might that be the root cause?

[ghost]

In spite of I don't know about coding java nor android, I think it worth a try. I can test it if @slp compiles a testing build :)

[slp]

@Bubu Good catch! That could be the root cause, but in that case, removing tgnet.dat should fix the problem.

@helfio I've just uploaded a version with Marcus' changes: https://sinrega.mooo.com/index.php/s/zE82BXl136WykZ3

[ghost]

@slp testing!

[ghost]

Still not working :(

[slp]

@helfio OK, it's time for some serious debugging ;-)

I've just built another apk, this time with debugging options for JNI based tgnet enabled too. This way, it'll generate a *_net.txt logfile, which will show us what is really happening with your connection when the app goes to background.

Please give it a try: https://sinrega.mooo.com/index.php/s/Z5FtGrb87owsjpa

[ghost]

@slp Ok I'll test it. Anything to catch this bug :)

[ghost]

Tests are done. @slp I'd rather prefer send you the logs in private since I don't know if there is sensitive data about my user etc. How do we do this? Email, telegram username...?
Maybe I should delete previous logs too.

[slp]

@helfio Just sent you an email to your account at hiperones.es

[slp]

@helfio Thanks for the logs. Looking at them, I've noticed that the timeout functionality of epoll_wait, called from the JNI code, is not working properly.

I suspect the problem comes from Android's power management. Telegram doesn't set any kind of wakelock or alarm for its push service, so if your phone is working properly, going to deep sleep, and has no other background service waking it periodically, the notifications will never arrive.

At this point you're probably wondering how this works for the official app. Well, the answer is easy, Google's GCM wakes the phone for them.

I've just built a version which re-enables a present but disabled Alarm service. I have no easy way to test it, so I'm not sure if it works, but I think it's worth a shot: https://sinrega.mooo.com/index.php/s/9i8X6kkLwIXHJNt

If still don't fix the problem, I'll try to implement a more sophisticated solution.

[ghost]

Thank you again, and of course testing begins :)

[vanitasvitae]

I do have GApps and GCM on my phone. Telegram Messages do arrive, though they are delayed from time to times.

[ghost]

It works as expected but I get a lot of wakelocks and the battery drains quickly.

It has to be another way since the official app works (AFAIK) just fine without GCM. I mean, somehow Telegram-FOSS must be able to use Telegram upstream own push notification system, right? Maybe something similar to webhook system which is used in bots API. Any idea?

@vanitasvitae do you also use a Google account? I think GCM doesn't work without an account.

[vanitasvitae]

Yes i use GCM with my Google acc.

[xavihernandez]

@helfio @vanitasvitae Telegram-FOSS does not contain GCM part. So even if you enable it (by disabling Telegram Notifications Service), it won't be used. As noticed in description :

_Several proprietary parts were removed from the original Telegram client, including Google Play Services for the location services and HockeySDK for self-updates. Push notifications through Google Cloud Messaging and the automatic SMS receiving features were also removed._

To use GCM you have to download Telegram client from Play Store.

[vanitasvitae]

@xavihernandez Ok, I just thought my installed GCM (I use it for Signal) wakes the phone anyways.

It makes sense, that measages are delayed if GCM does not wakr the phone on incoming telegram messages, but because of other events.

[slp]

@vanitasvitae

It makes sense, that measages are delayed if GCM does not wakr the phone on incoming telegram > messages, but because of other events.

Yup, that's exactly what's happening.

@helfio Telegram-FOSS _is_ using a push connection (you can see it mentioned in your *_net.txt logs). But said push connection requires some maintenance, which is done at fixed intervals (each 180 seconds, in this case) waking the phone. If the phone oversleeps, the connection is broken, and notifications never arrive.

But don't worry. The current approach, while better than holding a permanent partial wakelock, it's still quite crude, as it wakes the phone every 30 seconds, even if there's nothing to do.

I can easily coordinate the timeout from the epoll_wait call in JNI code with the native Alarm, waking up only when it's really needed for maintaining the push connection (aprox. 1/6 less wake ups).

[ghost]

OK so since we know the last beta at least receive notifications well enough, and you can adjust the alarms and so, the next step is testing another beta with this improvements, right?

[slp]

I finally found a phone where I can reproduce this problem. I've been testing a new version with a coordinated Alarm for a few hours, and seems to work fine, while keeping the phone asleep most of the time.

@helfio If you don't mind giving it a try, I've uploaded it here: https://sinrega.mooo.com/index.php/s/ZSpKoZkEkFmzxDb

[ghost]

@slp testing!

[slp]

I've just commited a fix for this to the merge-3.3.1 branch, but I'm not comfortable releasing this on F-Droid until more people as tested it. You can download a binary from here:
https://sinrega.mooo.com/index.php/s/H3CkzzR3Olhy7VM

If you're having this issue, or you don't but want to check if this change has any impact on your experience, please give it a try.

After working on this, I know why everyone is just using GCM. On recent versions of Android, maintaining a background connection in your app is ridiculously hard. The methods for waking the phone are not 100% reliable, and its behavior seem to change between phones.

The change I've implemented keeps the push connection alive, while keeping the CPU in Deep Sleep 95,4% of the time (on my test phone, at least). There are some punctual disconnections/reconnections from time to time, but those seem to be expected and innocuous.

[xavihernandez]

@slp

Thanks you!
Lots of people disable "Telegram Notification Service" to use GCM instead?

Can we test this version, and keep our secret chats ? Or do we have to uninstall and reinstall due to different key signing?

[slp]

@xavihernandez Telegram-FOSS doesn't support GCM, so if you disable the Notification Service, you'll get no notifications at all (well, unless some other app wakes the phone, if Telegram is still in background).

Can we test this version, and keep our secret chats ? Or do we have to uninstall and reinstall due to > different key signing?

Yes, Marcus prepared the merge-3.3.1 branch to be a good neighbour and install alongside the old one, with name "Telegram Beta".

[xavihernandez]

@slp

Lmao I know that, but since you said "After working on this, I know why everyone is just using GCM" I assume you were talking about upstream... which has GCM support.

Oh ok thanks @Bubu too then!

[ghost]

@slp Testing from right now. Thank you again for work on this :)

[ghost]

In my case the app didn't miss a single notification in two days of normal use.

I think notification system is good enough :)

[slp]

@helfio Great, thanks for the report!

[slp]

@vanitasvitae @geileszeuch Version 3.3.1 is available on F-Droid. Could you please check if this issue is fixed for you in that version?

[vanitasvitae]

I already updated yesterday and until now I didnt face the problem again. Seems to work now :)

[Ecron]

@slp when disabling Telegram's Notification Service, a message is shown warning users about using GCM. As Telegram-FOSS doesn't have it, should you change a little that warning saying that, disabling the Service will leave the app without notifications?

[Bubu]

No problems here either with the update.

@Ecron: We could probably update the Messagebox... but then this should be done in all languages which I cannot do.

I'd like to close this issue though. The message box string would be a new one.

[geileszeuch]

No problems so far, but haven't been using Telegram heavily recently. I can tell that search and channels started working. Well done.

[Ecron]

@Bubu You have a point, sir.

[xavihernandez]

@Bubu I still cannot get my messages on time.
I just received a SMS teling me that the sender has sent a message to me @ Telegram. WHen I opened Telegram I had 3 people who had talked to me without I have been noticed :(. Older message was sent 28 minutes before I opened Telegram.

Note that I'm connected in EDGE.
And when I opened Telegram it said "Connecting..." then "Updating..." and ten messages came.

[geileszeuch]

Unfortunately, I experienced similar behaviour. Sometimes receiving several messages from hours ago when opening the app.

[Ecron]

@xavihernandez @geileszeuch did you check what happens if you use official Telegram app? (sorry if it's a silly question :sweat:)

[xavihernandez]

@Ecron Nop I haven't, because I have no Gapps on my phone, then no Play Store.

[geileszeuch]

Since I installed microG the problems seem to be gone. This is exactly what @slp described above.

[xavihernandez]

I got microG installed, but I have hade some delay once. It may be a ponctual issue (I meant on v3.3.1)
I'll let you know, guys, if it happens again.

[slp]

@xavihernandez @geileszeuch I you're still experiencing problems, please consider running the beta version (https://sinrega.mooo.com/index.php/s/H3CkzzR3Olhy7VM) for a while. It will generate some logs I can use to diagnose the Alarm code (as I said before, I'm not surprised of its erratic behavior).

If you don't want to make such logs public (the *_net.txt ones may contain sensitive data), contact me on Twitter (@sergiolpascual).

[slp]

BTW, I've just took a look at microG GCM code (https://github.com/microg/android_packages_apps_GmsCore/blob/master/play-services-core/src/main/java/org/microg/gms/gcm/McsService.java), and they're just setting a repeating Alarm, which wakes the phone every 60 secs (by default, can be configured).

We're trying to be smarter than that, setting the alarm when the JNI's code expects it'll need to do some maintenance on the socket (usually, every 180 secs), but if microG's strategy works better (due to AlarmManager weird behavior on some Android versions), we can adopt (it's way easier than what we're doing right now).

Or, perhaps, we can make it configurable. That would add a purpose to that Notification Service check on the Settings screen ;-)

[geileszeuch]

@slp The last time I tried using that Beta I was unable to create secret chats, always being promted the message "An error occurred". It seems that the Beta does not create a separate account in Android's environment, but uses the same account name " Telegram" in Android's system, which may cause problems when running the official FOSS version and the Beta side by side with different numbers. So, some appropriate renaming in the source could easily fix that.

[xavihernandez]

@slp ok. I'll let you know if it happens again, thanks for your work :)

[xavihernandez]

@slp I just have had a delayed message. Sent at 17:26, received at 18:05. By the way I have received a second message at 18:04. I want to run Beta version but as @geileszeuch said it seems to be a little bit unsuable. I haven't installed it.

Want to notice you that delayed message appeared in Edge. Last time, it was when I was in Edge too. May be due to that. And then it may be due to my ISP. I would like to know if someone else experienced delayed messages in latest version.

[francescortiz]

I faced a similar situation and I solved it by using onTaskRemoved to relaunch the background service.

[ThoAppelsin]

I have resolved this issue by terminating many other (3-4) sessions I had open apparently, somehow, says Telegram Desktop, in the lands I've never been on. I may have used VPN at some point, and that could be the reason I had those sessions on my account.
Anyway, I don't have this issue anymore. A shoutout to devs to investigate that, because I don't think this should cause such issues.

[ryecoder]

I encountered this problem also and found out that it is my Autostart manager shutting it down. Check you Autostart manager guys and whitelist it.

[javotroya]

Oh god I didn't believe my friends when they had this issue. Now i changed my phone and I am having it.

It's been three years and it hasn't been solved.

I had a Motorola MotoG 3G and everything worked fine, I now have a Samsung J7 Prime and it went to Mars. I have to actually open Telegram when I am on the streets, or driving or else. There's no problem when I am home or when I am working cuz I use the desktop or web app and notifications work just fine. Is the phone that doesn't work

[Bubu]

This particular problem has looong been solved. But notifications in recent android versions are hard and there are potentially new problems now.

@javotroya please open a new issue posting your android version (old and new) and if you did exempt Telegram from the power-management-optimizations in android settings.

[FreeSpirit9009]

We seem to be experiencing similar issues using 5.15.0. We also have WeChat and WhatsApp on that same device and their notifications are working fine. Android 9, Huawei Mate 10.

[igorovic]

Seems like the issue is back again. Not on my phone but many friends are reporting this issue.