Mumble: No sound in Mumble after different times in Snapshot under Ubuntu 15.10

[ghost]

In the latest Snapshot in the PPA for Ubuntu is no sound after some times. Reconnecting solves the problem temporarily, but after three seconds or more there is still no sound.

• OS: Ubuntu GNOME 15.10
• Kernel: 4.2.0-23-generic
• Audio Device Manager: PulseAudio
• Mumble version: Don't know, latest snapshot, downgraded to release because of this bug

[mkrautz]

Probably related to the UDP issue in Qt 5.4 and 5.5 (but not 5.5.1 and later):

https://github.com/mumble-voip/mumble/issues/1968

[JoelTroch]

Also probably related to #2040 and #2007 because this issue seems to only appear on GNOME based desktops (like MATE) unless someone prove I'm wrong.

[kimond]

I have the same issue with these configs:
OS: Arch Linux
Kernel: 4.7.6-ARCH
Audio Device Manager: PulseAudio
Windows manager: I3wm with gnome shell

My friend has a bitrate of 93 kbit/s

The server uses Opus codec.

[Krzmbrzl]

Did this also occur with Mumble 1.3?

[nab-os]

I've got the same issue on Arch
Kernel: 5.6.3-arch1-1
Audio Device Manager: PulseAudio
Windows manager: sway
Mumble version: 1.3.0-8

[ghost]

@nab-os Do you know how to reproduce it? Do you use Bluetooth Headphones?

[nab-os]

No, i don't really know, i just lose sound after some time but others can still hear me and i need to reconnect to fix it.
I can't find a way to be verbose enough to log any error.

No, no bluetooth heaphones.
But i have this issue with a simple pulseaudio sink and with this pulseaudio modules:
module-jack-sink
module-remap-sink
module-echo-cancel

I haven't tried with Jack2, but this is hard to test correctly. Sometimes it can take a few hours to happen and sometimes i lose sound every 5 minutes or so (maybe linked to the current network bandwidth ?)

[nab-os]

Ok, that happend to me on Windows last night, 2 times in about 2 hours.
Same hardware, same internet connection.
Maybe this is on Murmur side ?

[Krzmbrzl]

Is it only the voice that is gone or do all sounds of Mumble (e.g. incoming chat, text-to-speech, etc) stop working?

What Mumble version were you using on Windows? And what version was the server? Is this the same server that you experienced this bug on before?

[nab-os]

Good question, i disabled the Mumble sounds, i re-enable them and i come back when i have more inputs.

murmurd --version:
2020-04-23 10:45:39.579 murmurd -- 1.2.18-1+deb9u1

[I reboot and i get the Windows mumble version]

Yes, this is the same server, i also have access to it, if needed.

[Krzmbrzl]

i also have access to it, if needed.

Access as in you are the admin? In that case we might be able to get some debug info into the server that'd show whether the audio packets are even sent to your client or not. This might be very valuable information :point_up:
Did you happen to have compiled the server yourself? Or is this the static binary from the release-page?

[nab-os]

Yes, i am the admin, and this server doesn't have any dependencies, so i can play with it.

Nope, i didn't compile it, i installed it from the package manager.

Mumble version on my Windows install:
1.3.0

My server distribution:
Debian GNU/Linux 9

Started murmur in tmux, i will come back next time i lose sound.

[nab-os]

Bad news, no log on murmur side...

[nab-os]

Is there a way to be more verbose than just the -v option ?

[nab-os]

Ok, so, i only lose voices, i can still hear mumble sounds

[Krzmbrzl]

Bad news, no log on murmur side...

Yeah sure. I said that we could inject logging-code into the application to get those logs. By default Murmur doesn't log where it sends audio packets (would produce huge logs).
That's why I asked if you were the admin. In that case you can switch out the server executable for one that actually does the logging :point_up:
I'll write you the patch and either you are able to compile it yourself (do you know how to do that) or I can compile it for you and give you an executable.

Is there a way to be more verbose than just the -v option ?

No - unless you compile the server in debug-mode

Ok, so, i only lose voices, i can still hear mumble sounds

Okay that means that it indeed seems like a bug in the server that somehow it won't pass audio packets to you anymore...
Are you using some sort of special setup (e.g linked channels or some ACL magic) when this occurs? Or does it (only) occur when whispered/shouted to? Or does it indeed happen in a regular channel with regular users (regular as in no special ACLs in effect)?

[nab-os]

Yeah sure. I said that we could inject logging-code into the application to get those logs. By default Murmur doesn't log where it sends audio packets (would produce huge logs).
That's why I asked if you were the admin. In that case you can switch out the server executable for one that actually does the logging point_up
I'll write you the patch and either you are able to compile it yourself (do you know how to do that) or I can compile it for you and give you an executable.

Ho, ok, i understand, yeah i could totally compile it, no problem.

Okay that means that it indeed seems like a bug in the server that somehow it won't pass audio packets to you anymore...
Are you using some sort of special setup (e.g linked channels or some ACL magic) when this occurs? Or does it (only) occur when whispered/shouted to? Or does it indeed happen in a regular channel with regular users (regular as in no special ACLs in effect)?

Nope, very simple setup.
But now that you mention it, it never happens it the middle of a sentence.
It is always in silent moments.
I tested it with a music bot and now that there is almost always sound playing, there is no bug.

I forgot to say that my mumble server is inside a Proxmox container.
Kernel version: 4.15.18-24-pve
I don't know if it matters, but here it is

[Krzmbrzl]

yeah i could totally compile it, no problem.

Awesome! That makes stuff a lot easier ^^
I'll throw together a path that should add some logging to the server. The you'd only have to clone from my branch, compile it and reproduce the bug with it :)
I'll let you know once the patch is ready

But now that you mention it, it never happens it the middle of a sentence.

:thinking:

I forgot to say that my mumble server is inside a Proxmox container.
Kernel version: 4.15.18-24-pve
I don't know if it matters, but here it is

I don't even know what that is exactly but since it is working sometimes (most times), I don't think this is relevant. We'll see though :)

[nab-os]

This last three days, this issue appearance rate is insane, but i've noticed that it happens more when i moderately/heavily use the bandwidth in parallel.

I tried to use the git version by compiling it but it doesn't solve anything.
At least now I'm ready to compile your debug branch easily.

[Krzmbrzl]

@nab-os here's the patched branch: https://github.com/Krzmbrzl/mumble/tree/debug-server-audio-distribution

This should then log all that's going on during normal audio routing. Note that this'll most likely create a rather huge log. You should also note the time at which point you stop receiving audio in order for you/us to find that point in the log file again :)

Once you reproduced this bug with the patched murmur-version, please upload the server-log (you can cut out all lines that don't contain DebugAudio) here.
Note that usernames will appear in that log. I hope that is not a problem?

[nab-os]

Hmm, i suspect that writing that much log makes murmur slow down, i get this error on client side:

Server connection failed: Server is not responding to TCP pings.

[nab-os]

We should maybe write them into a buffer and output this buffer every N entries ?

[Krzmbrzl]

It certainly does low it down. however I didn't expect the slowdown to be that big. Are you the only one on that server when this happens? What does the server-log say until that point?

We should maybe write them into a buffer and output this buffer every N entries ?

Yeah that's probably a good idea...

[nab-os]

It certainly does low it down. however I didn't expect the slowdown to be that big. Are you the only one on that server when this happens? What does the server-log say until that point?

Yeah i'm alone and it drops around 5 seconds after connection

Here is the tail of the log:
<W>2020-04-28 15:50:23.008 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.053 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.098 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.188 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.249 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.295 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.343 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.410 1 => AudioDebug: Sent audio of "nabos_" to "nabos_", <W>2020-04-28 15:50:23.460 1 => <2:nabos_(-1)> Connection closed: The remote host closed the connection [1] <W>2020-04-28 15:50:23.507 1 => Ending voice thread

[Krzmbrzl]

Yeah i'm alone and it drops around 5 seconds after connection

Ha I haven't tested for 5 secs. I basically just connected to the server, sent some audio frames, looked at the log and was happy xD

[Krzmbrzl]

Okay i just pushed an implementation of your suggestion.

[nab-os]

I don't have the timestamp in front of the logs anymore.
I tried to check on your commits but i think it will be faster if ask your help x)

[nab-os]

My bad, the timestamp is a the beginning of each group

[Krzmbrzl]

My bad, the timestamp is a the beginning of each group

Exactly. I think that is still enough to find the respective parts in the logs :point_up:

[nab-os]

Arf, paquets are well sent by the server...

AudioDebug: Sent audio of "Och2" to "nabos_", "Och2", AudioDebug: Sent audio of "Och2" to "nabos_", "Och2", AudioDebug: Sent audio of "Och2" to "nabos_", "Och2", AudioDebug: Sent audio of "Och2" to "nabos_", "Och2", AudioDebug: Sent audio of "Och2" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2", AudioDebug: Sent audio of "nabos_" to "nabos_", "Och2",

I just have this line in the middle of everything:
AudioDebug: Dropping suppressed packet
It appears only one time about when i lost sound.

[Krzmbrzl]

Oh that's interesting. Did someone enter or leave the channel during that time?

This message is emitted if audio is received but the corresponding user is not registered.
I guess I'll have to log the name of the affected user after all (I was kinda hoping that this case wouldn't occur). I'll let you know once I added the patch.

[nab-os]

Oh that's interesting. Did someone enter or leave the channel during that time?

Nope, just 2 users, we were connected for about 15 minutes.

[Krzmbrzl]

Are you the only one on that server that experiences this bug or does it also occur for your conversation partner?

[nab-os]

Ok, so:

• I experience it from Linux and Windows
• Only one of my partners experience it but only from Linux
• We are a total of 8 using this server

I do not have the worst internet bandwidth, but maybe not very stable ?

[Krzmbrzl]

Hm okay the stability of the connection could be am issue.
As I said though we'll have to do some more logging. I'll create the patch tomorrow :)

[nab-os]

Yeah sure :-)
Thanks for your time ^^

[Krzmbrzl]

Just pushed the patch.

I also realized though that the important bit would be if anywhere in the log it says that it sent the audio of your conversation partner but doesn't list your name as the receiver... If that doesn't happen, it should (try to) send the audio to you as well.

[nab-os]

I'm still only seeing this error:
AudioDebug: Dropping suppressed packet

When i'm bugged, here is the log when 'user2' talks to me:
AudioDebug: Sent audio of "user2" to "nabos_", "user2",
So packets are well sent to me, right ?

[Krzmbrzl]

I'm still only seeing this error:
AudioDebug: Dropping suppressed packet

Oh yeah lol I patched the wrong message xD
This message however means that the sent packet is invalid (exceeds the bandwidth limit). I pushed another patch for this one...

When i'm bugged, here is the log when 'user2' talks to me:
AudioDebug: Sent audio of "user2" to "nabos_", "user2",
So packets are well sent to me, right ?

Yeah. Damn that means I'll have to look deeper into the whole networking stuff :eyes:

It'd be good to know though if this only ever happens, if AudioDebug: Dropping suppressed packet was encountered before. If you could gather some data on that, that'd be great :)
Note though that the message now reads AudioDebug: Dropping packet from ... (too large - ...) (I left out some parts but I guess you'll find it)

[nab-os]

AudioDebug: Dropping packet from "nabos_" (too large - is: 90 allowed: 9000)
I don't understand why this is noted as 'from' me.

I will try to dig into that.

[nab-os]

Ok,
AudioDebug: Dropping packet from "nabos_" (too large - is: 90 allowed: 9000)
This error appears from time to time without triggering the bug, but only for my client.

[Krzmbrzl]

I don't understand why this is noted as 'from' me.

It seems as if an audio packet of yours was rejected because of some bandwidth limitation. I just realized though that the log message phrasing is a bit misleading.I also realized that I don't understand the logic behind that particular limitation, so let's stick with what we have xD

This error appears from time to time without triggering the bug, but only for my client.

Well then this is most likely yet another curiosity but (hopefully) unrelated)

[nab-os]

So in what direction should we go ?
Deeper into the server networking layers ? Or trying to log incoming packets on client side ?
Are you sure it comes from the server ?

[Krzmbrzl]

Or trying to log incoming packets on client side ?

I guess we should do that first but I don't think there is any routine inside Mumble that drops a received audio packet... Though I might be wrong.

I'll extend the patched branch with the respective log-statements. This could take a few days though as I have a lot going on right now :point_up:

Are you sure it comes from the server ?

No. That was just my best guess ^^

[nab-os]

I've tried to dump network packets with tcpdump and i effectively see UDP packets as my partner speaks.
I tried to dive into the networking stuff of Mumble, but no success for now.
I will continue in that direction.

This bug is driving me nuts, by moments it is just unusable.
(Ho and forget what I said about the music bot to prevent the connection to drop, maybe it was just luck).

[Krzmbrzl]

i effectively see UDP packets as my partner speaks.

Ah that's interesting. This indicates that at least the server is working as expected. Thus I suspect the issue to be somewhere on the client side.

This bug is driving me nuts, by moments it is just unusable.

Yeah I can definitely understand that. And I'm sorry that it took me so long for coming back to this issue (but as I said I have a lot going on atm)...

Anyways: I added debug logs to the client's code and pushed them to my debug-branch (the same you were using before) so if you could compile your client from that, that'd be great. You need to compile it in debug mode though as otherwise you'll probably get syntax errors :)

Now every received UDP packet is being logged and I drilled a decent amount into the processing and added logs to a lot of places where an audio frame could potentially be dropped. If that happens, you should find something in the console :point_up:

[nab-os]

And I'm sorry that it took me so long for coming back to this issue (but as I said I have a lot going on atm)...

Don't be sorry, you have no obligations to do anything. I'm very glad you're helping me, but take care of yourself in first place.

What is the required Celt version ?
I have to change CELT_SET_VBR_RATE to CELT_SET_BITRATE in src/mumble/AudioInput.cpp
Maybe there is a problem down the pipe linked to this dependency ?
Right know i have this version: 0.11.3-4

I'm compiling it, i'll come back when i have more inputs.

[Krzmbrzl]

Right know i have this version: 0.11.3-4

This version of Celt is unsupported. You'll have to use the bundled Celt version 0.7 (see #3727)

[nab-os]

Right know i have this version: 0.11.3-4

This version of Celt is unsupported. You'll have to use the bundled Celt version 0.7 (see #3727)

Thx, didn't see that.

Ok, I took some time because I couldn't understand what I was experiencing, it wasn't consistent.
Here is the results:
I did all my tests with the same partner
When everything is normal, incoming UDP packets I see with tcpdump are 108 bytes long.
But when the bug appear, the packets are only 24 bytes long, and I've got no debug logs on Mumble side.

[Krzmbrzl]

But when the bug appear, the packets are only 24 bytes long, and I've got no debug logs on Mumble side.

Okay now that is very interesting. This seems to me as if the issue was on the server after all. This sounds a lot like invalid packets being sent by the server :thinking:

[Krzmbrzl]

Could you monitor the packets that are sent from your server to see whether the server is indeed only sending 24 bytes packages when the bug appears?

Ah wait a sec. I just checked where I put debug logs and the ones that I didn't log are pings. Are these packets received every 5 seconds or so by any chance? Or is there basically a continuous stream of these packets? In that case I'd have to add a debug message to the ping as well in order to check whether these short packages are in fact recognized as pings...

[nab-os]

Could you monitor the packets that are sent from your server to see whether the server is indeed only sending 24 bytes packages when the bug appears?

Yeah sure

Ah wait a sec. I just checked where I put debug logs and the ones that I didn't log are pings. Are these packets received every 5 seconds or so by any chance? Or is there basically a continuous stream of these packets?

No, I took the time to ignore pings from my results.
The 24 bytes long packets were arriving continuously as a stream.

[Krzmbrzl]

No, I took the time to ignore pings from my results.
The 24 bytes long packets were arriving continuously as a stream.

Okay. I added a debug message for receiving pings to the respective branch. That way you should see it in Mumble's logs if the packets are in fact (wrongly?) interpreted as pings :thinking:

[nab-os]

Ok, two points:

• Today the 24 bytes packets are not appearing during the bug, but I noticed that the server selection menu was generating requests and that the responses packets are 24 bytes long. I have no idea if this is just a coincidence.
• By listening to the server outgoing packets, I can see that the packets are well sent...
  Does this mean that this is unrelated to mumble and this is a network issue ?
  Or is there some kind of packet routing handshake between the client and the server that could be unstable or something like that ? Like in WebRTC with ICE ?
  (I don't know what mumble uses)

[Krzmbrzl]

Today the 24 bytes packets are not appearing during the bug, but I noticed that the server selection menu was generating requests and that the responses packets are 24 bytes long. I have no idea if this is just a coincidence.

Afaik the server connect dialog should only send and receive pings if it's opened :thinking:

Does this mean that this is unrelated to mumble and this is a network issue ?

Without being proficient in the whole networking stuff I think from what you reported this might be a possibility, yes. I mean after all if the packages are sent from the server but not received by your client, that means that they have to get lost somewhere.
Unless of course the server sends the packages to the wrong IP or something like that...

Any chance your IP does indeed change while being connected to the server? Also it should be possible to inspect the sent packages to see where they are sent to (afaik they should have a non-encrypted header file that contain the destination). Do you happen to know how something like this could be done? :)

Or is there some kind of packet routing handshake between the client and the server that could be unstable or something like that ? Like in WebRTC with ICE ?

I actually don't know. From what I have seen I think we use some Qt stuff for managing the connection but I assume that at some point there has to be some sort of handshake-thingy

[nab-os]

Any chance your IP does indeed change while being connected to the server?

No, my IPs are stable

Also it should be possible to inspect the sent packages to see where they are sent to (afaik they should have a non-encrypted header file that contain the destination). Do you happen to know how something like this could be done? :)

Ho, yeah sure, I can do that.

[Krzmbrzl]

Ho, yeah sure, I can do that.

Then I guess that could be worth a try. If the IPs are the same for the working as well as the non-working packages then I think we can safely assume that the server seems to be sending the packets correctly.

In that case I think it pretty much only can be a network problem, if you say you don't even see the packages arriving at the client :thinking:

[Krzmbrzl]

I just recently experienced this issue for the first time myself. In my case it was connected to opening the settings, changing some audio input settings and then closing the settings by hitting "Ok". After that I had to restart Mumble in order to be able to hear my conversation partner again (didn't try if simply reconnecting to the server would have fixed the issue though).

[Krzmbrzl]

@nab-os what version of Qt is your client using? (This can be found out via the Mumble's Help menu)

@toby63 mentioned that this issue here might be connected to #1968
If this is indeed the case, everything should work fine when using TCP for audio as well. For that you'd have to go to Settings > Network and check "force TCP".

Could you try that out and report back, please? :)

[nab-os]

@nab-os what version of Qt is your client using? (This can be found out via the Mumble's Help menu)

My client uses Qt 5.14.2

@toby63 mentioned that this issue here might be connected to #1968
If this is indeed the case, everything should work fine when using TCP for audio as well. For that you'd have to go to Settings > Network and check "force TCP".

Could you try that out and report back, please? :)

Yes sure, I will try that too

[nab-os]

I have been less using Mumble these weeks, but the few times I have, the bug did not appear.
I have to say that I reworked my LAN architecture so it may be related.
Will keep you updated if this issue reappears.

[Krzmbrzl]

Do you mind sharing what changes you made in your LAN architecture? If this really was the case it might be important for other users with the same problem :point_up:

[nab-os]

Ho, previously I had my modem connected to a router, and my devices were connected to this router (mostly ethernet except for phones).
Now, my modem is connected to a gateway server and the router now only acts as a wifi hotspot behind this gateway and my devices are connected directly to this server too.
So basically my main computer is not passing through the router anymore. This is the only point I can logically see in this change. This is a Netgear WAC124. Maybe there is some kind of bad udp implementation or something like this.
But this is maybe just luck and it will reappear soon. I will try to stress out my connection to see if it changes anything.

[toby63]

I experienced this issue again.
I can't hear others, but they can hear me and talk to each other.
Forcing tcp mode did not help.
Restart did not help.

Unfortunately I hadn't had the time to try or debug more.
But something is really wrong.
I think there has to be a bug and even if it is somehow network related, mumble should be capable to deal with it some way, at least an error message.

[Krzmbrzl]

@toby63 next time it happens please check the incoming and outgoping UDP packets on your machine. Previous investigations of nab-os indicated that the client might not even receive packages anymore...

[nab-os]

Still no bug on my side. I tried to put some load on the connection but nothing happened.
I'm afraid that I will not be able to help you debug anymore :-/

[Krzmbrzl]

@nab-os well I guess it's a good thing for you though :D
This does indicate though that this is indeed an error that only occurs with certain network setups... :thinking:

[nab-os]

Ho ! I could try to isolate the router and build up a test network !
I will try that.

[AdamSzopa]

I seem to be having the same issue, and in fact my experience does make me think it might be a network setup related.

I started using Mumble a few weeks ago and had no issues. Then I needed to change my home router (the previous one died of old age) and I started experiencing the discussed bug - I suddenly stop hearing people, they can hear me and each other. A quick reconnect usually helps.

The router change I did was from a old TP-Link N600 to a quite new D-link DIR-853. I looked in the router logs and found nothing suspicious, and I didn't see any obvious settings/rules that might be related.

[nab-os]

Sorry, moved out and I had a lot of issues with my ISP, so I could'nt test anything.