Docker-jitsi-meet: Jicofo health checks fail since upgrade to stable-4857

[cedricroijakkers]

I've just upgraded to the latest container version stable-4857, and now the Jicofo container fails it's health checks with the following error:

Jicofo 2020-07-22 13:22:26.803 SEVERE: [32] org.jitsi.impl.protocol.xmpp.ChatRoomImpl.log() No presence packet obtained yet
Jicofo 2020-07-22 13:22:26.803 INFO: [32] org.jitsi.jicofo.FocusManager.log() Exception while trying to start the conference
net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:298)
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:209)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.joinTheRoom(JitsiMeetConferenceImpl.java:581)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.start(JitsiMeetConferenceImpl.java:404)
    at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:477)
    at org.jitsi.jicofo.health.Health.check(Health.java:200)
    at org.jitsi.jicofo.health.Health.performCheck(Health.java:133)
    at org.jitsi.health.AbstractHealthCheckService.run(AbstractHealthCheckService.kt:155)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor.run(RecurringRunnableExecutor.java:216)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor.runInThread(RecurringRunnableExecutor.java:292)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor.access$000(RecurringRunnableExecutor.java:36)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor$1.run(RecurringRunnableExecutor.java:328)
Caused by: org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from conference.meet.jitsi: XMPPError: not-allowed - cancel
    at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:132)
    at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:263)
    at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:214)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.discoverInfo(ServiceDiscoveryManager.java:540)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.discoverInfo(ServiceDiscoveryManager.java:506)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.supportsFeatures(ServiceDiscoveryManager.java:748)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.supportsFeatures(ServiceDiscoveryManager.java:744)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.supportsFeature(ServiceDiscoveryManager.java:740)
    at org.jivesoftware.smackx.muc.MultiUserChatManager.providesMucService(MultiUserChatManager.java:361)
    at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:311)
    at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:498)
    at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:444)
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:240)
    ... 11 more
Jicofo 2020-07-22 13:22:26.807 SEVERE: [32] org.jitsi.jicofo.health.Health.log() Health check failed in PT0.006S:
net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:298)
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:209)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.joinTheRoom(JitsiMeetConferenceImpl.java:581)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.start(JitsiMeetConferenceImpl.java:404)
    at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:477)
    at org.jitsi.jicofo.health.Health.check(Health.java:200)
    at org.jitsi.jicofo.health.Health.performCheck(Health.java:133)
    at org.jitsi.health.AbstractHealthCheckService.run(AbstractHealthCheckService.kt:155)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor.run(RecurringRunnableExecutor.java:216)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor.runInThread(RecurringRunnableExecutor.java:292)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor.access$000(RecurringRunnableExecutor.java:36)
    at org.jitsi.utils.concurrent.RecurringRunnableExecutor$1.run(RecurringRunnableExecutor.java:328)
Caused by: org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from conference.meet.jitsi: XMPPError: not-allowed - cancel
    at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:132)
    at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:263)
    at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:214)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.discoverInfo(ServiceDiscoveryManager.java:540)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.discoverInfo(ServiceDiscoveryManager.java:506)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.supportsFeatures(ServiceDiscoveryManager.java:748)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.supportsFeatures(ServiceDiscoveryManager.java:744)
    at org.jivesoftware.smackx.disco.ServiceDiscoveryManager.supportsFeature(ServiceDiscoveryManager.java:740)
    at org.jivesoftware.smackx.muc.MultiUserChatManager.providesMucService(MultiUserChatManager.java:361)
    at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:311)
    at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:498)
    at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:444)
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:240)
    ... 11 more

What am I missing here? Prosody is running, and all other configuration environment variables have not been changed since stable-4627-1 which worked fine?

[saghul]

Have you purged your config volume?

[cedricroijakkers]

I'm running in Kubernetes, my config volumes are deleted every time the pod restarts. So I basically start with a clean slate every time again.

[paweldomas]

It may be related to [this] change, but not sure exactly how. From the stack trace it looks like it's failing on the feature discovery. Maybe some features are not enabled in the Prosody config.
https://xmpp.org/extensions/xep-0030.html#errors

[saghul]

@cedricroijakkers Can you reproduce it with the Docker Compose setup? The Kubernetes example is community maintained, doesn't get tested before a release.

[fais3000]

@saghul @cedricroijakkers Kind of related. Since upgrading to container version stable-4857. The jitsi rooms will go in a state where if 2 people join the room the process crashes and it reloads the webview. Running via docker-compose setup with new configs.

[saghul]

Please provide logs.

[fais3000]

Please provide logs.

So it happens randomly after x number of hours. Right now I restarted. WIll record the logs and share when it happens again.

[saghul]

I just realized that @paweldomas might be right here. Our muc is called "muc" not "conference".

@cedricroijakkers can you try adding org.jitsi.jicofo.XMPP_MUC_COMPONENT_PREFIX=muc to jicofo's sip-communicator.properties file?

[cedricroijakkers]

That does indeed do the trick. I've created a quick build myself by adding the line to the configuration file and building a new container.

Do you want me to make a PR? Should the value be configurable? Should it be set to "conference" by default and make it configurable with an environment variable or should it just be "muc" by default?

[cedricroijakkers]

@prayagsingh I don't think that is the same issue. This happens when jicofo cannot connect to jvb. Are you sure your jvb instance is up and running? I get this issue sporadically, when jicofo has started before jvb is started, but it dissapears as soon as the connection between jicofo and jvb is re-established.

[prayagsingh]

Actually the health-check issue is still there but you are correct that the jicofo started first and jvb later hence that issue. That's why I deleted my comment(wrong logs). I'm trying what @saghul has shared.

[cedricroijakkers]

If you want to try the configuration fix, you can replace your jicofo image with cedricroijakkers/jicofo:stable-4857, that contains the configuration change. I'm looking into creating a PR to get this in the official repo.

[cedricroijakkers]

I've created PR https://github.com/jitsi/docker-jitsi-meet/pull/706 to fix this issue.