Docker-jitsi-meet: k8s: Can not invite participant -- no bridge available.

[josemolmomillan]

Hello
I am deploying docker-jitsi-meet in a K8s cluster and all seems it's working fine, except when two users are trying to connect to the same room, I got this ERROR message in jicofo logs. It seems it's not possible to access to jvb.

Jicofo 2019-09-30 19:49:18.743 SEVERE: [34] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant -- no bridge available.

We are using un ingress-nginx proxy to redirect 4443 jvb from external browsers to the jvb bridge.

is someone experiencing this issue in K8s environments?

Thanks in advance

These are the config for jvb deployment and service.

---

kubectl describe svc jvb-tcp -n develop
Name: jvb-tcp
Namespace: develop
Labels: service=jvb
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"service":"jvb"},"name":"jvb-tcp","namespace":"develop"},"spec":{"ports":[{"...
Selector: service=jvb
Type: ClusterIP
IP: 10.0.3.190
Port: 4443 4443/TCP
TargetPort: 4443/TCP
Endpoints: 10.52.2.173:4443
Port: 10000 10000/UDP
TargetPort: 10000/UDP
Endpoints: 10.52.2.173:10000
Session Affinity: None

Events:

jmolmo@jmolmo-pc:~/git/prevvy-deployments/kubernetes/prevvy_video$ kubectl describe deployment jvb -n develop
Name: jvb
Namespace: develop
CreationTimestamp: Mon, 30 Sep 2019 11:47:57 -0700
Labels: service=jvb
Annotations: deployment.kubernetes.io/revision=1
kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"service":"jvb"},"name":"jvb","namespace":"develop"},"spe...
Selector: service=jvb
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: service=jvb
Containers:
jvb:
Image: jitsi/jvb
Ports: 10000/UDP, 4443/TCP
Host Ports: 0/UDP, 0/TCP
Environment:
XMPP_SERVER: 10.0.7.47
XMPP_DOMAIN: XXXX.co
XMPP_AUTH_DOMAIN: XXXX.co
JVB_COMPONENT_SECRET: s3cr3t
JVB_STUN_SERVERS: stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302
JICOFO_AUTH_USER: focus
TZ: Europe/London
JVB_BREWERY_MUC: jvbbrewery
JVB_TCP_HARVESTER_DISABLED: true
JVB_TCP_PORT: 4443
Mounts:
Volumes:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets:
NewReplicaSet: jvb-7ddff65fbd (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------

Normal ScalingReplicaSet 12m deployment-controller Scaled up replica set jvb-7ddff65fbd to 1

Jicofo configuration
Name: jicofo
Namespace: develop
CreationTimestamp: Mon, 30 Sep 2019 11:47:58 -0700
Labels: service=jicofo
Annotations: deployment.kubernetes.io/revision=1
kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"service":"jicofo"},"name":"jicofo","namespace":"develop"...
Selector: service=jicofo
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: service=jicofo
Containers:
jicofo:
Image: jitsi/jicofo
Port:
Host Port:
Environment:
XMPP_SERVER: 10.0.7.47
XMPP_DOMAIN: XXXX.co
XMPP_AUTH_DOMAIN: auth.video.develop.prevvy.co
JICOFO_COMPONENT_SECRET: s3cr37
JICOFO_AUTH_USER: focus
JICOFO_AUTH_PASSWORD: passw0rd
TZ: Europe/London
JVB_BREWERY_MUC: jvbbrewery
JVB_TCP_HARVESTER_DISABLED: true
JVB_TCP_PORT: 4443
Mounts:
Volumes:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets:
NewReplicaSet: jicofo-7599dbd885 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 20m deployment-controller Scaled up replica set jicofo-7599dbd885 to 1

[sapkra]

The internal and external port for the JVB has to be the same because jicofo is sending the port of the right videobridge to the client and the client is trying to connect to the same port on your cluster. I don't know if it's part of your problem but it may be helpful for someone.
I'm using a NodePort to make my jvb available to the outside and everything works.

My Service looks like this:
https://github.com/jitsi/docker-jitsi-meet/blob/e8c81feda752c8c0181725a7db501c80a2fa7610/okd/jvb/jvb-service.yaml

But I still don't understand how your redirect works and why your are redirecting 4443?
Additionally it seems like you have disabled 4443/TCP anyway by setting JVB_TCP_HARVESTER_DISABLED to true.

[josemolmomillan]

Thanks Paul for your advice

I have modified my jvb-service based on your NodePort service.. I am using 30000 internally and externally. I am assigning DOCKER_HOST_ADDRESS to jvb-deployment.yaml and JVB_TCP_HARVESTER_DISABLED=false in jvb and jicofo.

I have deactivated my UDP/TCP redirect in my ingress-nginx. Right now, there is just rules for jitsi web.

But, sadly I have the same issue about " Can not invite participant -- no bridge available. ". It seems jicofo is not possible to connect to jvb.

From inside of jicofo container I have tested tcp/udp connection sending packages and it seems to work fine.
root@jicofo-64889b5fc4-tcxll:/# echo "PING" > /dev/tcp/jvb/30000
bash: connect: Connection refused
bash: /dev/tcp/jvb/30000: Connection refused
#OK bc. JVB_TCP_HARVESTER_DISABLED=true

root@jicofo-64889b5fc4-tcxll:/# echo "PING" > /dev/udp/jvb/30000
UDP port is working fine

This is the logs for kubectl commands:
https://drive.google.com/open?id=1myzpD4dbfBQRlOmY0tMjR2mU4HK3T7Zs

These are the yaml files:
https://drive.google.com/open?id=1aW9-pwBvPN-_Eh2JuE1LAcz5uS9GnaOD

I think for some reason jicofo is no connecting to jvb, although there is connectivity.

could it be HOST for jvb?

Thanks in advance

[sapkra]

Are both jicofo and jvb connected to prosody? Have your tried to restart the pods?
Can you send the jicofo and prosody logs please?

btw. my entire manifests you can find here - there are just some minor changes related to OpenShift like that I'm using Routes instead of Ingress.

But right now I couldn't find any related problems in your manifest.

[dimm0]

I've been trying to run this as well for almost a year, hitting the same problem..
What's the advantage of running separate pods for each service? Why not a single pod with 4 containers, accessing each other as localhost?

(my current state is here: https://gitlab.nautilus.optiputer.net/prp/jitsi/blob/master/common-deploy.yaml)

[dimm0]

Also opened the UDP port on the node

[saghul]

I noticed you used a hostname as the DOCKER_HOST_ADDRESS value. I'm not sure we support that, can you please try using an IP address instead?

It's the only thing that comes to mind, sorry.

[dimm0]

Same error.
Maybe jitsi gurus could create a minimal working example for kubernetes? One pod should be much easier than 4 and several services...

[aaronkvanmeerten]

Yes the current spike at k8s came when we were first attempting a k8s proof of concept. I like the idea of a single pod better, although as seen from the languishing current PoC I haven't had a ton of time to experiment with k8s recently. I will attempt this single pod chart at some point when I next touch this part of the infrastructure.

[dimm0]

Great, thanks!

[sapkra]

I have a working example in my OpenShift PR, like I've previously commented. A single Pod approach will not be cloud native in my opinion and more like a hack because you will not be able to scale the deployment properly.
There are also some other commuity members which are using a jitsi deployment on kubernetes.

[dimm0]

I agree regarding the scaling, but if single node is enough to run it, and just need something to start, it should work fine. Just compare the number of files in your example with my single deploy.
I'll try yours at some point, but it's looking huge..

[dimm0]

Jicofo 2019-12-11 10:27:07.418 SEVERE: [32] org.jitsi.meet.ComponentMain.log() java.net.ConnectException: Connection refused (Connection refused), host:localhost, port:5347
org.xmpp.component.ComponentException: java.net.ConnectException: Connection refused (Connection refused)
    at org.jivesoftware.whack.ExternalComponent.connect(ExternalComponent.java:296)
    at org.jivesoftware.whack.ExternalComponentManager.addComponent(ExternalComponentManager.java:242)
    at org.jivesoftware.whack.ExternalComponentManager.addComponent(ExternalComponentManager.java:222)
    at org.jitsi.meet.ComponentMain.lambda$getConnectCallable$0(ComponentMain.java:286)
    at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:193)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.jivesoftware.whack.ExternalComponent.connect(ExternalComponent.java:174)
    ... 11 more

Doesn this error in jicofo logs mean jvb is not available? How can I debug this? No errors in jvb

[saghul]

No, that error means Jicofo couldn't connect to prosody. Jicofo makes a component connection to prosody on port 5347. From the above log: host:localhost, port:5347 I guess it's not localhost for you, is it?

[dimm0]

It is localhost, all containers are in the same pod.

Jifoco has the error for cert:

<no value>:tls      error   Error creating context for c2s: No certificate present in SSL/TLS configuration for <no value>
<no value>:tls      error   Error creating contexts for s2sin: No certificate present in SSL/TLS configuration for <no value>

Am I required to have one?

The end of log is a bunch of:

c2s55fae5ecba50                           info  Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
c2s55fae5ecba50                           info  Client disconnected: connection closed
c2s55fae635ab70                           info  Client connected

[saghul]

<no value> this suggests the template wasn't applied correctly, check the prosody config files.

[dimm0]

check the prosody config files.

Which config files, and check for what? Here's all prosody config: https://gitlab.nautilus.optiputer.net/prp/jitsi/blob/master/common-deploy.yaml#L57-87

[dimm0]

Ok, I see it...

Component "<no value>" "muc"
    modules_enabled = {
        "ping";

    }
    storage = "memory"
    muc_room_cache_size = 1000

in conf.d/jitsi-meet.cfg.lua

[dimm0]

Solved that one by adding XMPP_INTERNAL_MUC_DOMAIN=internal-muc.meet.jitsi. No errors now in logs, but still the same problem

[sapkra]

I think you have to use the JVB_AUTH_USER and JVB_AUTH_PASSWORD instead of JVB_COMPONENT_SECRET.

[dimm0]

Wow, it works now!! Thanks! =)

[saghul]

Great to hear! @dimm0 is your config something you'd be willing to share? If so please make a PR, some others may find it useful!

[dimm0]

Will do!

[dmastag]

@dimm0 could you share how you got it working ?

[dimm0]

https://github.com/jitsi/docker-jitsi-meet/tree/master/examples/kubernetes