After some days - No Bridge available

[njamblauch]

What happened?

Hello,
i use Jitsi Meet in a local Network, just for local use.
Everything is running well, but after a few days / a week the Meeting disconnect immediately, when a second user joins.
After a restart, everything is running well again. Until some days later, this Problem repeat again.

jicofo.log
Jicofo 2024-09-02 05:43:03.212 WARNUNG: [5505] JvbDoctor$HealthCheckTask.doHealthCheck#271: Health check failed for: Bridge[[email protected]/6374ced8-3e48-4588-9a44-cc227164cea0, version=2.3.160-g97a1f15b, relayId=null, region=null, stress=0,00]: No health checks performed recently, the last result was PT1M30.852164S ago.
...
Jicofo 2024-09-02 06:41:16.229 WARNUNG: [6170] BridgeSelector.selectBridge#182: There are no operational bridges.
Jicofo 2024-09-02 06:41:16.344 SCHWERWIEGEND: [6170] [[email protected] meeting_id=5bacb78f-7024-46c1-8dee-ee6314bf48f0 participant=8b9d027a] ParticipantInviteRunnable.doRun#218: Can not invite participant, no bridge available.

In the jvb.log is nothing on this date.

Status Jitsi-videobridge2:
● jitsi-videobridge2.service - Jitsi Videobridge
Loaded: loaded (/usr/lib/systemd/system/jitsi-videobridge2.service; enabled; preset: enabled)
Active: active (running) since Fri 2024-08-23 10:35:21 CEST; 1 week 2 days ago
Main PID: 1234 (java)
Tasks: 56 (limit: 65000)
Memory: 467.3M (peak: 504.1M)
CPU: 2h 14min 14.871s
CGroup: /system.slice/jitsi-videobridge2.service
└─1234 java -Xmx3072m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djdk.t>

sip-communicator.properties:
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.DISABLE_STUN_HARVESTER=true
#org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.192.70.135.142
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=********
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.192.70.135.142
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=6374ced8-3e48-4588-9a44-cc227164cea0
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.70.135.142
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=192.70.135.142

Any Ideas where this problem comes from?

Thanks a lot.

Platform

• Chrome (or Chromium based)
• Firefox
• Safari
• Other desktop browser
• Android browser
• iOS browser
• Electron app
• Android mobile app
• iOS mobile app
• Custom app using a mobile SDK

Browser / app / sdk version

Jitsi-Meet: 2.0.9646

Relevant log output

No response

Reproducibility

• The problem is reproducible on meet.jit.si

More details?

Own local Server on Kubuntu 24.04 (VM) in VirtualBox.

[damencho]

Check first the jicofo logs in /var/log/jitsi and find the time when the bridge was removed from the brewery or became unhealthy, then check in the same folder the jvb logs, what do you see?

[njamblauch]

Thank you very much for help.

The bridge was removed today in the morning.

Jicofo 2024-09-02 05:43:03.212 WARNUNG: [5505] JvbDoctor$HealthCheckTask.doHealthCheck#271: Health check failed for: Bridge[[email protected]/6374ced8-3e48-4588-9a44-cc227164cea0, version=2.3.160-g97a1f15b, relayId=null, region=null, stress=0,00]: No health checks performed recently, the last result was PT1M30.852164S ago.
Jicofo 2024-09-02 05:43:03.242 INFORMATION: [37] JvbDoctor.bridgeRemoved#105: Stopping health-check task for: Bridge[[email protected]/6374ced8-3e48-4588-9a44-cc227164cea0, version=2.3.160-g97a1f15b, relayId=null, region=null, stress=0,00]

But on that time, there is nothing in the jvb.log.

The last entry is my test on Saturday. There it works.

But here is a warning.

JVB 2024-08-31 19:23:45.981 WARNUNG: [2190] [confId=4aa2f585558dc3ce [email protected] meeting_id=842154c0 epId=e6bf894e stats_id=Adolphus-WiI local_ufrag=ci74k1i6knd082 ufrag=ci74k1i6knd082 name=stream componentId=1] MergingDatagramSocket.doRemove#349: Removing the active socket. Won't be able to send until a new one is elected.
JVB 2024-08-31 19:23:45.991 INFORMATION: [2202] [confId=4aa2f585558dc3ce [email protected] meeting_id=842154c0 epId=e6bf894e stats_id=Adolphus-WiI local_ufrag=ci74k1i6knd082 ufrag=ci74k1i6knd082 name=stream componentId=1] MergingDatagramSocket.close#142: Closing.
JVB 2024-08-31 19:23:45.993 INFORMATION: [2202] [confId=4aa2f585558dc3ce [email protected] meeting_id=842154c0 epId=e6bf894e stats_id=Adolphus-WiI] Endpoint.expire#1131: Expired.
JVB 2024-08-31 19:23:45.993 INFORMATION: [2202] [confId=4aa2f585558dc3ce [email protected] meeting_id=842154c0] Conference.updateStatisticsOnExpire#617: expire_conf,duration=25

Next entry is my restart today. With some warnings...

JVB 2024-09-02 09:08:14.025 INFORMATION: [1] JitsiConfig.#47: Initialized newConfig: merge of /etc/jitsi/videobridge/jvb.conf: 1,application.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/application.conf: 1,system properties,reference.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/reference.conf: 1,reference.conf @ jar:file:/usr/share/jitsi-videobridge/lib/jitsi-media-transform-2.3-160-g97a1f15b.jar!/reference.conf: 1,reference.conf @ jar:file:/usr/share/jitsi-videobridge/lib/ice4j-3.0-72-g824cd4b.jar!/reference.conf: 1
JVB 2024-09-02 09:08:14.109 INFORMATION: [1] ReadOnlyConfigurationService.reloadConfiguration#51: loading config file at path /etc/jitsi/videobridge/sip-communicator.properties
JVB 2024-09-02 09:08:14.174 INFORMATION: [1] JitsiConfig.#68: Initialized legacyConfig: sip communicator props (no description provided)
JVB 2024-09-02 09:08:16.293 INFORMATION: [1] JvmMetrics.#36: Detected GC type G1
JVB 2024-09-02 09:08:16.408 INFORMATION: [1] MetricsUpdater.addUpdateTask#45: Scheduling metrics update task with interval PT5S.
JVB 2024-09-02 09:08:16.617 INFORMATION: [1] JitsiConfig$Companion.reloadNewConfig#94: Reloading the Typesafe config source (previously reloaded 0 times).
JVB 2024-09-02 09:08:16.652 INFORMATION: [1] MainKt.main#77: Starting jitsi-videobridge version 2.3.160-g97a1f15b
JVB 2024-09-02 09:08:16.888 INFORMATION: [1] UlimitCheck.printUlimits#109: Running with open files limit 65000 (hard 65000), thread limit 65000 (hard 65000).
JVB 2024-09-02 09:08:18.227 INFORMATION: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Adding a static mapping: StaticMapping(localAddress=192.70.135.142, publicAddress=192.70.135.142, localPort=null, publicPort=null, name=null)
JVB 2024-09-02 09:08:18.473 INFORMATION: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.maybeAdd: Discarding a mapping harvester: org.ice4j.ice.harvest.StaticMappingCandidateHarvester(face=192.70.135.142:9/udp, mask=192.70.135.142:9/udp)
JVB 2024-09-02 09:08:18.500 INFORMATION: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=939ms). stunDiscoveryFailed=false
JVB 2024-09-02 09:08:18.531 INFORMATION: [1] org.ice4j.ice.harvest.AbstractUdpListener.: Initialized AbstractUdpListener with address 192.70.135.142:10000/udp. Receive buffer size 10485760 (asked for 10485760)
JVB 2024-09-02 09:08:18.547 INFORMATION: [1] org.ice4j.ice.harvest.SinglePortUdpHarvester.: Initialized SinglePortUdpHarvester with address 192.70.135.142:10000/udp
JVB 2024-09-02 09:08:18.564 INFORMATION: [1] Smack.initialize#30: Setting XML parsing limits.
JVB 2024-09-02 09:08:18.647 INFORMATION: [1] Smack.initialize#42: Using JitsiXmppStringprep.
JVB 2024-09-02 09:08:20.870 INFORMATION: [21] [hostname=localhost id=shard] MucClient.initializeConnectAndJoin#288: Initializing a new MucClient for [ org.jitsi.xmpp.mucclient.MucClientConfiguration id=shard domain=auth.192.70.135.142 hostname=localhost port=null username=jvb mucs=[[email protected]] mucNickname=6374ced8-3e48-4588-9a44-cc227164cea0 disableCertificateVerification=false]
JVB 2024-09-02 09:08:21.145 INFORMATION: [1] TaskPools.#87: TaskPools detected 4 processors, creating the CPU pool with that many threads
JVB 2024-09-02 09:08:21.603 INFORMATION: [1] HealthChecker.start#122: Started with interval=60000, timeout=PT1M30S, maxDuration=PT3S, stickyFailures=false.
JVB 2024-09-02 09:08:21.615 INFORMATION: [1] MucPublisher.start#36: Starting with interval PT5S.
JVB 2024-09-02 09:08:21.631 INFORMATION: [1] MainKt.main#124: Starting public http server
JVB 2024-09-02 09:08:22.143 WARNUNG: [1] ColibriWebSocketService.#51: Websockets enabled, but no domains specified: URLs=[wss://192.70.135.142:443/colibri-ws/default-id], Relay URLs=[]
JVB 2024-09-02 09:08:22.586 WARNUNG: [22] [hostname=localhost id=shard] MucClient.setPresenceExtensions#467: Cannot set presence extension: not connected.
JVB 2024-09-02 09:08:23.081 INFORMATION: [21] [hostname=localhost id=shard] MucClient.initializeConnectAndJoin#350: Dispatching a thread to connect and login.
JVB 2024-09-02 09:08:26.069 INFORMATION: [21] [hostname=localhost id=shard] MucClient$2.connected#321: Connected.
JVB 2024-09-02 09:08:26.071 INFORMATION: [21] [hostname=localhost id=shard] MucClient.lambda$getConnectAndLoginCallable$9#646: Logging in.
JVB 2024-09-02 09:08:26.294 INFORMATION: [1] ColibriWebSocketService.registerServlet#84: Registering servlet with baseUrls = [wss://192.70.135.142:443/colibri-ws/default-id], relayBaseUrls = []
JVB 2024-09-02 09:08:26.452 INFORMATION: [21] [hostname=localhost id=shard] MucClient$2.authenticated#327: Authenticated, b=false
JVB 2024-09-02 09:08:26.527 INFORMATION: [1] org.eclipse.jetty.server.Server.doStart: jetty-11.0.21; built: 2024-05-14T03:19:28.958Z; git: 996cd61addad9cb033e0e3eba6fa3f0fa3dc270d; jvm 11.0.24+8-post-Ubuntu-1ubuntu324.04.1
JVB 2024-09-02 09:08:26.832 INFORMATION: [21] [hostname=localhost id=shard] MucClient$MucWrapper.join#771: Joined MUC: [email protected]
JVB 2024-09-02 09:08:27.338 INFORMATION: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@263bbfeb{/,null,AVAILABLE}
JVB 2024-09-02 09:08:27.465 INFORMATION: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@6fd5717c{HTTP/1.1, (http/1.1)}{0.0.0.0:9090}
JVB 2024-09-02 09:08:27.545 INFORMATION: [1] org.eclipse.jetty.server.Server.doStart: Started Server@289778cd{STARTING}[11.0.21,sto=0] @19673ms
JVB 2024-09-02 09:08:27.546 INFORMATION: [1] MainKt.main#142: Starting private http server
JVB 2024-09-02 09:08:28.048 INFORMATION: [1] org.eclipse.jetty.server.Server.doStart: jetty-11.0.21; built: 2024-05-14T03:19:28.958Z; git: 996cd61addad9cb033e0e3eba6fa3f0fa3dc270d; jvm 11.0.24+8-post-Ubuntu-1ubuntu324.04.1
JVB 2024-09-02 09:08:29.510 WARNUNG: [1] org.glassfish.jersey.server.wadl.WadlFeature.configure: JAXBContext implementation could not be found. WADL feature is disabled.
JVB 2024-09-02 09:08:29.876 WARNUNG: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Health registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.Health will be ignored.
JVB 2024-09-02 09:08:29.877 WARNUNG: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Version registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.Version will be ignored.
JVB 2024-09-02 09:08:29.878 WARNUNG: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.prometheus.Prometheus registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.prometheus.Prometheus will be ignored.
JVB 2024-09-02 09:08:30.725 INFORMATION: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@5a566922{/,null,AVAILABLE}
JVB 2024-09-02 09:08:30.736 INFORMATION: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@54f4a7f0{HTTP/1.1, (http/1.1)}{127.0.0.1:8080}
JVB 2024-09-02 09:08:30.737 INFORMATION: [1] org.eclipse.jetty.server.Server.doStart: Started Server@68e7c8c3{STARTING}[11.0.21,sto=0] @22875ms

[damencho]

What it is the available memory on the machine?
We recommend 8GB of ram on the machine for default installation.

[njamblauch]

Right now the Machine has 4 GB Ram.
~ 2GB are in use.

But of course, i can give 8 GB.

Is Jitsi really Ram-Intensive?
Didnt see much in use.

[damencho]

The default settings are that jicofo and jvb can use up to 3Gb of ram each.
You can limit them
https://github.com/jitsi/jicofo/blob/b9d061281aac651be5cac07238678b0d769cbf13/resources/jicofo.sh#L22
https://github.com/jitsi/jitsi-videobridge/blob/6e441e62bd4a47bce8a75ded2dc8d886459eaaf9/jvb/resources/jvb.sh#L19

You can add that variable in the config file of both.

[njamblauch]

I leave it on default settings.

The machine has now 8 GB-Ram.