Threadlock?

[BondByteBlaster]

Hi,

I’ve been asked to resolve a communication issue in a .NET Framework 4.8 client/service project. Several installations are experiencing connection refusals and timeouts on the NetMQ port of the server. However, there are no indications of CPU or memory issues on the servers. The system can run fine for several days before the problem starts, requiring a restart of both services and clients.

I have captured memory dumps of the service, and during connection failures, I can see that 5–500 threads are waiting on a specific thread in NetMQ:

[External Code]
Mailbox.TryRecv
SocketBase.ProcessCommands
SocketBase.TrySend
NetMQSocket.TrySend
NetMQQueue<>.Enqueue
SessionService.SendStatusMessages
SessionService.ServiceLoop
ServiceHandler.StartServiceLoop.Anonymous

public void EnqueueMessage(PublisherBaseMessage message)
{
        _pubQueue.Enqueue(message);
}
private void SendStatusMessages()
{
        KeyValuePair<Guid, StoredSessionInfo>[] snapshot = null;
        lock (_lock)
        {
            snapshot = _sessionInfo.ToArray();
        }

        foreach (var item in snapshot)
            _publisher.EnqueueMessage(new PublisherPrivateMessage(item.Key, true, Guid.Empty, CallbackType.Unevaluated, _serverId, true));
}

public override void ServiceLoop()
{
        base.ServiceLoop();
        if (_nextCheck <= DateTime.Now)
        {
            _nextCheck = DateTime.Now.Add(_options.CheckInterval);
            RemoveInactiveSessions();
            if (DateTime.Now - _options.SessionStatusInterval > _lastStatusSent)
            {
                SendStatusMessages();
            }
        }
}

private void StartServiceLoop(TimeSpan serviceLoopInterval)
{

    var serviceLoopThread = new Thread(new ThreadStart(() =>
    {
        List<BaseService> _baseServices = new List<BaseService>();
        foreach (var v in _injectedServices.Values.OfType<BaseService>())
            _baseServices.Add(v);
        while (true)
        {
            try
            {
                if (_stopRequested)
                    break;
                var nextRun = DateTime.Now + serviceLoopInterval;
                foreach (var v in _baseServices)
                    v.ServiceLoop();
                var remainingTime = nextRun - DateTime.Now;
                if (remainingTime.Ticks > 0)
                    Thread.Sleep(remainingTime);
            }
            catch (Exception ex)
            {
                //Log some fancy service error here
                MQLogHandler.LogExceptionDefault("ServiceHandler: StartServiceLoop",ex);
            }
        }
        _stopped.SetResult(true);
    }));
    serviceLoopThread.Start();
}

I understand that NetMQ requires thread-safe handling, and it(netMQ library) appears to be managed by a single thread only, even while tha application of course is multithreded. The memory usage of NetMQ.Core.Utils.YQueue+Chunk<NetMQ.Msg> is between 50–200 MB.

At this point, I suspect the communication failure is due to a lock preventing the service from responding properly. However, I’m unsure why the thread is locked—perhaps the "thread-safe" implementation isn’t as safe as it seems, or there could be a limit on the NetMQ queue that blocks new messages until previous ones are read or dropped.

I’m new to NetMQ and trying to understand how it works. Do you have any suggestions on where to start looking to save me some time?

The architecture of the solution is that each service (1–4) has its own NetMQ queue. All services, including other clients, listen to all available queues to track changes in the data.

I understand that the information provided here may be too limited to draw conclusions. If you need additional details, please let me know, and I’ll do my best to provide them.

Regards