feat: increase outbound q size for pubsub

[chaitanyaprem]

Description

while going through logs in status-desktop relay mode, i had noticed following log occuring quite consistently.

DEBUG[09-05|06:35:08.182|go.uber.org/zap/sugar.go:198] dropping message to peer 16Uiu2HAmGAA54bBTE78MYidSy3P7Q9yAWFNTAEReJYD69VRvtL5r: queue full

On quick analysis, noticed that this is happening towards almost all peers.

In a span of ~24 hours i could see similar logs towards almost 200 peers which shows that we need to increase queue-size that might be causing this.

grep "dropping message to peer" geth* | awk '{print $6}' | sort | uniq  | wc -l
     202

This may help improve message reliability as messages would not be dropped towards peers be it control msgs at gossipsub layer or application messages.

Changes

• increased default outbound queue size towards peers from 32 to 1024 (in alignment with nwaku feat: add max number of elements to non-prio queue vacp2p/nim-libp2p#1077)

Tests

dogfooding in progress, will update with results

[chaitanyaprem]

still noticing these logs for some peers after applying the fix. In last 2.5 hours, noticed these logs for 5 peers

# grep "dropping message to peer" geth* | awk '{print $6}' | sort | uniq  | wc -l
       5

# grep "dropping message to peer" geth* | awk '{print $6}' | sort  | wc -l       
    4759

# grep "dropping message to peer" geth* | awk '{print $6}' | sort | uniq         
16Uiu2HAm2M7xs7cLPc3jamawkEqbr7cUJX11uvY7LxQ6WFUdUKUT:
16Uiu2HAm9CQhsuwPR54q27kNj9iaQVfyRzTGKrhFmr94oD8ujU6P:
16Uiu2HAm9aDJPkhGxc2SFcEACTFdZ91Q5TJjp76qZEhq9iF59x7R:
16Uiu2HAmAUdrQ3uwzuE4Gy4D56hX6uLKEeerJAnhKEHZ3DxF1EfT:
16Uiu2HAmNTpGnyZ8W1BK2sXEmgSCNWiyDKgRU3NBR2DXST2HzxRU:

Interestingly so far, i have noticed these logs only towards store nodes and 1 bootnode in the fleet in past 2 hours. will keep monitoring for longer duration to get better results and idea of what else could be happening.
Update after few more hours: now i see queue full towards other peers as well. total peers noticed 18, which include upto 9 fleet peers.

[chaitanyaprem]

Update after few days of running....noticed a significant reduction in the occurance of this issue now after increaseing the outgoing queue size.

I could only see this happen towards 14 peers which is less than more than 100 peers that was before. But it is still happening which needs further investigaton which will be tracked via separate issue.

So, i think we are good to go with this PR.
@richard-ramos need your eyes on this one.

[richard-ramos]

LGTM!