Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation is flaky #11299

Closed
gaobinlong opened this issue Nov 22, 2023 · 4 comments
Closed

[BUG]org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation is flaky #11299

gaobinlong opened this issue Nov 22, 2023 · 4 comments
Assignees
@Poojita-Raj
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Storage:Durability Issues and PRs related to the durability framework

Comments

@gaobinlong
Copy link
Collaborator

Describe the bug

See the test failure: https://build.ci.opensearch.org/job/gradle-check/28158/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testSingleIndexShardAllocation/

To Reproduce
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation" -Dtests.seed=7F70CFAC7DF5CE33 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=tr -Dtests.timezone=Asia/Oral -Druntime.java=17

Expected behavior
Fix the flaky test.
@gaobinlong gaobinlong added bug Something isn't working untriaged labels Nov 22, 2023
@gaobinlong gaobinlong mentioned this issue Nov 22, 2023
Merged
@dhwanilpatel dhwanilpatel added flaky-test Random test failure that succeeds on second run and removed untriaged labels Nov 28, 2023
@mch2 mch2 mentioned this issue Jan 3, 2024
Merged
8 tasks
@dreamer-89
Copy link
Member

dreamer-89 commented Jan 11, 2024 •
edited
Loading

The test creates 50 shard 1 replica for single index on a 5 data node cluster with one dedicated cluster manager node. The assertion to ensure green cluster trips post stopping one data node. The request to obtain cluster health times out even though all shards are in assigned state. I don't understand the reason for request time out here. Tagging folks who may know and can help @shwetathareja @soosinha.

1> [2023-11-15T00:46:57,384][INFO ][o.o.i.r.SegmentReplicationAllocationIT] [testSingleIndexShardAllocation] ensureGreen timed out, cluster state:
  1> cluster uuid: orO7R8vsRYmor5yeWwLSVw [committed: true]
  1> version: 76
  1> state uuid: vN8ySEMoTPyOw54N0g3mtg
  1> from_diff: false
  1> meta data version: 54
  1>    coordination_metadata:
  1>       term: 1
  1>       last_committed_config: VotingConfiguration{hVmxaxMrQnqidcz_30lqBA,4gBVx4fvSQmkrUvpb9YwPA,ItVjyrpyQRK4BPoNPVzKMw,VQ5hwxBzR1u_0saX5eJzKg,jtKLIchnSLKJA_TLr7-VtA}
  1>       last_accepted_config: VotingConfiguration{hVmxaxMrQnqidcz_30lqBA,4gBVx4fvSQmkrUvpb9YwPA,ItVjyrpyQRK4BPoNPVzKMw,VQ5hwxBzR1u_0saX5eJzKg,jtKLIchnSLKJA_TLr7-VtA}
  1>       voting tombstones: []
  1>    [test/1odmZmTERqKM3BcOCvW2Sg]: v[48], mv[1], sv[1], av[1]
  1>       0: p_term [2], isa_ids [deGaVB2PQdGMyU1OaNXcDQ, hjXVyVkBRSqwCdxENtVpHQ]
  1>       1: p_term [1], isa_ids [7VYxsFXpRwGkCqqbjdvMgg, b19Gr5OiQiSVDEQ9uJnWeg]
  1>       2: p_term [1], isa_ids [4ujnJ1jyQfShoXE10tP0UA, Ci5I848fT7Wf-NQbSojTZg]
  1>       3: p_term [1], isa_ids [LFeh8eUFRNKxsou_EZo7XQ, LyfDArqURXuOJ7C65p8G_Q]
  1>       4: p_term [1], isa_ids [bkKknGCuTZq6hgHGw1DA7A, nJ-nZyWDSj6MEHDYKOeEcw]
  1>       5: p_term [2], isa_ids [DxjASXPoRf27u083gp3sPw, Z-iPAWRFTFGZoZK0JGlrCA]
  1>       6: p_term [1], isa_ids [AFHX1UUHQA-xmXMBdpvdMw, 8auYrWZITsaIGDQIPlMpCg]
  1>       7: p_term [1], isa_ids [-KGuOnHHSCWiHpwK1L9EyA, 81nP9Yf1QfiFlSU_zC1iaQ]
  1>       8: p_term [1], isa_ids [tmRh-2aAT-eDSDWO2eZ4SA, F2mur4pEQXe7wJEn9zXD7Q]
  1>       9: p_term [1], isa_ids [wOFcXY3_TNuye_bIpDN3pg, e4-NSG9iQ82qle6XQjmzLw]
  1>       10: p_term [2], isa_ids [QyBP1RilSaaY1SSO3uTuzQ, 3S-lferyQmu1x-If-IJLmg]
  1>       11: p_term [1], isa_ids [nvxUjNyuS9OScxj-eZ6EFg, WlebT756TiSScAgeDaaKMw]
  1>       12: p_term [1], isa_ids [FB85wH_6Q8aJve1DVJjJDg, h8q-yn2QSUGArxLqTtCx1A]
  1>       13: p_term [1], isa_ids [ACZtruzESOGVP-psdFpMDw, rPtuLHnAQlGDqEsfmHQq3A]
  1>       14: p_term [1], isa_ids [5lHnCMmuRUm5R_MWZGjSKQ, Adc2olK2TTqs9QcyarsFWw]
  1>       15: p_term [2], isa_ids [qAMXN76CR0OPmJcxqHfKCA, zx-Zkzo-T7yBrPX0oK9t6A]
  1>       16: p_term [1], isa_ids [y3SZY72GTXeYmXRRIGHm3g, yt8eDezxQhqSSWrXxB5Y7Q]
  1>       17: p_term [1], isa_ids [JtapsSksS4iKHyCgzVR4Dg, ZmjHitbCRQaZEmTlsMghVA]
  1>       18: p_term [1], isa_ids [ehLiaBGqRAmWavoVzvbx1A, T8yV7xSiT3G_lcKODs0_gg]
  1>       19: p_term [1], isa_ids [Q5DJ1pvfTRef-tw-ySzQaA, y1aksXU2SpSOUslswcGt5Q]
  1>       20: p_term [2], isa_ids [dr8VbJmUQzyje4CFGDsr1A, EzKxlA9TTPO9mK8No6q__g]
  1>       21: p_term [1], isa_ids [vhDE4A0iTDue0PD1hLgoEQ, b3oTcdBJQc2pvFyBTxPhaw]
  1>       22: p_term [1], isa_ids [MFq0Enm0SdmpENs9AnnoNA, NbojWS97SPyQjRQG1qFndA]
  1>       23: p_term [1], isa_ids [2AV73Y_NTR22R3aSl6BhSQ, uNCk4y5pS6-YE6aNlinjnQ]
  1>       24: p_term [1], isa_ids [Xn-CFyG-RtyBFn8EZiJCCg, FaLFQdrqQU2YaJlpSdjibg]
  1>       25: p_term [2], isa_ids [phTMobimSgCo8UlSi-8zWQ, yMRiyBQOTZmh5j8rLREEog]
  1>       26: p_term [1], isa_ids [dxvqXVg-S1qMMWzgh0Gr0w, Nzk5wSXWS4G_PNfpXqF6CA]
  1>       27: p_term [1], isa_ids [-2rKc6LRR-iP9NyQec1z7w, -T57QsNVQdO2LuP0Wf_XPA]
  1>       28: p_term [1], isa_ids [m-NLbQHGSbal29P-IlMMDA, _MPqHzmMTomzAty3NVQfkg]
  1>       29: p_term [1], isa_ids [bMQikTTpQFCdtyy1T6LURg, IdG6PeHZTEqkFc5D7g4y0Q]
  1>       30: p_term [2], isa_ids [wuC_77srRDWWc9vnLYfwBg, fb_ykPCSQniCIFslTAtL3g]
  1>       31: p_term [1], isa_ids [oGz94lSOTU2NBS5mUrxNBg, zHingh4sRUiM5S1uSR8ITQ]
  1>       32: p_term [1], isa_ids [ZnglMc7-SRyEKz6IKpmLmg, ZbLdfcnPSZWUGTaTTpY22g]
  1>       33: p_term [1], isa_ids [u2nCeo7HSAye0RWPeFPK9A, DG898wP9SCOESTgdYQnS1Q]
  1>       34: p_term [1], isa_ids [JxZEiSDES0mrHbQVM0MEXQ, zTtBO_j2TBCIkJ9SRABrZg]
  1>       35: p_term [2], isa_ids [LLKBA2pUT4yc0x7T_fuKJw, LFXtfalaQv--QPM3WjnOzg]
  1>       36: p_term [1], isa_ids [7f9idJ2hSgedehHsEsDkdA, J37QgnpbRB2af3WxUPtdPA]
  1>       37: p_term [1], isa_ids [CPa6gTmsQlu1n_zWh3eOKQ, UUSSq4AiTjuUXf37MZ56VQ]
  1>       38: p_term [1], isa_ids [1b6ABJHpRvSHxothbhnaow, kqcMgBG6QTKjMakskQ50BA]
  1>       39: p_term [1], isa_ids [Ryc5wl0MRPuHoU_IC2STUA, wJfNJOmiQ-6qFaIdNn37NQ]
  1>       40: p_term [2], isa_ids [8A7ecngXRE-HQUY2vew-DA, lHprH2HfQVaHeKCubaOFoQ]
  1>       41: p_term [1], isa_ids [8n5425nPQ6WyF00LmdLzxg, gIEM8cD4TT2UPuaa0u5WOw]
  1>       42: p_term [1], isa_ids [lnWnS8nCSfa8DJlxLpV0gA, rQljkQ9eTnahn26TiT1Cbg]
  1>       43: p_term [1], isa_ids [RqlSoxI7Qm26FF-h77uSlA, Ai5QEcCzTmeJ2QK3Y4oUAQ]
  1>       44: p_term [1], isa_ids [FYgspJVcTQCRDcjxNR8AVQ, azwdytVWSYa5RczROskYsg]
  1>       45: p_term [2], isa_ids [J-jLCzsbTvOsNneB-mKcKg, il5FyZqnTtyTXWXtT1pdZw]
  1>       46: p_term [1], isa_ids [542okCcdRvCA-XAb-Tl56A, SE6y9IskRaK6RC1qEIVfXw]
  1>       47: p_term [1], isa_ids [9cMvLNZ6Sk-_z60phHlnTg, b9J0KHAVT0KUMDf6ONVOzQ]
  1>       48: p_term [1], isa_ids [jTkiV3VkR4qDLLAex9nqvw, qGmwJAmRSfOC6CtuOxAFyQ]
  1>       49: p_term [1], isa_ids [Ur2fESeFTl6rQWdTUVNg1Q, 0LAzPGBWQqyXJIW5biyWIQ]
  1> metadata customs:
  1>    index-graveyard: IndexGraveyard[[]]
  1> nodes: 
  1>    {node_t3}{hVmxaxMrQnqidcz_30lqBA}{E5FK75_sQtOhMebdY5fMBA}{127.0.0.1}{127.0.0.1:34173}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t2}{4gBVx4fvSQmkrUvpb9YwPA}{CWueQ7yLTcCX0iCgrV35Sw}{127.0.0.1}{127.0.0.1:33193}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t5}{VQ5hwxBzR1u_0saX5eJzKg}{_SsnYimkRC-gvhzDb3iDvQ}{127.0.0.1}{127.0.0.1:38423}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t0}{jtKLIchnSLKJA_TLr7-VtA}{Uo2ZyVO6SZ6ytmYM033rKw}{127.0.0.1}{127.0.0.1:37789}{m}{shard_indexing_pressure_enabled=true}, cluster-manager
  1>    {node_t4}{ItVjyrpyQRK4BPoNPVzKMw}{4tshpCS9TymLF93SAiZR4g}{127.0.0.1}{127.0.0.1:43149}{dimr}{shard_indexing_pressure_enabled=true}
  1> routing_table (version 64):
  1> -- index [[test/1odmZmTERqKM3BcOCvW2Sg]]
  1> ----shard_id [test][0]
  1> --------[test][0], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=deGaVB2PQdGMyU1OaNXcDQ]
  1> --------[test][0], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=hjXVyVkBRSqwCdxENtVpHQ]
  1> ----shard_id [test][1]
  1> --------[test][1], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=7VYxsFXpRwGkCqqbjdvMgg]
  1> --------[test][1], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=b19Gr5OiQiSVDEQ9uJnWeg]
  1> ----shard_id [test][2]
  1> --------[test][2], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=Ci5I848fT7Wf-NQbSojTZg]
  1> --------[test][2], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=4ujnJ1jyQfShoXE10tP0UA]
  1> ----shard_id [test][3]
  1> --------[test][3], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=LFeh8eUFRNKxsou_EZo7XQ]
  1> --------[test][3], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=LyfDArqURXuOJ7C65p8G_Q]
  1> ----shard_id [test][4]
  1> --------[test][4], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=nJ-nZyWDSj6MEHDYKOeEcw]
  1> --------[test][4], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=bkKknGCuTZq6hgHGw1DA7A]
  1> ----shard_id [test][5]
  1> --------[test][5], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=DxjASXPoRf27u083gp3sPw]
  1> --------[test][5], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Z-iPAWRFTFGZoZK0JGlrCA]
  1> ----shard_id [test][6]
  1> --------[test][6], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=8auYrWZITsaIGDQIPlMpCg]
  1> --------[test][6], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=AFHX1UUHQA-xmXMBdpvdMw]
  1> ----shard_id [test][7]
  1> --------[test][7], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-KGuOnHHSCWiHpwK1L9EyA]
  1> --------[test][7], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=81nP9Yf1QfiFlSU_zC1iaQ]
  1> ----shard_id [test][8]
  1> --------[test][8], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=F2mur4pEQXe7wJEn9zXD7Q]
  1> --------[test][8], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=tmRh-2aAT-eDSDWO2eZ4SA]
  1> ----shard_id [test][9]
  1> --------[test][9], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=wOFcXY3_TNuye_bIpDN3pg]
  1> --------[test][9], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=e4-NSG9iQ82qle6XQjmzLw]
  1> ----shard_id [test][10]
  1> --------[test][10], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=3S-lferyQmu1x-If-IJLmg]
  1> --------[test][10], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=QyBP1RilSaaY1SSO3uTuzQ]
  1> ----shard_id [test][11]
  1> --------[test][11], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=WlebT756TiSScAgeDaaKMw]
  1> --------[test][11], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=nvxUjNyuS9OScxj-eZ6EFg]
  1> ----shard_id [test][12]
  1> --------[test][12], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=h8q-yn2QSUGArxLqTtCx1A]
  1> --------[test]  1> [12], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=FB85wH_6Q8aJve1DVJjJDg]
  1> ----shard_id [test][13]
  1> --------[test][13], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=ACZtruzESOGVP-psdFpMDw]
  1> --------[test][13], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=rPtuLHnAQlGDqEsfmHQq3A]
  1> ----shard_id [test][14]
  1> --------[test][14], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=Adc2olK2TTqs9QcyarsFWw]
  1> --------[test][14], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=5lHnCMmuRUm5R_MWZGjSKQ]
  1> ----shard_id [test][15]
  1> --------[test][15], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=qAMXN76CR0OPmJcxqHfKCA]
  1> --------[test][15], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zx-Zkzo-T7yBrPX0oK9t6A]
  1> ----shard_id [test][16]
  1> --------[test][16], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=yt8eDezxQhqSSWrXxB5Y7Q]
  1> --------[test][16], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=y3SZY72GTXeYmXRRIGHm3g]
  1> ----shard_id [test][17]
  1> --------[test][17], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=JtapsSksS4iKHyCgzVR4Dg]
  1> --------[test][17], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=ZmjHitbCRQaZEmTlsMghVA]
  1> ----shard_id [test][18]
  1> --------[test][18], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=ehLiaBGqRAmWavoVzvbx1A]
  1> --------[test][18], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=T8yV7xSiT3G_lcKODs0_gg]
  1> ----shard_id [test][19]
  1> --------[test][19], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=y1aksXU2SpSOUslswcGt5Q]
  1> --------[test][19], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Q5DJ1pvfTRef-tw-ySzQaA]
  1> ----shard_id [test][20]
  1> --------[test][20], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=EzKxlA9TTPO9mK8No6q__g]
  1> --------[test][20], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dr8VbJmUQzyje4CFGDsr1A]
  1> ----shard_id [test][21]
  1> --------[test][21], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=b3oTcdBJQc2pvFyBTxPhaw]
  1> --------[test][21], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=vhDE4A0iTDue0PD1hLgoEQ]
  1> ----shard_id [test][22]
  1> --------[test][22], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=MFq0Enm0SdmpENs9AnnoNA]
  1> --------[test][22], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=NbojWS97SPyQjRQG1qFndA]
  1> ----shard_id [test][23]
  1> --------[test][23], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=2AV73Y_NTR22R3aSl6BhSQ]
  1> --------[test][23], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=uNCk4y5pS6-YE6aNlinjnQ]
  1> ----shard_id [test][24]
  1> --------[test][24], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=FaLFQdrqQU2YaJlpSdjibg]
  1> --------[test][24], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Xn-CFyG-RtyBFn8EZiJCCg]
  1> ----shard_id [test][25]
  1> --------[test][25], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=phTMobimSgCo8UlSi-8zWQ]
  1> --------[test][25], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=yMRiyBQOTZmh5j8rLREEog]
  1> ----shard_id [test][26]
  1> --------[test][26], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=Nzk5wSXWS4G_PNfpXqF6CA]
  1> --------[test][26], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dxvqXVg-S1qMMWzgh0Gr0w]
  1> ----shard_id [test][27]
  1> --------[test][27], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-2rKc6LRR-iP9NyQec1z7w]
  1> --------[test][27], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=-T57QsNVQdO2LuP0Wf_XPA]
  1> ----shard_id [test][28]
  1> --------[test][28], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=m-NLbQHGSbal29P-IlMMDA]
  1> --------[test][28], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=_MPqHzmMTomzAty3NVQfkg]
  1> ----shard_id [test][29]
  1> --------[test][29], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=bMQikTTpQFCdtyy1T6LURg]
  1> --------[test][29], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=IdG6PeHZTEqkFc5D7g4y0Q]
  1> ----shard_id [test][30]
  1> --------[test][30], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=fb_ykPCSQniCIFslTAtL3g]
  1> --------[test][30], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=wuC_77srRDWWc9vnLYfwBg]
  1> ----shard_id [test][31]
  1> --------[test][31], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=oGz94lSOTU2NBS5mUrxNBg]
  1> --------[test][31], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=zHingh4sRUiM5S1uSR8ITQ]
  1> ----shard_id [test][32]
  1> --------[test][32], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=ZnglMc7-SRyEKz6IKpmLmg]
  1> --------[test][32], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=ZbLdfcnPSZWUGTaTTpY22g]
  1> ----shard_id [test][33]
  1> --------[test][33], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=u2nCeo7HSAye0RWPeFPK9A]
  1> --------[test][33], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=DG898wP9SCOESTgdYQnS1Q]
  1> ----shard_id [test][34]
  1> --------[test][34], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=JxZEiSDES0mrHbQVM0MEXQ]
  1> --------[test][34], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zTtBO_j2TBCIkJ9SRABrZg]
  1> ----shard_id [test][35]
  1> --------[test][35], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=LLKBA2pUT4yc0x7T_fuKJw]
  1> --------[test][35], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=LFXtfalaQv--QPM3WjnOzg]
  1> ----shard_id [test][36]
  1> --------[test][36], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=J37QgnpbRB2af3WxUPtdPA]
  1> --------[test][36], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=7f9idJ2hSgedehHsEsDkdA]
  1> ----shard_id [test][37]
  1> --------[test][37], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=CPa6gTmsQlu1n_zWh3eOKQ]
  1> --------[test][37], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=UUSSq4AiTjuUXf37MZ56VQ]
  1> ----shard_id [test][38]
  1> --------[test][38], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=kqcMgBG6QTKjMakskQ50BA]
  1> --------[test][38], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=1b6ABJHpRvSHxothbhnaow]
  1> ----shard_id [test][39]
  1> --------[test][39], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=wJfNJOmiQ-6qFaIdNn37NQ]
  1> --------[test][39], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Ryc5wl0MRPuHoU_IC2STUA]
  1> ----shard_id [test][40]
  1> --------[test][40], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=lHprH2HfQVaHeKCubaOFoQ]
  1> --------[test][40], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=8A7ecngXRE-HQUY2vew-DA]
  1> ----shard_id [test][41]
  1> --------[test][41], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=gIEM8cD4TT2UPuaa0u5WOw]
  1> --------[test][41], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=8n5425nPQ6WyF00LmdLzxg]
  1> ----shard_id [test][42]
  1> --------[test][42], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=rQljkQ9eTnahn26TiT1Cbg]
  1> --------[test][42], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=lnWnS8nCSfa8DJlxLpV0gA]
  1> ----shard_id [test][43]
  1> --------[test][43], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=RqlSoxI7Qm26FF-h77uSlA]
  1> --------[test][43], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=Ai5QEcCzTmeJ2QK3Y4oUAQ]
  1> ----shard_id [test][44]
  1> --------[test][44], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=azwdytVWSYa5RczROskYsg]
  1> --------[test][44], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=FYgspJVcTQCRDcjxNR8AVQ]
  1> ----shard_id [test][45]
  1> --------[test][45], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=il5FyZqnTtyTXWXtT1pdZw]
  1> --------[test][45], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=J-jLCzsbTvOsNneB-mKcKg]
  1> ----shard_id [test][46]
  1> --------[test][46], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=SE6y9IskRaK6RC1qEIVfXw]
  1> --------[test][46], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=542okCcdRvCA-XAb-Tl56A]
  1> ----shard_id [test][47]
  1> --------[test][47], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=9cMvLNZ6Sk-_z60phHlnTg]
  1> --------[test][47], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=b9J0KHAVT0KUMDf6ONVOzQ]
  1> ----shard_id [test][48]
  1> --------[test][48], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=jTkiV3VkR4qDLLAex9nqvw]
  1> --------[test][48], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=qGmwJAmRSfOC6CtuOxAFyQ]
  1> ----shard_id [test][49]
  1> --------[test][49], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=0LAzPGBWQqyXJIW5biyWIQ]
  1> --------[test][49], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=Ur2fESeFTl6rQWdTUVNg1Q]

  1> routing_nodes:
  1> -----node_id[hVmxaxMrQnqidcz_30lqBA][V]
  1> --------[test][0], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTE  1> D], a[id=deGaVB2PQdGMyU1OaNXcDQ]
  1> --------[test][1], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=7VYxsFXpRwGkCqqbjdvMgg]
  1> --------[test][6], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=8auYrWZITsaIGDQIPlMpCg]
  1> --------[test][10], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=3S-lferyQmu1x-If-IJLmg]
  1> --------[test][11], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=WlebT756TiSScAgeDaaKMw]
  1> --------[test][16], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=yt8eDezxQhqSSWrXxB5Y7Q]
  1> --------[test][20], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=EzKxlA9TTPO9mK8No6q__g]
  1> --------[test][21], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=b3oTcdBJQc2pvFyBTxPhaw]
  1> --------[test][26], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=Nzk5wSXWS4G_PNfpXqF6CA]
  1> --------[test][30], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=fb_ykPCSQniCIFslTAtL3g]
  1> --------[test][31], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=oGz94lSOTU2NBS5mUrxNBg]
  1> --------[test][36], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=J37QgnpbRB2af3WxUPtdPA]
  1> --------[test][41], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=gIEM8cD4TT2UPuaa0u5WOw]
  1> --------[test][29], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=bMQikTTpQFCdtyy1T6LURg]
  1> --------[test][32], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=ZnglMc7-SRyEKz6IKpmLmg]
  1> --------[test][34], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=JxZEiSDES0mrHbQVM0MEXQ]
  1> --------[test][35], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=LLKBA2pUT4yc0x7T_fuKJw]
  1> --------[test][39], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=wJfNJOmiQ-6qFaIdNn37NQ]
  1> --------[test][40], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=lHprH2HfQVaHeKCubaOFoQ]
  1> --------[test][42], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=rQljkQ9eTnahn26TiT1Cbg]
  1> --------[test][43], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=RqlSoxI7Qm26FF-h77uSlA]
  1> --------[test][44], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=azwdytVWSYa5RczROskYsg]
  1> --------[test][45], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=il5FyZqnTtyTXWXtT1pdZw]
  1> --------[test][46], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=SE6y9IskRaK6RC1qEIVfXw]
  1> --------[test][47], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=9cMvLNZ6Sk-_z60phHlnTg]
  1> -----node_id[4gBVx4fvSQmkrUvpb9YwPA][V]
  1> --------[test][2], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=Ci5I848fT7Wf-NQbSojTZg]
  1> --------[test][7], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-KGuOnHHSCWiHpwK1L9EyA]
  1> --------[test][12], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=h8q-yn2QSUGArxLqTtCx1A]
  1> --------[test][17], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=JtapsSksS4iKHyCgzVR4Dg]
  1> --------[test][22], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=MFq0Enm0SdmpENs9AnnoNA]
  1> --------[test][27], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-2rKc6LRR-iP9NyQec1z7w]
  1> --------[test][32], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=ZbLdfcnPSZWUGTaTTpY22g]
  1> --------[test][37], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=CPa6gTmsQlu1n_zWh3eOKQ]
  1> --------[test][42], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=lnWnS8nCSfa8DJlxLpV0gA]
  1> --------[test][45], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=J-jLCzsbTvOsNneB-mKcKg]
  1> --------[test][47], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=b9J0KHAVT0KUMDf6ONVOzQ]
  1> --------[test][49], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=0LAzPGBWQqyXJIW5biyWIQ]
  1> --------[test][0], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=hjXVyVkBRSqwCdxENtVpHQ]
  1> --------[test][3], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=LFeh8eUFRNKxsou_EZo7XQ]
  1> --------[test][4], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=nJ-nZyWDSj6MEHDYKOeEcw]
  1> --------[test][6], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=AFHX1UUHQA-xmXMBdpvdMw]
  1> --------[test][10], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=QyBP1RilSaaY1SSO3uTuzQ]
  1> --------[test][13], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=ACZtruzESOGVP-psdFpMDw]
  1> --------[test][14], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=Adc2olK2TTqs9QcyarsFWw]
  1> --------[test][16], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=y3SZY72GTXeYmXRRIGHm3g]
  1> --------[test][20], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dr8VbJmUQzyje4CFGDsr1A]
  1> --------[test][23], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=2AV73Y_NTR22R3aSl6BhSQ]
  1> --------[test][24], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=FaLFQdrqQU2YaJlpSdjibg]
  1> --------[test][26], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dxvqXVg-S1qMMWzgh0Gr0w]
  1> --------[test][30], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=wuC_77srRDWWc9vnLYfwBg]
  1> -----node_id[VQ5hwxBzR1u_0saX5eJzKg][V]
  1> --------[test][3], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=LyfDArqURXuOJ7C65p8G_Q]
  1> --------[test][8], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=F2mur4pEQXe7wJEn9zXD7Q]
  1> --------[test][13], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=rPtuLHnAQlGDqEsfmHQq3A]
  1> --------[test][18], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=ehLiaBGqRAmWavoVzvbx1A]
  1> --------[test][23], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=uNCk4y5pS6-YE6aNlinjnQ]
  1> --------[test][28], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=m-NLbQHGSbal29P-IlMMDA]
  1> --------[test][33], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=u2nCeo7HSAye0RWPeFPK9A]
  1> --------[test][38], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=kqcMgBG6QTKjMakskQ50BA]
  1> --------[test][40], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=8A7ecngXRE-HQUY2vew-DA]
  1> --------[test][43], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=Ai5QEcCzTmeJ2QK3Y4oUAQ]
  1> --------[test][46], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=542okCcdRvCA-XAb-Tl56A]
  1> --------[test][48], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=jTkiV3VkR4qDLLAex9nqvw]
  1> --------[test][1], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=b19Gr5OiQiSVDEQ9uJnWeg]
  1> --------[test][2], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=4ujnJ1jyQfShoXE10tP0UA]
  1> --------[test][5], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=DxjASXPoRf27u083gp3sPw]
  1> --------[test][9], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=wOFcXY3_TNuye_bIpDN3pg]
  1> --------[test][11], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=nvxUjNyuS9OScxj-eZ6EFg]
  1> --------[test][12], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=FB85wH_6Q8aJve1DVJjJDg]
  1> --------[test][15], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=qAMXN76CR0OPmJcxqHfKCA]
  1> --------[test][19], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=y1aksXU2SpSOUslswcGt5Q]
  1> --------[test][21], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=vhDE4A0iTDue0PD1hLgoEQ]
  1> --------[test][22], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=NbojWS97SPyQjRQG1qFndA]
  1> --------[test][25], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=phTMobimSgCo8UlSi-8zWQ]
  1> --------[test][31], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=zHingh4sRUiM5S1uSR8ITQ]
  1> --------[test][41], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=8n5425nPQ6WyF00LmdLzxg]
  1> -----node_id[ItVjyrpyQRK4BPoNPVzKMw][V]
  1> --------[test][4], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=bkKknGCuTZq6hgHGw1DA7A]
  1> --------[test][5], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Z-iPAWRFTFGZoZK0JGlrCA]
  1> --------[test][9], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=e4-NSG9iQ82qle6XQjmzLw]
  1> --------[test][14], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=5lHnCMmuRUm5R_MWZGjSKQ]
  1> --------[test][15], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zx-Zkzo-T7yBrPX0oK9t6A]
  1> --------[test][19], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Q5DJ1pvfTRef-tw-ySzQaA]
  1> --------[test][24], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Xn-CFyG-RtyBFn8EZiJCCg]
  1> --------[test][25], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=yMRiyBQOTZmh5j8rLREEog]
  1> --------[test][29], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=IdG6PeHZTEqkFc5D7g4y0Q]
  1> --------[test][34], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zTtBO_j2TBCIkJ9S  1> RABrZg]
  1> --------[test][35], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=LFXtfalaQv--QPM3WjnOzg]
  1> --------[test][39], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Ryc5wl0MRPuHoU_IC2STUA]
  1> --------[test][44], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=FYgspJVcTQCRDcjxNR8AVQ]
  1> --------[test][7], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=81nP9Yf1QfiFlSU_zC1iaQ]
  1> --------[test][8], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=tmRh-2aAT-eDSDWO2eZ4SA]
  1> --------[test][17], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=ZmjHitbCRQaZEmTlsMghVA]
  1> --------[test][18], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=T8yV7xSiT3G_lcKODs0_gg]
  1> --------[test][27], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=-T57QsNVQdO2LuP0Wf_XPA]
  1> --------[test][28], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=_MPqHzmMTomzAty3NVQfkg]
  1> --------[test][33], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=DG898wP9SCOESTgdYQnS1Q]
  1> --------[test][36], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=7f9idJ2hSgedehHsEsDkdA]
  1> --------[test][37], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=UUSSq4AiTjuUXf37MZ56VQ]
  1> --------[test][38], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=1b6ABJHpRvSHxothbhnaow]
  1> --------[test][48], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=qGmwJAmRSfOC6CtuOxAFyQ]
  1> --------[test][49], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=Ur2fESeFTl6rQWdTUVNg1Q]
  1> ---- unassigned

@soosinha
Copy link
Member

@dreamer-89 Looks like the cluster health is not moving to green. Can you print and check the cluster health information while ensureGreen method is running ?

@dreamer-89
Copy link
Member

@dreamer-89 Looks like the cluster health is not moving to green. Can you print and check the cluster health information while ensureGreen method is running ?

Thanks @soosinha for the reply. From logs, I see cluster indeed turned green post node shutdown but did not respond to cluster health call.

 1> [2023-12-12T23:49:25,492][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][44], [test][48]]]).
  1> [2023-12-12T23:49:27,687][INFO ][o.o.i.s.CheckpointRefreshListener] [node_t3] All permits are acquired and refresh listener is closed
  1> [2023-12-12T23:49:27,723][INFO ][o.o.i.s.CheckpointRefreshListener] [node_t4] All permits are acquired and refresh listener is closed
  1> [2023-12-12T23:49:27,960][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [test][43][recover to node_t5] finalizing recovery took [283.4ms]
  1> [2023-12-12T23:49:28,014][INFO ][o.o.i.r.RecoverySourceHandler] [node_t4] [test][45][recover to node_t3] finalizing recovery took [328.3ms]
  1> [2023-12-12T23:49:31,721][INFO ][o.o.i.s.CheckpointRefreshListener] [node_t3] All permits are acquired and refresh listener is closed
...
  1> [2023-12-12T23:49:44,958][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [test][48][recover to node_t4] finalizing recovery took [86.2ms]
  1> [2023-12-12T23:49:45,225][INFO ][o.o.i.r.RecoverySourceHandler] [node_t4] [test][35][recover to node_t5] finalizing recovery took [417.8ms]
  1> [2023-12-12T23:49:50,087][INFO ][o.o.i.r.RecoverySourceHandler] [node_t5] [test][49][recover to node_t4] finalizing recovery took [77.3ms]
  1> [2023-12-12T23:49:50,111][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [test][38][recover to node_t4] finalizing recovery took [96.3ms]
  1> [2023-12-12T23:49:54,504][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [test][37][recover to node_t4] finalizing recovery took [69.2ms]
  1> [2023-12-12T23:49:54,683][INFO ][o.o.i.r.RecoverySourceHandler] [node_t5] [test][46][recover to node_t4] finalizing recovery took [101.9ms]
  1> [2023-12-12T23:49:59,199][INFO ][o.o.i.r.SegmentReplicationAllocationIT] [testSingleIndexShardAllocation] ensureGreen timed out, cluster state:
  1> cluster uuid: QbCDs-UAQaaz2qkoD9LInA [committed: true]

@dreamer-89 dreamer-89 mentioned this issue Jan 12, 2024
Merged
8 tasks
@andrross andrross added the Indexing:Replication Issues and PRs related to core replication framework eg segrep label Feb 21, 2024
@ashking94 ashking94 mentioned this issue Mar 20, 2024
Merged
6 tasks
@soosinha soosinha added the Storage:Durability Issues and PRs related to the durability framework label Apr 16, 2024
@soosinha soosinha removed their assignment Apr 16, 2024
@soosinha soosinha removed the Indexing:Replication Issues and PRs related to core replication framework eg segrep label Apr 16, 2024
@reta
Copy link
Collaborator

reta commented Jun 19, 2024

Closing in favour of #14327

@reta reta closed this as completed Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Poojita-Raj Poojita-Raj

Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Storage:Durability Issues and PRs related to the durability framework
Projects
Storage Project Board
Status: ✅ Done
Milestone
No milestone
Development

No branches or pull requests
7 participants
@reta @dhwanilpatel @andrross @gaobinlong @Poojita-Raj @dreamer-89 @soosinha