[CDAP-21135] Add pagination in apps list call made during pre-upgrade job

[adrikagupta]

During pre-upgrade job, we make a API call to list apps to stop the pipelines before upgrade. In case of huge number of pipelines, it can lead to OOM issue. To fix this we have added pagination during the list apps call.

Testing:

• Did patch revision upgrade for a basic instance
• Did instance upgrade from 6.9.2 to 6.10.1
• Upgraded an instance containing more than 3L pipelines from 6.9.2 to 6.10.1

Pre-upgrade job pod logs (total 60 pipelines and kept max page size as 10 for testing purposes) -

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2025-02-11 17:02:47,850 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:testPipeline18.198c6d04-e891-11ef-8bd2-f252ca9a0325
2025-02-11 17:02:47,862 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline1.ceb81583-e890-11ef-9764-f252ca9a0325
2025-02-11 17:02:57,923 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline10.f881b74c-e890-11ef-a8a2-f252ca9a0325
2025-02-11 17:03:07,973 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline11.fcb6875d-e890-11ef-b7e8-f252ca9a0325
2025-02-11 17:03:18,024 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline12.00f36dbe-e891-11ef-aad8-f252ca9a0325
2025-02-11 17:03:28,070 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline13.0520277f-e891-11ef-83e7-f252ca9a0325
2025-02-11 17:03:38,114 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline14.09508ac0-e891-11ef-8791-f252ca9a0325
2025-02-11 17:03:48,162 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline15.0d8c86c1-e891-11ef-aa3b-f252ca9a0325
2025-02-11 17:03:58,208 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline16.11bf33f2-e891-11ef-b7ec-f252ca9a0325
2025-02-11 17:04:08,255 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline17.15e759d3-e891-11ef-90ef-f252ca9a0325
2025-02-11 17:04:18,302 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline18.198c6d04-e891-11ef-8bd2-f252ca9a0325
2025-02-11 17:04:28,430 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:testPipeline27.3f612ded-e891-11ef-a2dd-f252ca9a0325
2025-02-11 17:04:28,431 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline19.1dc22775-e891-11ef-a816-f252ca9a0325
2025-02-11 17:04:38,461 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline2.d6d3bcb4-e890-11ef-93dc-f252ca9a0325
2025-02-11 17:04:48,508 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline20.21f019b6-e891-11ef-9a9a-f252ca9a0325
2025-02-11 17:04:58,552 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline21.26303467-e891-11ef-945b-f252ca9a0325
2025-02-11 17:05:08,594 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline22.2a5acb48-e891-11ef-8591-f252ca9a0325
2025-02-11 17:05:18,645 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline23.2e9b0d09-e891-11ef-8fcd-f252ca9a0325
2025-02-11 17:05:28,693 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline24.32cb704a-e891-11ef-ad17-f252ca9a0325
2025-02-11 17:05:38,736 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline25.370ac7ab-e891-11ef-985a-f252ca9a0325
2025-02-11 17:05:48,781 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline26.3b33fefc-e891-11ef-8d35-f252ca9a0325
2025-02-11 17:05:58,826 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline27.3f612ded-e891-11ef-a2dd-f252ca9a0325
2025-02-11 17:06:08,994 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:testPipeline36.64a8ca06-e891-11ef-aeca-f252ca9a0325
2025-02-11 17:06:08,994 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline28.439365ee-e891-11ef-84d0-f252ca9a0325
2025-02-11 17:06:19,022 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline29.473bd47f-e891-11ef-9ab3-f252ca9a0325
2025-02-11 17:06:29,068 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline3.db035ca5-e890-11ef-b3ca-f252ca9a0325
2025-02-11 17:06:39,112 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline30.4b6de570-e891-11ef-a2e8-f252ca9a0325
2025-02-11 17:06:49,158 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline31.4fa85ad1-e891-11ef-8920-f252ca9a0325
2025-02-11 17:06:59,203 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline32.53db5622-e891-11ef-878a-f252ca9a0325
2025-02-11 17:07:09,246 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline33.5817a043-e891-11ef-a51b-f252ca9a0325
2025-02-11 17:07:19,331 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline34.5c3eb4b4-e891-11ef-8256-f252ca9a0325
2025-02-11 17:07:29,375 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline35.60777c65-e891-11ef-933b-f252ca9a0325
2025-02-11 17:07:39,418 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline36.64a8ca06-e891-11ef-aeca-f252ca9a0325
2025-02-11 17:07:49,536 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:testPipeline45.8a8504ff-e891-11ef-8a93-f252ca9a0325
2025-02-11 17:07:49,536 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline37.68dc3a87-e891-11ef-96ca-f252ca9a0325
2025-02-11 17:07:59,567 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline38.6d163ab8-e891-11ef-aeb4-f252ca9a0325
2025-02-11 17:08:09,611 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline39.713d9d49-e891-11ef-b05a-f252ca9a0325
2025-02-11 17:08:19,658 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline4.df39b356-e890-11ef-b59e-f252ca9a0325
2025-02-11 17:08:29,703 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline40.757812aa-e891-11ef-9d76-f252ca9a0325
2025-02-11 17:08:39,754 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline41.79ad30db-e891-11ef-b34b-f252ca9a0325
2025-02-11 17:08:49,798 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline42.7dee5cfc-e891-11ef-bb03-f252ca9a0325
2025-02-11 17:08:59,843 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline43.82280f0d-e891-11ef-8b1f-f252ca9a0325
2025-02-11 17:09:09,885 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline44.8656c49e-e891-11ef-abbb-f252ca9a0325
2025-02-11 17:09:19,934 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline45.8a8504ff-e891-11ef-8a93-f252ca9a0325
2025-02-11 17:09:30,047 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:testPipeline54.b053ab68-e891-11ef-9f31-f252ca9a0325
2025-02-11 17:09:30,047 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline46.8ebabf70-e891-11ef-9295-f252ca9a0325
2025-02-11 17:09:40,077 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline47.92ef1a51-e891-11ef-81d7-f252ca9a0325
2025-02-11 17:09:50,123 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline48.97208f02-e891-11ef-8d31-f252ca9a0325
2025-02-11 17:10:00,167 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline49.9b5d2743-e891-11ef-8e55-f252ca9a0325
2025-02-11 17:10:10,214 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline5.e36f6dc7-e890-11ef-8d0b-f252ca9a0325
2025-02-11 17:10:20,258 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline50.9f8aa454-e891-11ef-85d0-f252ca9a0325
2025-02-11 17:10:30,302 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline51.a3b00b15-e891-11ef-adee-f252ca9a0325
2025-02-11 17:10:40,350 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline52.a7f22196-e891-11ef-b53d-f252ca9a0325
2025-02-11 17:10:50,396 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline53.ac2ff257-e891-11ef-a894-f252ca9a0325
2025-02-11 17:11:00,440 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline54.b053ab68-e891-11ef-9f31-f252ca9a0325
2025-02-11 17:11:10,545 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:testPipeline9.f452657b-e890-11ef-9b98-f252ca9a0325
2025-02-11 17:11:10,545 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline55.b49043a9-e891-11ef-aea4-f252ca9a0325
2025-02-11 17:11:20,574 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline56.b8bc3a1a-e891-11ef-9db7-f252ca9a0325
2025-02-11 17:11:30,617 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline57.bce682db-e891-11ef-a779-f252ca9a0325
2025-02-11 17:11:40,658 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline58.c122f40c-e891-11ef-8493-f252ca9a0325
2025-02-11 17:11:50,703 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline59.c55f171d-e891-11ef-ab01-f252ca9a0325
2025-02-11 17:12:00,747 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline6.e79fd108-e890-11ef-b7d0-f252ca9a0325
2025-02-11 17:12:10,789 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline60.c9836c6e-e891-11ef-93cc-f252ca9a0325
2025-02-11 17:12:20,840 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline7.ebde8c29-e890-11ef-9df8-f252ca9a0325
2025-02-11 17:12:30,885 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline8.f019289a-e890-11ef-897a-f252ca9a0325
2025-02-11 17:12:40,928 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:default.testPipeline9.f452657b-e890-11ef-9b98-f252ca9a0325
2025-02-11 17:12:50,989 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:null
2025-02-11 17:12:52,318 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@131] - Start listing pipelines for pagination, isLastPage:false token:null
2025-02-11 17:12:52,319 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:system.dataprep.-SNAPSHOT
2025-02-11 17:13:02,323 - DEBUG [main:i.c.c.m.u.UpgradeJobMain@141] - Trying to stop schedule and workflows for application application:system.pipeline.-SNAPSHOT

[itsankit-google]

minor comment, otherwise LGTM.

[sonarqubecloud]

Quality Gate failed

Failed conditions
12.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud