[testing-on-gke part 4] Fixes and improvements in the test setup #2329

gargnitingoogle · 2024-08-09T19:04:40Z

Description

This removes dependence on user having to pass absolute paths for arguments in the run-script.
Passes gcsfuse-mount-options, machine-type from run-script down to the python run script for better programmability rather than manipulating pod configs using SED.
Uses kubectl namespace properly in the run-script.
Makes output file/dir configurable.
Added overall timeout on pod runs.

This is on top of #2359 (and #2319) .

This is followed up in #2348 .

Link to the issue in case of a bug fix.

NA

Testing details

Manual - NA
Unit tests - NA
Integration tests - NA

codecov · 2024-08-09T19:05:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.76%. Comparing base (3e95145) to head (c93cfac).
Report is 9 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2329      +/-   ##
==========================================
- Coverage   78.80%   78.76%   -0.05%     
==========================================
  Files         106      106              
  Lines       11668    11668              
==========================================
- Hits         9195     9190       -5     
- Misses       2003     2006       +3     
- Partials      470      472       +2

Flag	Coverage Δ
unittests	`78.76% <ø> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

perfmetrics/scripts/testing_on_gke/examples/dlio/parse_logs.py

perfmetrics/scripts/testing_on_gke/examples/dlio/run_tests.py

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

Use the give branch i.e. gcsfuse_branch env var only when cloning code afresh. If user passed gcsfuse_src_dir, don't checkout gcsfuse_branch on it.

... from run-gke-tests.sh to run_tests.py.

This is needed as helm takes comma(,) in the helm argument values as a separator for values. This is a problem for gcsfuse-mount-options as they may contain commas and are passed to helm as a single argument. So it is necessary to escape them while passing the value to helm.

Remove some commented out, unnecessary lines. Move CSI code fetch dependency to csi-custom-driver-build path. Remove some unnecessary function calls.

* Fix some docstrings * Add some missing input checks

1. Remove check of pre-existence of implicit workload_config/output_dir. 2. Minor redundant code removal

Add an overall pod-timeout to kill the run if the pods fail or get stuck somehow.

gargnitingoogle changed the base branch from master to garnitin/add-gke-load-testing/support-special-cases/v1 August 9, 2024 19:05

gargnitingoogle commented Aug 9, 2024

View reviewed changes

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh Show resolved Hide resolved

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh Outdated Show resolved Hide resolved

gargnitingoogle marked this pull request as ready for review August 9, 2024 20:41

gargnitingoogle requested review from Tulsishah and a team as code owners August 9, 2024 20:41

gargnitingoogle requested review from ashmeenkaur and removed request for a team August 9, 2024 20:41

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/support-special-cases/v1 branch from f93aa87 to fdfc07a Compare August 12, 2024 04:21

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch from 7240f6d to b581cbc Compare August 12, 2024 05:14

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/support-special-cases/v1 branch from fdfc07a to 2d7028e Compare August 12, 2024 06:37

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch 5 times, most recently from bd29f19 to b292b8e Compare August 14, 2024 04:31

gargnitingoogle changed the title ~~Improve run-script~~ [testing-on-gke part 5] Improve gke test-run script Aug 14, 2024

This was referenced Aug 14, 2024

[testing-on-gke part 5] Support per-workload gcsfuse-mount-options and other improvements #2348

Merged

[testing-on-gke part 3] Accidentally closed PR.. reopened at PR#2359 #2319

Closed

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch from b292b8e to 867f50f Compare August 14, 2024 10:48

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/support-special-cases/v1 branch 3 times, most recently from 1576c15 to 6fd5de1 Compare August 19, 2024 06:06

gargnitingoogle mentioned this pull request Aug 19, 2024

[testing-on-gke part 3] Support instance-id and more complex configuration combinations (reopen of PR#2319) #2359

Merged

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch from 61f2418 to b201e4e Compare August 19, 2024 07:22

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/support-special-cases/v1 branch from e35a174 to bc506bc Compare August 19, 2024 09:00

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch 2 times, most recently from 2ec75eb to a4552be Compare August 19, 2024 15:14

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/support-special-cases/v1 branch 2 times, most recently from 1dc79a2 to 120e393 Compare August 20, 2024 05:42

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch from baa90a6 to 4c995cc Compare August 29, 2024 04:21

kislaykishore reviewed Aug 30, 2024

View reviewed changes

perfmetrics/scripts/testing_on_gke/examples/dlio/parse_logs.py Outdated Show resolved Hide resolved

perfmetrics/scripts/testing_on_gke/examples/dlio/run_tests.py Show resolved Hide resolved

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh Show resolved Hide resolved

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch 4 times, most recently from ae680fe to e85ce99 Compare September 3, 2024 07:17

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch from e85ce99 to 6725598 Compare September 9, 2024 09:28

gargnitingoogle requested a review from kislaykishore September 10, 2024 07:24

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch 2 times, most recently from 9b50a76 to d29b830 Compare September 16, 2024 02:53

gargnitingoogle added 16 commits September 17, 2024 07:04

dont switch branch on preexisting code

a509b34

Use the give branch i.e. gcsfuse_branch env var only when cloning code afresh. If user passed gcsfuse_src_dir, don't checkout gcsfuse_branch on it.

Pass gcsfuse_mount_options, machine_type down

fbdd6e3

... from run-gke-tests.sh to run_tests.py.

Remove pod yaml update by sed

c55efbe

take workload_config as argument to run-script

1b890cf

Add output_dir argument in run-script

4dc8d92

add instruction to connect to any pod

820ed47

Resolve all paths passed to absolute paths

a783d77

Improvements

d778d69

Remove some commented out, unnecessary lines. Move CSI code fetch dependency to csi-custom-driver-build path. Remove some unnecessary function calls.

fix logs about csi driver enabling/disabling

6362303

use namespace properly in run-script

a7ee67e

Address review comments

487ffd3

* Fix some docstrings * Add some missing input checks

Minor fixes

1ac27ed

1. Remove check of pre-existence of implicit workload_config/output_dir. 2. Minor redundant code removal

Address review comments

84823f3

Add pod timeout

25b4e24

Add an overall pod-timeout to kill the run if the pods fail or get stuck somehow.

increase default-timeout to 1wk

c93cfac

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/add-run-script/v2 branch from d29b830 to c93cfac Compare September 17, 2024 07:10

kislaykishore approved these changes Sep 18, 2024

View reviewed changes

gargnitingoogle merged commit 9209696 into master Sep 18, 2024
12 checks passed

gargnitingoogle mentioned this pull request Sep 27, 2024

[testing-on-gke part 6.5] Move code to access bucket iam role to backend #2496

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[testing-on-gke part 4] Fixes and improvements in the test setup #2329

[testing-on-gke part 4] Fixes and improvements in the test setup #2329

gargnitingoogle commented Aug 9, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading

[testing-on-gke part 4] Fixes and improvements in the test setup #2329

[testing-on-gke part 4] Fixes and improvements in the test setup #2329

Conversation

gargnitingoogle commented Aug 9, 2024 • edited Loading

Description

Link to the issue in case of a bug fix.

Testing details

codecov bot commented Aug 9, 2024 • edited Loading

Codecov Report

gargnitingoogle commented Aug 9, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading