DontReview Garnitin/add gke load testing/v1 #2225

gargnitingoogle · 2024-07-26T21:27:18Z

Description

Link to the issue in case of a bug fix.

NA

Testing details

Manual - NA
Unit tests - NA
Integration tests - NA

codecov · 2024-07-26T21:32:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.04%. Comparing base (17414f3) to head (a0b0096).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2225      +/-   ##
==========================================
+ Coverage   77.02%   77.04%   +0.01%     
==========================================
  Files         113      113              
  Lines       15910    15918       +8     
==========================================
+ Hits        12255    12264       +9     
+ Misses       3129     3128       -1     
  Partials      526      526

Flag	Coverage Δ
unittests	`77.04% <ø> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gargnitingoogle · 2024-07-26T21:29:20Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+fi
+
+function validateMachineConfig() {
+  echo "Validiting input parameters ..."


gargnitingoogle · 2024-07-26T21:29:52Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  local cluster_name=${1}
+  local zone=${2}
+  local node_pool=${3}
+  if [ $(gcloud container node-pools list --cluster=${cluster_name} --zone=${zone} | grep -ow ${node_pool} | wc -l) -gt 0 ] ; then


simplify this with grep -q

gargnitingoogle · 2024-07-26T21:30:59Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  if ClusterExists ${cluster_name} ; then
+    gcloud container clusters update ${cluster_name} --location=${zone} --workload-pool=${project_id}.svc.id.goog
+  else
+    # gcloud container --project "${project_id}" clusters create ${cluster_name} --zone "${zone}" --cluster-version "${cluster_version}" --workload-pool=${project_id}.svc.id.goog


remove commented code

gargnitingoogle · 2024-07-26T21:32:10Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  echo "Enabling/disabling csi add-on ..."
+  # By default, disable the managed csi driver.
+  if ${useCustomCsiDriver}; then
+    # gcloud -q container clusters update ${cluster_name} \


re-enable this

gargnitingoogle · 2024-07-26T21:32:29Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  gcloud container clusters get-credentials ${cluster_name} --location=${zone}
+  kubectl create namespace ${appnamespace}
+  kubectl create serviceaccount ${ksa} --namespace ${appnamespace}
+  for workload_bucket in ${buckets} ; do


add code to get buckets from somewhere.

gargnitingoogle · 2024-07-26T21:35:00Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+}
+
+# validateMachineConfig ${machine_type} ${num_nodes} ${num_ssd}
+# installDependencies


re-enable all these disabled steps.

gargnitingoogle · 2024-07-26T21:36:51Z

perfmetrics/scripts/testing_on_gke/utils/utils.py

+def get_cpu(pod_name: str, start: str, end: str) -> Tuple[float, float]:
+    # for some reason, the mash filter does not always work, so we fetch all the metrics for all the pods and filter later.
+    result = subprocess.run(["mash", "--namespace=cloud_prod", "--output=csv", 
+                             f"Query(Fetch(Raw('cloud.kubernetes.K8sContainer', 'kubernetes.io/container/cpu/core_usage_time'), {{'project': '927584127901'}})| Window(Rate('10m'))| GroupBy(['pod_name', 'container_name'], Max()), TimeInterval('{start}', '{end}'), '5s')"], 


put the project-number back to the original and change it by code during runtime and then revert it back when done during runtime.

gargnitingoogle · 2024-07-26T21:37:31Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+function updateMachineTypeInPodConfigs() {
+  for file in ${gke_testing_dir}/examples/fio/loading-test/values.yaml ${gke_testing_dir}/examples/dlio/unet3d-loading-test/values.yaml ; do
+    test -f ${file}
+    sed -i -E "s/nodeType: [0-9a-z_-]+$/nodeType: ${machine_type}/g" ${file}


add code to revert this back

gargnitingoogle · 2024-07-26T21:37:42Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  for file in ${gke_testing_dir}/examples/fio/loading-test/values.yaml ${gke_testing_dir}/examples/dlio/unet3d-loading-test/values.yaml ; do
+    test -f ${file}
+    # sed -i -E "s/mountOptions: [0-9a-zA-Z,\:\"-_]+$/mountOptions: \"${gcsfuse_mount_options}\"/g" ${file}
+    sed -i -E "s/mountOptions:[ \t]*\"?[a-zA-Z0-9,:_-]+\"? *$/mountOptions: \"${gcsfuse_mount_options}\"/g" "${file}"


add code to revert this back when done

gargnitingoogle · 2024-07-27T04:27:38Z

perfmetrics/scripts/testing_on_gke/utils/utils.py

+    return utc_timestamp_string
+
+def standard_timestamp(timestamp: int) -> str:
+    return timestamp.split('.')[0].replace('T', ' ') + " UTC"


insert newline after this line

gargnitingoogle · 2024-07-27T04:28:16Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

@@ -0,0 +1,425 @@
+#!/bin/bash
+


Add a header and a help option at the top.

gargnitingoogle commented Jul 26, 2024

View reviewed changes

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 2958a3c to ad97f82 Compare July 29, 2024 10:47

gargnitingoogle commented Jul 29, 2024

View reviewed changes

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 7 times, most recently from ff7ee55 to 478921c Compare August 6, 2024 09:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 9 times, most recently from 1e4353a to b6a0e76 Compare August 16, 2024 14:45

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 9 times, most recently from b62b517 to d008a84 Compare August 22, 2024 11:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 0daf723 to 6dbb552 Compare August 26, 2024 09:05

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 2 times, most recently from a0b0096 to 4726215 Compare November 27, 2024 06:08

gargnitingoogle changed the base branch from master to garnitin/add-gke-load-testing/enhancements/remove-mash-dependency November 27, 2024 19:59

gargnitingoogle marked this pull request as ready for review November 27, 2024 19:59

gargnitingoogle requested review from Tulsishah and a team as code owners November 27, 2024 19:59

gargnitingoogle requested review from ankitaluthra1 and removed request for a team November 27, 2024 19:59

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 84a82e8 to 8b5355d Compare November 28, 2024 06:08

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 4726215 to 0451ae5 Compare November 28, 2024 06:08

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 8b5355d to 23d7228 Compare November 29, 2024 06:08

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 0451ae5 to e3f0a0a Compare November 29, 2024 06:08

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 23d7228 to 53def3b Compare November 30, 2024 06:08

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from e3f0a0a to ace404a Compare November 30, 2024 06:08

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 53def3b to d423275 Compare December 3, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from ace404a to 841048c Compare December 3, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from d423275 to 711aa0f Compare December 4, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 841048c to c1f5947 Compare December 4, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 711aa0f to 789b5f4 Compare December 5, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from c1f5947 to db9edf9 Compare December 5, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 789b5f4 to a3feb0d Compare December 6, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from db9edf9 to 1ea03c4 Compare December 6, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from a3feb0d to 8294243 Compare December 6, 2024 08:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 1ea03c4 to fab819c Compare December 6, 2024 08:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 8294243 to dcfb13b Compare December 6, 2024 10:13

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from fab819c to 137aed1 Compare December 6, 2024 10:13

gargnitingoogle added 2 commits December 9, 2024 06:19

expose force_update_gcsfuse_code

23f75f5

empty commit

8c4d392

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from dcfb13b to 924758e Compare December 9, 2024 06:20

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 137aed1 to 8c4d392 Compare December 9, 2024 06:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DontReview Garnitin/add gke load testing/v1 #2225

DontReview Garnitin/add gke load testing/v1 #2225

gargnitingoogle commented Jul 26, 2024

codecov bot commented Jul 26, 2024 •

edited

Loading

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 27, 2024

gargnitingoogle Jul 27, 2024

DontReview Garnitin/add gke load testing/v1 #2225

Are you sure you want to change the base?

DontReview Garnitin/add gke load testing/v1 #2225

Conversation

gargnitingoogle commented Jul 26, 2024

Description

Link to the issue in case of a bug fix.

Testing details

codecov bot commented Jul 26, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 26, 2024 •

edited

Loading