Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove jobs from sig-node-presubmits.yaml and generate them for dra-canary.yaml using jinja #34163

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions config/jobs/kubernetes/sig-node/dra-canary.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -254,3 +254,97 @@ presubmits:
requests:
cpu: 2
memory: 9Gi

- name: pull-kubernetes-node-e2e-crio-cgrpv1-dra-kubetest2-canary
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't have to create "kubetest2-canary". Just modify the existing "canary" job.

That conflicts with other, concurrent changes to canary, like my timeout changes. In practice, most of the time we are only going to try one experimental change at a time, so that should be okay.

Let me wrap up my experimental work by merging it into the main jobs...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #34166 is merged, you can make whatever change you want for the canary jobs inside if sections, similar to how I did it in #34150.

cluster: k8s-infra-prow-build
skip_branches:
- release-\d+\.\d+ # per-release image
always_run: false
optional: true
labels:
preset-service-account: "true"
annotations:
testgrid-dashboards: sig-node-dynamic-resource-allocation, sig-node-presubmits, sig-node-cri-o
description: Runs E2E node tests for Dynamic Resource Allocation beta features with CRI-O using cgroup v1
testgrid-alert-email: [email protected], [email protected]
decorate: true
decoration_config:
timeout: 25m # artificially low to test timeout handling, needs to be reverted
path_alias: k8s.io/kubernetes
spec:
containers:
- image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20241230-3006692a6f-master
command:
- runner.sh
args:
- kubetest2
- noop
- --test=node
- --
- --repo-root=.
- --gcp-zone=us-west1-b
- --parallelism=1
- '--label-filter=Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf { Beta, DynamicResourceAllocation } && !Flaky && !Slow'
- '--test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=api/beta=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv1-serial.yaml
env:
- name: IGNITION_INJECT_GCE_SSH_PUBLIC_KEY_FILE
value: "1"
- name: GOPATH
value: /go
- name: KUBE_SSH_USER
value: core
resources:
limits:
cpu: 2
memory: 9Gi
requests:
cpu: 2
memory: 9Gi

- name: pull-kubernetes-node-e2e-crio-cgrpv2-dra-kubetest2-canary
cluster: k8s-infra-prow-build
skip_branches:
- release-\d+\.\d+ # per-release image
always_run: false
optional: true
labels:
preset-service-account: "true"
annotations:
testgrid-dashboards: sig-node-dynamic-resource-allocation, sig-node-presubmits, sig-node-cri-o
description: Runs E2E node tests for Dynamic Resource Allocation beta features with CRI-O using cgroup v2
testgrid-alert-email: [email protected], [email protected]
decorate: true
decoration_config:
timeout: 25m # artificially low to test timeout handling, needs to be reverted
path_alias: k8s.io/kubernetes
spec:
containers:
- image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20241230-3006692a6f-master
command:
- runner.sh
args:
- kubetest2
- noop
- --test=node
- --
- --repo-root=.
- --gcp-zone=us-west1-b
- --parallelism=1
- '--label-filter=Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf { Beta, DynamicResourceAllocation } && !Flaky && !Slow'
- '--test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=api/beta=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv2-serial.yaml
env:
- name: IGNITION_INJECT_GCE_SSH_PUBLIC_KEY_FILE
value: "1"
- name: GOPATH
value: /go
- name: KUBE_SSH_USER
value: core
resources:
limits:
cpu: 2
memory: 9Gi
requests:
cpu: 2
memory: 9Gi
22 changes: 22 additions & 0 deletions config/jobs/kubernetes/sig-node/dra.generate.conf
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,25 @@ inject_ssh_public_key = true
job_type = node
description = Runs E2E node tests for Dynamic Resource Allocation beta features with containerd
image_config_file = /home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/dra/image-config-containerd-1.7.yaml

# This job uses kubetest2 to run e2e_node.test with a focus on tests for the Dynamic Resource Allocation feature (currently beta)
[node-e2e-crio-cgrpv1-dra-kubetest2]
job_type = kubetest2
description = Runs E2E node tests for Dynamic Resource Allocation beta features with CRI-O using cgroup v1
image_config_file = /home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv1-serial.yaml
inject_ssh_public_key = true
# Automatically testing with one container runtime in one configuration is sufficient to detect basic problems in kubelet early.
# CRI-O was picked because it was solid for testing so far.
run_if_changed = (/dra/|/dynamicresources/|/resourceclaim/|/deviceclass/|/resourceslice/|/resourceclaimtemplate/|/dynamic-resource-allocation/|/pkg/apis/resource/|/api/resource/|/test/e2e_node/dra_).*\.(go|yaml)
only_canary = true

# This job uses kubetest2 to run e2e_node.test with a focus on tests for the Dynamic Resource Allocation feature (currently beta)
[node-e2e-crio-cgrpv2-dra-kubetest2]
job_type = kubetest2
description = Runs E2E node tests for Dynamic Resource Allocation beta features with CRI-O using cgroup v2
image_config_file = /home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv2-serial.yaml
inject_ssh_public_key = true
# Automatically testing with one container runtime in one configuration is sufficient to detect basic problems in kubelet early.
# CRI-O was picked because it was solid for testing so far.
run_if_changed = (/dra/|/dynamicresources/|/resourceclaim/|/deviceclass/|/resourceslice/|/resourceclaimtemplate/|/dynamic-resource-allocation/|/pkg/apis/resource/|/api/resource/|/test/e2e_node/dra_).*\.(go|yaml)
only_canary = true
25 changes: 25 additions & 0 deletions config/jobs/kubernetes/sig-node/dra.jinja
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{% if (file == "canary" and only_canary == "true") or not(only_canary == "true") %}
{%- if header %}{{header}}
{%- if file == "ci" %}periodics:
{%- else %}presubmits:
Expand Down Expand Up @@ -99,6 +100,29 @@
- name: GOPATH
value: /go
{%- endif %}
{%- elif job_type == "kubetest2" %}
args:
- kubetest2
- noop
- --test=node
- --
- --repo-root=.
- --gcp-zone=us-west1-b
- --parallelism=1
- '--label-filter={{label_filter}}'
- '--test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=api/beta=true --container-runtime-endpoint=unix:///var/run/{{runtime}}/{{runtime}}.sock --container-runtime-process-name=/usr/local/bin/{{runtime}} --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/{{runtime}}.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"{{runtime}}.log\", \"journalctl\": [\"-u\", \"{{runtime}}\"]}"'
- --image-config-file={{image_config_file}}
{%- if inject_ssh_public_key == "true" %}
env:
- name: IGNITION_INJECT_GCE_SSH_PUBLIC_KEY_FILE
value: "1"
- name: GOPATH
value: /go
{%- if job_type == "kubetest2" %}
- name: KUBE_SSH_USER
value: core
{%- endif %}
{%- endif %}
{%- else %}
args:
- /bin/bash
Expand Down Expand Up @@ -149,4 +173,5 @@
requests:
cpu: 2
memory: 9Gi
{% endif -%}

111 changes: 0 additions & 111 deletions config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3752,117 +3752,6 @@ presubmits:
cpu: 4
memory: 6Gi

- name: pull-kubernetes-node-e2e-crio-cgrpv1-dra-kubetest2 # experimental alternative to pull-kubernetes-node-e2e-crio-cgrpv1-dra
cluster: k8s-infra-prow-build
# explicitly needs /test pull-kubernetes-node-e2e-crio-cgrpv1-dra-kubetest2 to run
always_run: false
# Don't run automatically while experimental!
# run_if_changed: (/dra/|/dynamicresources/|/resourceclaim/|/deviceclass/|/resourceslice/|/resourceclaimtemplate/|/dynamic-resource-allocation/|/pkg/apis/resource/|/api/resource/|/test/e2e_node/dra_).*\.(go|yaml)
optional: true
skip_report: false
skip_branches:
- release-\d+\.\d+ # per-release image
decorate: true
path_alias: k8s.io/kubernetes
extra_refs:
- org: kubernetes
repo: test-infra
base_ref: master
path_alias: k8s.io/test-infra
decoration_config:
timeout: 90m
labels:
preset-service-account: "true"
preset-k8s-ssh: "true"
preset-pull-kubernetes-e2e: "true"
preset-pull-kubernetes-e2e-gce: "true"
annotations:
testgrid-dashboards: sig-node-cri-o, sig-node-presubmits
testgrid-tab-name: pr-node-kubelet-crio-cgrpv1-dra-kubetest2
spec:
containers:
- image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20241230-3006692a6f-master
command:
- runner.sh
args:
- kubetest2
- noop
- --test=node
- --
- --repo-root=.
- --gcp-zone=us-west1-b
- --parallelism=1
- '--label-filter=Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf DynamicResourceAllocation && !Flaky && !Slow'
- '--test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=api/alpha=true,api/beta=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv1-serial.yaml
resources:
limits:
cpu: 4
memory: 6Gi
requests:
cpu: 4
memory: 6Gi
env:
- name: KUBE_SSH_USER
value: core
- name: IGNITION_INJECT_GCE_SSH_PUBLIC_KEY_FILE
value: "1"

- name: pull-kubernetes-node-e2e-crio-cgrpv2-dra-kubetest2 # experimental alternative to pull-kubernetes-node-e2e-crio-cgrpv2-dra
cluster: k8s-infra-prow-build
# explicitly needs /test pull-kubernetes-node-e2e-crio-cgrpv2-dra-kubetest2 to run
always_run: false
# Don't run automatically while experimental!
# run_if_changed: (/dra/|/dynamicresources/|/resourceclaim/|/deviceclass/|/resourceslice/|/resourceclaimtemplate/|/dynamic-resource-allocation/|/pkg/apis/resource/|/api/resource/|/test/e2e_node/dra_).*\.(go|yaml)
optional: true
skip_report: false
skip_branches:
- release-\d+\.\d+ # per-release image
decorate: true
path_alias: k8s.io/kubernetes
extra_refs:
- org: kubernetes
repo: test-infra
base_ref: master
path_alias: k8s.io/test-infra
decoration_config:
timeout: 90m
labels:
preset-service-account: "true"
preset-k8s-ssh: "true"
preset-pull-kubernetes-e2e: "true"
preset-pull-kubernetes-e2e-gce: "true"
annotations:
testgrid-dashboards: sig-node-cri-o, sig-node-presubmits
testgrid-tab-name: pr-node-kubelet-crio-cgrpv2-dra-kubetest2
spec:
containers:
- image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20241230-3006692a6f-master
command:
- runner.sh
args:
- kubetest2
- noop
- --test=node
- --
- --repo-root=.
- --gcp-zone=us-west1-b
- --parallelism=1
- '--test-args=--ginkgo.timeout=1h --ginkgo.label-filter="Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf DynamicResourceAllocation && !Flaky && !Slow" --feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=api/alpha=true,api/beta=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv2-serial.yaml
resources:
limits:
cpu: 4
memory: 6Gi
requests:
cpu: 4
memory: 6Gi
env:
- name: KUBE_SSH_USER
value: core
- name: IGNITION_INJECT_GCE_SSH_PUBLIC_KEY_FILE
value: "1"

- name: pull-kubernetes-node-e2e-resource-health-status
cluster: k8s-infra-prow-build
skip_branches:
Expand Down