CFE-1131: AWS Tags DAY2 Update #297

anirudhAgniRedhat · 2024-10-11T11:25:26Z

This PR introduces a custom EBSVolumeTagController that monitors the OpenShift Infrastructure resource for changes in AWS ResourceTags. When tags are updated, the controller automatically fetches all AWS EBS-backed PersistentVolumes (PVs) in the cluster, retrieves their volume IDs, and updates the associated EBS tags in AWS.

Key Changes:

Monitors Infrastructure resource for AWS ResourceTags updates.
Directly fetches all PVs using the AWS EBS CSI driver (ebs.csi.aws.com).
Updates AWS EBS tags by merging new and existing tags using the AWS SDK.

openshift-ci-robot · 2024-10-11T11:25:29Z

@anirudhAgniRedhat: This pull request references CFE-1131 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

This PR introduces a custom EBSVolumeTagController that monitors the OpenShift Infrastructure resource for changes in AWS ResourceTags. When tags are updated, the controller automatically fetches all AWS EBS-backed PersistentVolumes (PVs) in the cluster, retrieves their volume IDs, and updates the associated EBS tags in AWS.

Key Changes:

Monitors Infrastructure resource for AWS ResourceTags updates.
Directly fetches all PVs using the AWS EBS CSI driver (ebs.csi.aws.com).
Updates AWS EBS tags by merging new and existing tags using the AWS SDK.
Graceful operator restart on tag changes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

anirudhAgniRedhat · 2024-10-11T11:25:54Z

/hold
Please don't review now currently WIP

openshift-ci-robot · 2024-10-14T15:35:16Z

@anirudhAgniRedhat: This pull request references CFE-1131 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

This PR introduces a custom EBSVolumeTagController that monitors the OpenShift Infrastructure resource for changes in AWS ResourceTags. When tags are updated, the controller automatically fetches all AWS EBS-backed PersistentVolumes (PVs) in the cluster, retrieves their volume IDs, and updates the associated EBS tags in AWS.

Key Changes:

Monitors Infrastructure resource for AWS ResourceTags updates.
Directly fetches all PVs using the AWS EBS CSI driver (ebs.csi.aws.com).
Updates AWS EBS tags by merging new and existing tags using the AWS SDK.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

anirudhAgniRedhat · 2024-10-15T18:07:36Z

/unhold
open for reviews

TrilokGeer · 2024-10-16T07:35:47Z

cmd/aws-ebs-csi-driver-operator/main.go

+
+	go ebsTagsController.Run(ctx)
+
+	klog.Info("EBS Volume Tag Controller is running")


nit: TBD - cleanup info logs before merge.

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

anirudhAgniRedhat · 2024-10-21T04:16:29Z

/cc @jsafrane
PTAL!!

jsafrane

Please update cluster-storage-operator to add token-minter sidecar.

cmd/aws-ebs-csi-driver-operator/main.go

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

jsafrane · 2024-10-21T14:55:31Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+			if err != nil {
+				klog.Errorf("Error updating tags for volume %s: %v", volumeID, err)


There should be retry with exp. backoff. Esp. when CreateTags calls are throttled by AWS.
That probably implies a queue of PVs.

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

gnufied · 2024-10-22T19:42:36Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	infraInformer := c.commonClient.ConfigInformers.Config().V1().Infrastructures().Informer()
+
+	// Add event handler to process updates only when ResourceTags change
+	infraInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{


We don't have do this. If we are using factory.WithInformers, we will reconcile this within Sync function.

Hey @gnufied I was bit confused with this!! basically I don't want to run reconciliation on every change on InfraStructure resource. I need to run reconciliation only if there is a any change in infra.Status.PlatformStatus.AWS.ResourceTags.

Can you suggest a better way to do this so that we can remove the unnecessarily computes.

How will this work when controller is restarted? You are also doing WithInformers below and hence any change in infra object will still trigger Sync.

So - most bulletproof way of ensuring that, we don't unnecessarily process all PVs is to store the information that we have processed these PVs somewhere in a persistent way.

So, what we have currently is worst of both the worlds. If I were to design this, I will probably make a hash of sorted tags and annotate PV with tag hash. If tag hash annotation in PV and computed tag hash don't change, then I will not update the PV or else I will.

How will this work when controller is restarted? You are also doing WithInformers below and hence any change in infra object will still trigger Sync.

What I am thinking is on every restart I would like to run a sync function and update the volumes if there is a change and further On each change in resource-tags we would again run the sync.

So, what we have currently is worst of both the worlds. If I were to design this, I will probably make a hash of sorted tags and annotate PV with tag hash. If tag hash annotation in PV and computed tag hash don't change, then I will not update the PV or else I will.

This looks easy way to manage this!! I will then add a new field in controller struct which will have the updated hash for the sorted map of tags! now in each reconciliation I will update tags only if the hash is different from the other one? Does this sounds better to you??

But that is not what I said. I said, we should store hash of sorted tags in PV objects as annotation and compare those with current tags we are about to apply. We should only apply tags with AWS if hashes change.

Storing them just in-memory doesn't help us much.

Added Thanks For Suggestion!!

gnufied · 2024-10-23T13:28:01Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	if err != nil {
+		return err
+	}
+	err = c.processInfrastructure(infra, ec2Client)


So, does this controller needs to be an opt-in or a default controller? I don't assume every OCP customer wants this feature and if tagging were to fail after OCP upgrade, their clusters will be degraded and we will have support nightmare.

cc @jsafrane

It is presented as enabled by default + no opt out for all HyperShift clusters in the enhancement.

gnufied · 2024-10-23T13:28:24Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+}
+
+// startFailedQueueWorker runs a worker that processes failed batches independently
+func (c *EBSVolumeTagsController) startFailedQueueWorker(ctx context.Context) {


I am not sure if we need this function at all.

So the reason I brought this change is I would like to retry the the batches which have failed to update tags!! Here I would like to update tags in a serial order(one By one) Discussuion link for the volumes. I cannot use the similar sync function here or else you could suggest a better way for this!!

If we are going to per-pv hash, then we are not going to need this right?

The points I want to make sure here are

I definitely need to batch volumes in order to not to hit throttling condition.

Now on failure we would need to retry!!

Since we are using batch APIs, so AWS SDK's APIs will give us error if any one of the VolumeID in the batch hits the any error(May be validation, Auth, Permission etc), All the VolumeIDs in that batch will not be able to update the tags! In this condition I would like to add a worker queue that will handle the the serial update of PV's tags and will retry to update the tags in exponential back-off time-period.

Since We need to figure out the trade-off between the either in first place we should update all PVs using AWS API's in serial order and retry using similar sync function or should we use Batch Volumes to be called in the sync Functions and later retry should be processed with the queue function serially until the queue is empty!!

I definitely need to batch volumes in order to not to hit throttling condition.

That is fine. You are already batching PVs in fetchPVsAndUpdateTags.

Now on failure we would need to retry!!

We are unnecessarily complicating this. The controller is going to resync every 20 minutes anyways, so it will try to tag all the PVs which aren't tagged. So do we even need to keep separate queue for failed PVs? I am also afraid that, your failed worker queue is going to race with normal controller resync.

What is the point of separate failed worker queue when every 20 minutes, we are going to try and sync tag for all PVs which doesn't have matching tag-hash? If you really want a separate worker queue, you will have to redesign the whole thing, so as at least they are not racy. But I don't really understand point of doing it.

Hey @gnufied I completely agree that this retrying the failure is unnecessarily making the changes complicated.

@jsafrane I had a chat with @TrilokGeer regarding this, IMHO we should drop the idea to add degraded condition on tags update failure. As the cluster should not be degraded for failure to apply Tags. as the sync will anyhow retry to tag the volumes withing resync period, in this way we will not immediately require to retry the failures and can remove this queue worker.
Also anyways we are emitting the warning events from the this controller if the tags update is failed. so user will know that the tags update has been failed due the the certain reason also can think of alerts based on that.

/cc @TrilokGeer
Can you also put your views on this!!

BTW are we planning to backport this PR to older releases?

@gnufied I guess we would need to backport this to 4.17. Slack Thread.

So I noticed that you removed failed worker logic. The thing is - I was talking to @jsafrane offline and he has me convinced that, we do need some kind of additional logic so as we can try retagging of "failed" PVs one-by-one, rather than in a batch. This will ensure that, one bad apple in a batch doesn't prevent tagging of rest of the PVs.

But - we need to be careful when doing this.

We should make sure that, PVs which will be tagged via failed worker, doesn't get processed via regular controller resync (so no race).

I would move the entire failed worker code in a separate file.

Ack;
I believe we can remove the ResyncEvery parameter from the controller builder as this is a overkill for us why do we want to run the sync function in every 20 mins if nothing has changed in the resource Tags.

Alternatively, If you really think resync is important here then, I think that we can add the handle the race condition by using another annotation in PVs for Tagging status and filtering based on tagshash and status in the annotation. But this will also cost us some volume Update calls and further need to handle cases where we are not able to update the status within batches. WDYT?

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

gnufied · 2025-01-27T16:45:13Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+
+// writeCredentialsToTempFile writes credentials data to a temporary file and returns the filename
+func writeCredentialsToTempFile(data []byte) (string, error) {
+	f, err := os.CreateTemp("", "aws-shared-credentials")


Will this work with readOnlyFilesystem field we are planning to add to operator pods? cc @dfajmon @jsafrane

We will have to exclude tmp or mount tmp as emptydir for this to work once readOnlyFileSystem is enabled.

yes it's as you say, this would mean excluding the /tmp

We have updated the implementation required for AWS session authentication. Now, the session is created using the role-arn data and will be re-authenticated if it expires. We will not need to create the temp file for authenticating the session.

pkg/driver/aws-ebs/aws_ebs.go

# Conflicts: # go.mod

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

dobsonj · 2025-02-05T00:48:35Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+		c.mutex.Lock()
+		c.awsSession = sess
+		c.mutex.Unlock()


Same questions about locking and if we can make this a single-threaded sync loop apply here.
#313 (comment)

I have already addressed same in #313 (comment)

PTAL

Hey @dobsonj based on the suggestion in #313 (comment)
I have updated the implementation. Thanks for the suggestion.

Putting some points based on the things I changed it would be easier to review then.

Using a single worker just to update the tags by AWS API calls.

I am passing the updateType and []pvNames which needs to be updated via queue worker thread.

Have added updateType because we wanted to handle the batch and individual request seperately.

Used the Events to let the user know that something has failed on tags update.

removed rate limiter as we would not need it now.

we are still using mutex to lock the queueSet mao to know whether pv's still in worker queue we would not want to push them again if they are still in the queue to be updated.

Not passing completely set of PV resource to optimize the memory resources for the operator. Also updating PV annotation if new update is done on top of it will returns us error.

Added some unit tests

Thanks a lot @anirudhAgniRedhat , this is a very nice improvement. I added a few follow-up comments, but you can mark this one resolved.

dobsonj · 2025-02-06T20:42:29Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	updateTypeBatch      = "batch"
+	updateTypeIndividual = "individual"


Can you create a type for these, to prevent someone from setting updateType = "foobar" where behavior is undefined? Something like the following:

type UpdateType string const ( updateTypeBatch UpdateType = "batch" updateTypeIndividual UpdateType = "individual" ) type pvUpdateQueueItem struct { updateType UpdateType pvNames []string }

Please first check my other comments though, I wonder if we really need updateType.

dobsonj · 2025-02-06T20:59:50Z

pkg/driver/aws-ebs/aws_ebs_tags_queue_worker.go

+	} else {
+		if len(item.pvNames) == 0 {
+			c.queue.Forget(item)
+			return
+		}
+		pv, err := c.getPersistentVolumeByName(item.pvNames[0])


Broader question: do we really need to separate out the batch and individual update types? I could be missing something, but the individual case looks mostly the same as the batch case, just with an array length of 1. Can this function handle both cases with out an if/else for updateType?

If there are special cases for batch updates, you could use len(item.pvNames) > 1 the same as item.updateType == "batch". But it would be better to minimize those special cases and handle an array length of 1 the same as >1 as a general rule.

I was thinking of doing that in the same way but it makes the code a bit messy and a bit hard to understand.
putting updateType clarifies what kind of operation is expected to happen where In batch operation there can be a batch that will have only 1 volume.
Putting some points down:

AWS errors are not very detailed. In a batch operation, they will return the first error message let's say in a batch operation: if the nth volume ID does not exist, it will return an error for that volume only maybe n+xth volume might give some different error message. which we are not sure of.

Better readability of code.

Better Events can be published with error messages in case of batch and individual updates.

There can be cases where the user has again changed the infrastructure tags to previous tags, then we would not need to make more requests for a particular PV. IMHO including them within the same function will make it a bit confusing

I just want to put my reasoning for this field else I am open to refactoring it in the same way as you suggested.
WDYT?

Ok, thanks for explaining, I'm okay with keeping the update types as you have them.

One more idea to simplify this function a bit: would it be possible to refactor the code inside the if / else block into separate methods? So that processVolumes could just have the common code and then call the right method, instead of nesting the code inside the if / else?

switch item.updateType { case updateTypeBatch: c.processBatchVolumes(ctx, item, infra, ec2Client) case updateTypeIndividual: c.processIndividualVolumes(ctx, item, infra, ec2Client) default: panic("invalid update type: %v", item.updateType) }

Thanks; I have added the same.

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

anirudhAgniRedhat · 2025-02-13T04:04:13Z

Hey @dobsonj, do you feel anything else is needed for this change?
Else we can consider to merge this..

/retest

openshift-ci · 2025-02-13T06:17:26Z

@anirudhAgniRedhat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/hypershift-e2e-openstack-csi-cinder	`869a428`	link	true	`/test hypershift-e2e-openstack-csi-cinder`
ci/prow/hypershift-e2e-openstack-csi-manila	`869a428`	link	true	`/test hypershift-e2e-openstack-csi-manila`
ci/prow/aws-efs-operator-e2e-extended	`20a925b`	link	false	`/test aws-efs-operator-e2e-extended`
ci/prow/smb-win2022-operator-e2e	`20a925b`	link	false	`/test smb-win2022-operator-e2e`
ci/prow/e2e-azurestack-csi	`6afec9d`	link	false	`/test e2e-azurestack-csi`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

dobsonj · 2025-02-15T00:00:21Z

Hey @dobsonj, do you feel anything else is needed for this change? Else we can consider to merge this..

I think it looks good, thanks for all your work on this @anirudhAgniRedhat

/lgtm
/approve

/cc @ropatil010
for pre-merge testing + qe-approved label

/hold for openshift/cluster-storage-operator#528 to merge first

openshift-ci · 2025-02-15T00:00:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anirudhAgniRedhat, dobsonj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dobsonj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

anirudhAgniRedhat · 2025-02-25T05:12:13Z

Hey @ropatil010 Have you completed the pre-merge testing? so we can close this.
Thanks

anirudhAgniRedhat · 2025-02-25T05:15:59Z

/retest

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 11, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 11, 2024

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 11, 2024

openshift-ci bot requested review from dobsonj and RomanBednar October 11, 2024 11:28

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 4d92e01 to 14ecc0e Compare October 14, 2024 12:42

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from d68d71a to 44e34e7 Compare October 14, 2024 16:00

anirudhAgniRedhat changed the title ~~[WIP] CFE-1131: AWS Tags DAY2 Update~~ CFE-1131: AWS Tags DAY2 Update Oct 14, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 14, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 15, 2024

TrilokGeer reviewed Oct 16, 2024

View reviewed changes

pkg/driver/aws-ebs/aws_ebs_tags_controller.go Outdated Show resolved Hide resolved

openshift-ci bot requested a review from jsafrane October 21, 2024 04:16

jsafrane reviewed Oct 21, 2024

View reviewed changes

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch 2 times, most recently from 146634c to 858b442 Compare October 22, 2024 18:47

gnufied reviewed Oct 22, 2024

View reviewed changes

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 858b442 to fe7b17e Compare October 23, 2024 08:31

gnufied reviewed Oct 23, 2024

View reviewed changes

gnufied reviewed Oct 24, 2024

View reviewed changes

pkg/driver/aws-ebs/aws_ebs_tags_controller.go Outdated Show resolved Hide resolved

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch 3 times, most recently from a9b8b54 to 42579da Compare October 25, 2024 11:40

gnufied reviewed Oct 26, 2024

View reviewed changes

pkg/driver/aws-ebs/aws_ebs_tags_controller.go Outdated Show resolved Hide resolved

gnufied reviewed Jan 27, 2025

View reviewed changes

gnufied reviewed Jan 28, 2025

View reviewed changes

pkg/driver/aws-ebs/aws_ebs.go Show resolved Hide resolved

anirudhAgniRedhat added 6 commits January 30, 2025 15:39

Added tag controller for tags reconciliation

220c9cd

Added Review nits

6df01b4

# Conflicts: # go.mod

Added Validation for hypershift clusters

4a2434b

Added Reviews Nits

2facf9f

Added Predodic reconcile period for tags controller

bb7dbbf

Added failedset map for storing failed volumes for tag update

801285d

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch 2 times, most recently from 1ec389b to 48122e1 Compare February 3, 2025 06:09

updated aws session auth method

b6cc22d

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 48122e1 to b6cc22d Compare February 5, 2025 12:38

dobsonj reviewed Feb 5, 2025

View reviewed changes

updated the tags update only via single queue worker

dc3328e

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 610c351 to dc3328e Compare February 6, 2025 13:49

dobsonj reviewed Feb 6, 2025

View reviewed changes

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from ee77c98 to 4c93dea Compare February 10, 2025 03:48

added review nits

ceb3d9c

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 4c93dea to ceb3d9c Compare February 10, 2025 03:53

openshift-ci bot requested a review from ropatil010 February 15, 2025 00:00

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 15, 2025

openshift-ci bot assigned dobsonj Feb 15, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 15, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 15, 2025


		go ebsTagsController.Run(ctx)

		klog.Info("EBS Volume Tag Controller is running")

		if err != nil {
		klog.Errorf("Error updating tags for volume %s: %v", volumeID, err)

		updateTypeBatch = "batch"
		updateTypeIndividual = "individual"

CFE-1131: AWS Tags DAY2 Update #297

Are you sure you want to change the base?

CFE-1131: AWS Tags DAY2 Update #297

Conversation

anirudhAgniRedhat commented Oct 11, 2024 • edited Loading

openshift-ci-robot commented Oct 11, 2024 • edited by openshift-ci bot Loading

anirudhAgniRedhat commented Oct 11, 2024

openshift-ci-robot commented Oct 14, 2024 • edited by openshift-ci bot Loading

anirudhAgniRedhat commented Oct 15, 2024

Choose a reason for hiding this comment

anirudhAgniRedhat commented Oct 21, 2024

jsafrane left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

anirudhAgniRedhat Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhAgniRedhat Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhAgniRedhat Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhAgniRedhat commented Feb 13, 2025

openshift-ci bot commented Feb 13, 2025 • edited Loading

dobsonj commented Feb 15, 2025

openshift-ci bot commented Feb 15, 2025

anirudhAgniRedhat commented Feb 25, 2025

anirudhAgniRedhat commented Feb 25, 2025

anirudhAgniRedhat commented Oct 11, 2024 •

edited

Loading

openshift-ci-robot commented Oct 11, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Oct 14, 2024 •

edited by openshift-ci bot

Loading

gnufied Oct 26, 2024 •

edited

Loading

anirudhAgniRedhat Oct 28, 2024 •

edited

Loading

anirudhAgniRedhat Oct 30, 2024 •

edited

Loading

anirudhAgniRedhat Feb 6, 2025 •

edited

Loading

openshift-ci bot commented Feb 13, 2025 •

edited

Loading