Fix/support non mandatory key file

[JBOClara]

What this PR does:

While storageSecretRef is optional, there is no way to configure medusa without passing a secret. But when we want to use role with IRSA , we don't want to passe a ~/.aws/credentials file to medusa.

Which issue(s) this PR fixes:
Fixes #1152
Fixes thelastpickle/cassandra-medusa#581

Linked to thelastpickle/cassandra-medusa#691

Checklist

• Changes manually tested
• Automated Tests added/updated
• Documentation added/updated
• CHANGELOG.md updated (not required for documentation PRs)
• CLA Signed: DataStax CLA

[codecov]

Codecov Report

Attention: 22 lines in your changes are missing coverage. Please review.

Comparison is base (a6271b4) 57.17% compared to head (289587a) 57.15%.
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1129      +/-   ##
==========================================
- Coverage   57.17%   57.15%   -0.02%     
==========================================
  Files         103      103              
  Lines       10648    10655       +7     
==========================================
+ Hits         6088     6090       +2     
- Misses       4027     4032       +5     
  Partials      533      533              

Files	Coverage Δ
controllers/k8ssandra/medusa_reconciler.go	62.19% <0.00%> (-1.96%)	⬇️
pkg/medusa/reconcile.go	62.22% <34.78%> (+0.15%)	⬆️

... and 3 files with indirect coverage changes

[laurentbardin]

LGTM, but I know very little about Go and Cassandra so... 🤷

[adejanovski]

Hi @JBOClara, thanks for pushing this PR!
I did a quick review and posted a few comments to start the conversation.

[JBOClara]

Hello @adejanovski
Could you give me a hint about this error https://github.com/k8ssandra/k8ssandra-operator/actions/runs/7120365184/job/19387445258?pr=1129#step:6:402 ?

I've no clue why is this failing.

[adejanovski]

I've restarted it, since we have some flakes on the K8ssandraTask tests. Let's see how it goes.

[adejanovski]

it passed ;)

[JBOClara]

Hello @adejanovski

Could you please review my PR and let me know if there's anything to add ? Are there any additional tests or documentation updates required to support this change?

Thank you for your guidance.

[yesterdays-vigilante]

This is blocking us at the moment as well - what's required to get this into the operator? Happy to help if I can.

[adejanovski]

This is blocking us at the moment as well - what's required to get this into the operator? Happy to help if I can.

Hi @yesterdays-vigilante,

if you could test this PR and verify that the feature works as expected, it would be much appreciated.
I think it's going to need a rebase on top of main first to get the upgrade to the latest patch release of Medusa which supports role based auth.
@JBOClara, do you think you could rebase and force push?

[yesterdays-vigilante]

Hi @adejanovski! I've deployed this PR into our staging environment, and it seems to work alright - Medusa is able to put the backups in S3 using role-based authentication, and Cassandra still seems to be running fine.

If needed I can also rebase this branch on my own fork and make a new PR.

[adejanovski]

thanks a lot for the quick turnaround on testing this @yesterdays-vigilante 🙏

Let's give @JBOClara a tiny bit of time to get back to this. That PR has been open for a while and I'm the one who dropped the ball on following up with it.

[adejanovski]

Hi @JBOClara, it looks like you still have a conflict on go.sum.

[adejanovski]

@JBOClara, for the unit tests to pass you'll need to regenerate the manifests I think (using make manifests) and commit the new CRDs before pushing again.

[sonarqubecloud]

Quality Gate failed

Failed conditions

7.4% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

[adejanovski]

CI is green and everything looks good. Thanks @JBOClara !

[bradfordcp]

Thank you so much for working through this issue! I had worked around it in the past by providing an empty file via the referenced secret, but that always felt like a hack. Thanks again!

[c3-clement]

Hello @JBOClara , thank you so much for this change!
We were doing the same hack than @bradfordcp .

Is it also planned to support IRSA for Google storage and Azure storage?

We have to implement similar hacks like we did for S3 in order to support Google Cloud and Azure backups.

FYI @adejanovski

[JBOClara]

Hi @c3-clement,

I'm not able to test it on GCP or Azure.

But, have you already tried? Could you share the errors encountered?

Unfortunately, I don't plan to dedicate time to this topic.

[JamesXu2017]

Thank you so much for working through this issue! I had worked around it in the past by providing an empty file via the referenced secret, but that always felt like a hack. Thanks again!

I tried the fake the secret, and using sa I created with S3 policay and role setup, I still not able to get around on this. I am getting erros inside medusa-stanadalone
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
Any suggestion? @adejanovski

[JBOClara]

There is no need for a fake secret. Just use role-based as credentialsType.

Then, check this is correctly configured :

• Medusa runs with a specific k8s service account.
• This k8s sa is annotated with the IAM Role ARN annotation for IRSA.
• this IAM role has the permissions to write to the backup bucket
• this role has a trusted policy, that allows to assume role with web identity, with a condition based on the namespace and k8s SA name (see AWS Doc).
• your k8s cluster has a correctly configured IAM oidc provider https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
• this oidc provider is used in the trusted policy.

[adejanovski]

That's a great explanation that we'd need to put in the docs @JBOClara !

[c3-clement]

Hi @c3-clement,

I'm not able to test it on GCP or Azure.

But, have you already tried? Could you share the errors encountered?

Unfortunately, I don't plan to dedicate time to this topic.

Hello @JBOClara,

Yes I already tried. I had to patch k8ssandra-operator and medusa - I don't have anymore error logs, but here is what went wrong:

Azure:

medusa was crashing at this line https://github.com/thelastpickle/cassandra-medusa/blob/master/medusa/storage/azure_storage.py#L48 because it's expects a key to be provided, which is not needed with Workload identity. Only the storage account name is needed.

Also, Azure Workload Identity auth requires to set the label azure.workload.identity/use: "true" to pods, and k8ssandra-operator does not propagate labels set in data centers to the medusa standalone deployment.

Google:

In the backup secret mounted to medusa pods, we have to set a field credentials with an arbitrary value, otherwise medusa will crash.

Also, in medusa codebase, the GCS list implementation is accessing unsafely the field 'md5Hash' in the object returned by Google API
https://github.com/thelastpickle/cassandra-medusa/blob/master/medusa/storage/google_storage.py#L81

However, if the GCS bucket is KMS encrypted, the Google API does not expose the 'md5Hash' field (I have talked with Google cloud support about this)
So if the bucket is KMS encrypted, Medusa will crash. We fixed it by using the etag field instead.

I will open proper GitHub issues for all of this when I got a sec.
I will also like to offer fixes for those issues I got time @JBOClara @adejanovski

[c3-clement]

Thank you so much for working through this issue! I had worked around it in the past by providing an empty file via the referenced secret, but that always felt like a hack. Thanks again!

I tried the fake the secret, and using sa I created with S3 policay and role setup, I still not able to get around on this. I am getting erros inside medusa-stanadalone botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity Any suggestion? @adejanovski

I believe that the serviceAccount configured in datacenters is not propagated to the Medusa standalone deployment.
Could it be the cause of your issue @JamesXu2017 ?
FYI @JBOClara

[JBOClara]

Here is a PR #1293 @adejanovski