Add support for arm cca #211

MatiasVara · 2024-07-29T15:46:06Z

This PR aims at adding support to build realm guests. First commit adds support for create_guest_memfd() and set_user_memory_region2(). To do this, the memory_init() is modified by adding a boolean parameter. This is required when building a confidential guest for arm cca and probably also required by other cases.
The second commit imports the virtee/cca crate and adds the steps to build a cca guest. The following items should be completed before merge the PR:

use populate() only for kernel and initialize() for non-kernel area
handle the case in which the guest switch from shared -> private
correctly calculation of max_ipa

This has been testing using the v7 series for Linux as a guest and v6 series for KVM on FVP model.

Feedback is welcome.

jakecorrenti · 2025-01-09T21:33:07Z

@MatiasVara is this ready for review, or should I wait?

tylerfanelli · 2025-01-09T21:40:26Z

@jakecorrenti I'm addressing the KVM guest_memfd changes (which @MatiasVara also added in this PR) to my latest SEV-SNP patches. This will probably require a rebase after that. Once that happens, it will likely be ready for a review.

MatiasVara · 2025-01-10T09:34:36Z

@jakecorrenti I'm addressing the KVM guest_memfd changes (which @MatiasVara also added in this PR) to my latest SEV-SNP patches. This will probably require a rebase after that. Once that happens, it will likely be ready for a review.

I think the PR is ready to review. The current issue is that it does not work for the latest series for KVM (v6). I am investigating the issue.

MatiasVara · 2025-02-12T10:59:30Z

I just updated the PR with the fix for v6 series. The fix is only to set the KVM_ARM_VCPU_REC feature during the initialization of VCPUs otherwise vcpu_finalize would fail.

MatiasVara · 2025-02-12T11:39:58Z

@jakecorrenti @tylerfanelli @slp Feel free to give a quick look. I am pretty sure there many things to fix.

jakecorrenti

I didn't actually run it, just looked at the code. Overall looks good. Few comments, though.

Thanks for the work!

src/arch/Cargo.toml

jakecorrenti · 2025-02-19T17:43:46Z

src/arch/Cargo.toml

+kvm-bindings = { version = ">=0.8", features = ["fam-wrappers"] , git = "https://github.com/virtee/kvm-bindings", branch = "add_bindings_for_realms" }
+kvm-ioctls = { version = ">=0.17", git = "https://github.com/virtee/kvm-ioctls", branch = "cca" }


Can you make a PR once these branches are ready? I'd like to have the deps be on the main branch

Do you mean to use the main branch rather than the cca?

Exactly. I'm going to have to make some changes for my TDX work, so I think we'll need to merge your CCA work and my TDX work into the main branch and then use that here.

@tylerfanelli Do you think we can merge the cca branches into main for kvm-ioctl/bindings repos? Or, shall we have one branch per flavor? Bear in mind that those repo are temporal until changes are upstreamed.

jakecorrenti · 2025-02-19T17:44:31Z

src/cpuid/Cargo.toml

+kvm-bindings = { version = ">=0.8", features = ["fam-wrappers"] , git = "https://github.com/virtee/kvm-bindings", branch = "add_bindings_for_realms" }
+kvm-ioctls = { version = ">=0.17", git = "https://github.com/virtee/kvm-ioctls", branch = "cca" }


Same comment here

src/devices/Cargo.toml

jakecorrenti · 2025-02-24T20:18:52Z

src/libkrun/Cargo.toml

 crossbeam-channel = "0.5"
 env_logger = "0.9.0"
 libc = ">=0.2.39"
 log = "0.4.0"
 once_cell = "1.4.1"

+kvm-bindings = { version = ">=0.8", features = ["fam-wrappers"] , git = "https://github.com/virtee/kvm-bindings", branch = "add_bindings_for_realms" }


same comment here

jakecorrenti · 2025-02-24T21:04:55Z

src/vmm/src/builder.rs

+        //vmm.kernel_cmdline.insert_str("tsi_hijack")?;
        #[cfg(feature = "net")]


Should this be CCA only? There's another one a few lines below as well

I can't remember why I commented this line.

src/vmm/src/linux/vstate.rs

jakecorrenti · 2025-02-24T21:15:31Z

src/vmm/src/linux/vstate.rs

+        id: u8,
+        vm_fd: &VmFd,
+        exit_evt: EventFd,
+        sender_io: Sender<MemProperties>,


should be CCA only

The memory fault exception isn't TEE-specific (although it could be used that way), so I don't think I agree that this should be CCA only.

You're right. I did this part of the review before you and I had talked about the order of patches.

I think we can enable the extra this extra thread only when the tee feature is enabled.

I agree. I made the original comment before it became clear that all of the TEE functionality would end up adopting it

jakecorrenti · 2025-02-24T21:16:32Z

src/vmm/src/builder.rs

@@ -1103,17 +1231,19 @@ fn create_vcpus_aarch64(
    guest_mem: &GuestMemoryMmap,
    entry_addr: GuestAddress,
    exit_evt: &EventFd,
+    sender_io: Sender<MemProperties>,


should be CCA only

MatiasVara · 2025-03-17T13:14:35Z

After reviewing the Jake comments I came to the conclusion that this PR should be split otherwise is hard to review:

Add Use create_guest_memfd() and set_user_memory_region2() only when tee is enabled.
Add the handling of KVM_MEMORY_FAULT only for tee, which includes the extra thread for handling shared to private and viceverse.
Tee should be replaced with tee and x86-64 since all tee is for x86-64.
Add CCA support. This shall be added with cca and tee enabled.
Eventually some code will be shared for tee in x86-64 and aarch64 but I can't see it yet.

@jakecorrenti @tylerfanelli @slp WDYT?

jakecorrenti · 2025-03-17T21:33:24Z

After reviewing the Jake comments I came to the conclusion that this PR should be split otherwise is hard to review:

0. Add `Use create_guest_memfd() and set_user_memory_region2()` only when `tee` is enabled.

1. Add the handling of `KVM_MEMORY_FAULT` only for `tee`, which includes the extra thread for handling shared to private and viceverse.

2. `Tee` should be replaced with `tee and x86-64` since all `tee` is for `x86-64`.

I don't know if this is idiomatic by any means, and it depends on whether you can have target arch-specific features, but could we have something like tee-x86 and tee? tee-x86 for the x86 specific functionality, and then tee for the shared bits? Would that be cleaner than just writing #[cfg(all(feature = "tee", target_arch = "x86_64"))]? Again, I don't know if that's "proper" Rust or even possible.

3. Add CCA support. This shall be added with `cca and tee` enabled.

4. Eventually some code will be shared for `tee` in `x86-64` and `aarch64` but I can't see it yet.

@jakecorrenti @tylerfanelli @slp WDYT?

Other than my question above, that all sounds good to me.

tylerfanelli · 2025-03-18T00:22:15Z

After reviewing the Jake comments I came to the conclusion that this PR should be split otherwise is hard to review:

Add Use create_guest_memfd() and set_user_memory_region2() only when tee is enabled.

I don't think set_user_memory_region2 should be tee only, as I believe that on newer kernels, set_user_memory_region is deprecated.

Add the handling of KVM_MEMORY_FAULT only for tee, which includes the extra thread for handling shared to private and viceverse.

I'm addressing that in my SEV-SNP PR currently.

Tee should be replaced with tee and x86-64 since all tee is for x86-64.

Makes sense to me.

Add CCA support. This shall be added with cca and tee enabled.

Do you mean aarch64 and tee?

MatiasVara · 2025-03-18T10:15:19Z

After reviewing the Jake comments I came to the conclusion that this PR should be split otherwise is hard to review:
0. Add `Use create_guest_memfd() and set_user_memory_region2()` only when `tee` is enabled.

1. Add the handling of `KVM_MEMORY_FAULT` only for `tee`, which includes the extra thread for handling shared to private and viceverse.

2. `Tee` should be replaced with `tee and x86-64` since all `tee` is for `x86-64`.
I don't know if this is idiomatic by any means, and it depends on whether you can have target arch-specific features, but could we have something like tee-x86 and tee? tee-x86 for the x86 specific functionality, and then tee for the shared bits? Would that be cleaner than just writing #[cfg(all(feature = "tee", target_arch = "x86_64"))]? Again, I don't know if that's "proper" Rust or even possible.

I do not know either. I though that tee and x86 would have code that tdx and snv share, if that exists.

3. Add CCA support. This shall be added with `cca and tee` enabled.

4. Eventually some code will be shared for `tee` in `x86-64` and `aarch64` but I can't see it yet.
@jakecorrenti @tylerfanelli @slp WDYT?
Other than my question above, that all sounds good to me.

MatiasVara · 2025-03-18T10:31:23Z

After reviewing the Jake comments I came to the conclusion that this PR should be split otherwise is hard to review:

Add Use create_guest_memfd() and set_user_memory_region2() only when tee is enabled.

I don't think set_user_memory_region2 should be tee only, as I believe that on newer kernels, set_user_memory_region is deprecated.

That's OK. I could just leave set_user_memory_region2 for the future.

Add the handling of KVM_MEMORY_FAULT only for tee, which includes the extra thread for handling shared to private and viceverse.

I'm addressing that in my SEV-SNP PR currently.

The first two items in my list are for all tee but they were in my PR for almost a year already. In any case, I need to split those chunks from my PR.

Tee should be replaced with tee and x86-64 since all tee is for x86-64.

Makes sense to me.

Add CCA support. This shall be added with cca and tee enabled.

Do you mean aarch64 and tee?

Yes for the moment.

When tee feature is enabled, use create_guest_memfd() and set_user_memory_region2(). The created regions are marked as private by using set_memory_attributes() thus imitating QEMU behavior. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

These descriptors are required for handling the KVM_FAULT_MEMORY vmexit. Although this vmexit is only present when the tee feature is enabled, changing for a reference does not seem the right thing to do by using feratures. Use reference and protect it to be able to share these descriptors. These changes prepare the code for handling KVM_EXIT_FAULT_MEMORY Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

The KVM_EXIT_MEMORY_FAULT vmexit is triggered when guest wants to switch a region of memory from private to shared and viceversa. To support this when tee is enabled, add an extra thread named sender_io that gets the parameters from the vcpu thread and triggers the set_memory_properties(). The vcpu fd is owned only by this thread. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

This commit will be removed when changes are in upstream. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

The `tee` feature has been used by having in mind only the x86-64 case. Currently, `tee` means sense only if `x86-64` is also enabled. In the future, the `tee` will enable code that is shared by arm and x86-64. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

Enable to build confidential guests using ARM CCA (Confidential Computing Architecture). This work relies on v7 series for Linux and v5 series for KVM. This has been tested only on the corresponding FVP model simulator. For testing, you require specific kvm-ioctls and kvm-bindings crates. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

This commit will be remove when cca use virtio-blk. For the moment, enable virtio-fs in cca. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

MatiasVara · 2025-03-19T17:02:20Z

After reviewing the Jake comments I came to the conclusion that this PR should be split otherwise is hard to review:

Add Use create_guest_memfd() and set_user_memory_region2() only when tee is enabled.

Add the handling of KVM_MEMORY_FAULT only for tee, which includes the extra thread for handling shared to private and viceverse.

Tee should be replaced with tee and x86-64 since all tee is for x86-64.

Add CCA support. This shall be added with cca and tee enabled.

Eventually some code will be shared for tee in x86-64 and aarch64 but I can't see it yet.

@jakecorrenti @tylerfanelli @slp WDYT?

I just split the PR into a few commits. It is the same code though. The first three commits are independent of cca and should be shared by all tees. Also [Enable tee only in x86-64 shall for all tees since it aims at separating the code for tee and x86 and tee and aach64.

MatiasVara force-pushed the use-guest-memfd branch from 702d17f to 86f75cd Compare July 29, 2024 16:04

MatiasVara force-pushed the use-guest-memfd branch from b47ef94 to 9e6e5bb Compare August 14, 2024 08:09

MatiasVara force-pushed the use-guest-memfd branch from 9e6e5bb to 68dff0c Compare August 22, 2024 15:20

MatiasVara changed the title ~~Enable creating guest memory regions with create_guest_memfd() and set_user_memory_region2()~~ Add support for arm cca Aug 26, 2024

MatiasVara force-pushed the use-guest-memfd branch 2 times, most recently from d5952be to d4362d6 Compare September 4, 2024 12:10

MatiasVara force-pushed the use-guest-memfd branch 3 times, most recently from cae99a3 to 4a0e61b Compare September 10, 2024 11:40

MatiasVara force-pushed the use-guest-memfd branch 3 times, most recently from 5089b4f to e7e2a90 Compare September 13, 2024 17:30

MatiasVara force-pushed the use-guest-memfd branch 3 times, most recently from dae13da to 6cebf95 Compare October 1, 2024 10:44

MatiasVara force-pushed the use-guest-memfd branch from 6cebf95 to 94688b2 Compare October 24, 2024 09:46

MatiasVara force-pushed the use-guest-memfd branch from 94688b2 to 2077cfc Compare November 21, 2024 11:41

MatiasVara force-pushed the use-guest-memfd branch 2 times, most recently from 18e9a7d to edb2f4a Compare December 19, 2024 13:58

MatiasVara marked this pull request as ready for review December 19, 2024 15:05

MatiasVara requested review from mtjhrc, slp and tylerfanelli as code owners December 19, 2024 15:05

MatiasVara force-pushed the use-guest-memfd branch from edb2f4a to 5ba47b6 Compare December 19, 2024 15:59

MatiasVara force-pushed the use-guest-memfd branch 2 times, most recently from ba8a4b1 to 0cf214d Compare February 12, 2025 10:53

MatiasVara force-pushed the use-guest-memfd branch from 0cf214d to 866514c Compare February 12, 2025 11:03

MatiasVara force-pushed the use-guest-memfd branch from 866514c to 3d4903a Compare February 12, 2025 12:46

MatiasVara force-pushed the use-guest-memfd branch 2 times, most recently from 7b1f763 to 8b97dc4 Compare February 20, 2025 12:30

jakecorrenti requested changes Feb 24, 2025

View reviewed changes

MatiasVara added 7 commits March 19, 2025 07:51

Use kvm-ioctl and kvm-bindings for cca

b243670

This commit will be removed when changes are in upstream. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

Enable virtio-fs in cca

c17ae42

This commit will be remove when cca use virtio-blk. For the moment, enable virtio-fs in cca. Signed-off-by: Matias Ezequiel Vara Larsen <[email protected]>

MatiasVara force-pushed the use-guest-memfd branch from 8b97dc4 to c17ae42 Compare March 19, 2025 16:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for arm cca #211

Add support for arm cca #211

MatiasVara commented Jul 29, 2024 •

edited

Loading

jakecorrenti commented Jan 9, 2025

tylerfanelli commented Jan 9, 2025 •

edited

Loading

MatiasVara commented Jan 10, 2025

MatiasVara commented Feb 12, 2025

MatiasVara commented Feb 12, 2025

jakecorrenti left a comment

jakecorrenti Feb 19, 2025

MatiasVara Feb 25, 2025

jakecorrenti Feb 25, 2025

MatiasVara Feb 26, 2025

jakecorrenti Feb 19, 2025

jakecorrenti Feb 24, 2025

jakecorrenti Feb 24, 2025

MatiasVara Feb 25, 2025

jakecorrenti Feb 24, 2025

tylerfanelli Feb 26, 2025

jakecorrenti Feb 26, 2025

MatiasVara Mar 17, 2025

jakecorrenti Mar 17, 2025

jakecorrenti Feb 24, 2025

MatiasVara commented Mar 17, 2025 •

edited

Loading

jakecorrenti commented Mar 17, 2025 •

edited

Loading

tylerfanelli commented Mar 18, 2025

MatiasVara commented Mar 18, 2025

MatiasVara commented Mar 18, 2025 •

edited

Loading

MatiasVara commented Mar 19, 2025 •

edited

Loading

		kvm-bindings = { version = ">=0.8", features = ["fam-wrappers"] , git = "https://github.com/virtee/kvm-bindings", branch = "add_bindings_for_realms" }
		kvm-ioctls = { version = ">=0.17", git = "https://github.com/virtee/kvm-ioctls", branch = "cca" }

		//vmm.kernel_cmdline.insert_str("tsi_hijack")?;
		#[cfg(feature = "net")]

Add support for arm cca #211

Are you sure you want to change the base?

Add support for arm cca #211

Conversation

MatiasVara commented Jul 29, 2024 • edited Loading

jakecorrenti commented Jan 9, 2025

tylerfanelli commented Jan 9, 2025 • edited Loading

MatiasVara commented Jan 10, 2025

MatiasVara commented Feb 12, 2025

MatiasVara commented Feb 12, 2025

jakecorrenti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatiasVara commented Mar 17, 2025 • edited Loading

jakecorrenti commented Mar 17, 2025 • edited Loading

tylerfanelli commented Mar 18, 2025

MatiasVara commented Mar 18, 2025

MatiasVara commented Mar 18, 2025 • edited Loading

MatiasVara commented Mar 19, 2025 • edited Loading

MatiasVara commented Jul 29, 2024 •

edited

Loading

tylerfanelli commented Jan 9, 2025 •

edited

Loading

MatiasVara commented Mar 17, 2025 •

edited

Loading

jakecorrenti commented Mar 17, 2025 •

edited

Loading

MatiasVara commented Mar 18, 2025 •

edited

Loading

MatiasVara commented Mar 19, 2025 •

edited

Loading