[LibOS,Pal/Linux-SGX] Replace old protected files subsystem

Documentation/attestation.rst

@@ -204,13 +204,23 @@ use the :program:`quote_dump`, :program:`ias_request` and
 EPID based quote verification) or to use the Intel DCAP libraries and tools (for
 DCAP based quote verification).
 
-The ``/dev/attestation`` pseudo-filesystem also exposes a pseudo-file to set the
-protected files wrap (master) key (see also :doc:`manifest-syntax`):
+The ``/dev/attestation`` pseudo-filesystem also exposes pseudo-files to set the
+encryption keys (see also :doc:`manifest-syntax`):
 
-- ``/dev/attestation/protected_files_key`` pseudo-file can be opened for write
-  access. Typically, it is opened before the actual application runs and filled
-  with a 128-bit key obtained from a remote secret provisioning service.
+- ``/dev/attestation/keys/<key_name>`` file contains the encryption key with a
+  given name (the default key name is ``default``). Typically, it is opened
+  before the actual application runs and filled with a 128-bit key obtained from
+  a remote secret provisioning service. The format of the file is a 16-byte raw
+  binary value.
 
+.. note::
+   Previously, ``/dev/attestation/protected_files_key`` was used for setting the
+   default encryption key, and Gramine still supports that file for backward
+   compatibility.
+
+   Note that the old file (``/dev/attestation/protected_files_key``) uses a
+   32-character hex value, and the new files
+   (``/dev/attestation/keys/<key_name>``) use a 16-byte raw binary value.
 
 Mid-level RA-TLS interface
 --------------------------
@@ -398,7 +408,7 @@ EPID based ``secret_prov_verify_epid.so`` and DCAP/ECDSA based
 
 The examples of using RA-TLS can be found under ``CI-Examples/ra-tls-secret-prov``.
 The examples include minimalistic provisioning of constant-string secrets as
-well as provisioning of an encryption key and its later use for protected files.
+well as provisioning of an encryption key and its later use for encrypted files.
 
 ``secret_prov_attest.so``
 ^^^^^^^^^^^^^^^^^^^^^^^^^

Documentation/devel/performance.rst

@@ -425,7 +425,7 @@ encrypts many means of communication:
 #. Inter-Process Communication (IPC) is encrypted via TLS-PSK. Regular pipes,
    FIFO pipes, UNIX domain sockets are all transparently encrypted.
 
-#. Files marked as ``sgx.protected_files`` are transparently encrypted/decrypted
+#. Files mounted as ``type = "encrypted"`` are transparently encrypted/decrypted
    on each file access via SGX SDK Merkle-tree format.
 
 #. ``Fork/vfork/clone`` all require to generate an encrypted checkpoint of the

Documentation/manifest-syntax.rst

@@ -134,14 +134,15 @@ If you want your application to use commandline arguments you need to either set
 ``loader.argv_src_file`` to a file containing output of
 :ref:`gramine-argv-serializer<gramine-argv-serializer>`.
 
-``loader.argv_src_file`` is intended to point to either a trusted file or a
-protected file. The former allows to securely hardcode arguments (current
+``loader.argv_src_file`` is intended to point to either a trusted file or an
+encrypted file. The former allows to securely hardcode arguments (current
 manifest syntax doesn't allow to include them inline), the latter allows the
 arguments to be provided at runtime from an external (trusted) source.
 
 .. note ::
-   Pointing to a protected file is currently not supported, due to the fact that
-   PF wrap key provisioning currently happens after setting up arguments.
+   Pointing to an encrypted file is currently not supported, due to the fact
+   that encryption key provisioning currently happens after setting up
+   arguments.
 
 Environment variables
 ^^^^^^^^^^^^^^^^^^^^^
@@ -180,15 +181,15 @@ value or be a passthrough.
 ``loader.env_src_file`` allows to specify a URI to a file containing serialized
 environment, which can be generated using
 :ref:`gramine-argv-serializer<gramine-argv-serializer>`. This option is intended
-to point to either a trusted file or a protected file. The former allows to
+to point to either a trusted file or an encrypted file. The former allows to
 securely hardcode environments (in a more flexible way than
 ``loader.env.[ENVIRON]`` option), the latter allows the environments to be
 provided at runtime from an external (trusted) source.
 
 .. note ::
-   Pointing to a protected file is currently not supported, due to the fact that
-   PF wrap key provisioning currently happens after setting up environment
-   variables.
+   Pointing to an encrypted file is currently not supported, due to the fact
+   that encryption key provisioning currently happens after setting up
+   environment variables.
 
 If the same variable is set in both, then ``loader.env.[ENVIRON]`` takes
 precedence. It is prohibited to specify both ``value`` and ``passthrough`` keys
@@ -254,7 +255,7 @@ Gramine internal metadata size
     (default: "0")
 
 This syntax specifies how much additional memory Gramine reserves for its
-internal use (e.g., metadata for trusted/protected files, internal handles,
+internal use (e.g., metadata for trusted files, internal handles,
 etc.). By default, Gramine pre-allocates 64MB of internal memory for this
 metadata, but for huge workloads this limit may be not enough. In this case,
 Gramine loudly fails with "out of PAL memory" error. To run huge workloads,
@@ -367,7 +368,7 @@ The ``type`` parameter specifies the mount point type. If omitted, it defaults
 to ``"chroot"``. The ``path`` parameter must be an absolute path (i.e. must
 begin with ``/``).
 
-Gramine currently supports two types of mount points:
+Gramine currently supports the following types of mount points:
 
 * ``chroot`` (default): Host-backed files. All host files and sub-directories
   found under ``[URI]`` are forwarded to the Gramine instance and placed under
@@ -378,6 +379,9 @@ Gramine currently supports two types of mount points:
   Docker's named volumes. Files under ``chroot`` mount points support mmap and
   fork/clone.
 
+* ``encrypted``: Host-backed encrypted files. See :ref:`encrypted-files` for
+  more information.
+
 * ``tmpfs``: Temporary in-memory-only files. These files are *not* backed by
   host-level files. The tmpfs files are created under ``[PATH]`` (this path is
   empty on Gramine instance startup) and are destroyed when a Gramine instance
@@ -598,7 +602,7 @@ are not cryptographically hashed and are thus not protected.
 
 .. warning::
    It is insecure to allow files containing code or critical information;
-   developers must not allow files blindly! Instead, use trusted or protected
+   developers must not allow files blindly! Instead, use trusted or encrypted
    files.
 
 Trusted files
@@ -631,67 +635,67 @@ Marking files as trusted is especially useful for shared libraries: a |~|
 trusted library cannot be silently replaced by a malicious host because the hash
 verification will fail.
 
-Protected files
+.. _encrypted-files:
+
+Encrypted files
 ^^^^^^^^^^^^^^^
 
 ::
 
-    sgx.insecure__protected_files_key = "[16-byte hex value]"
-
-    sgx.protected_files = [
-      "[URI]",
-      "[URI]",
-    ]
-
-    sgx.protected_mrenclave_files = [
-      "[URI]",
-      "[URI]",
+    fs.mounts = [
+      { type = "encrypted", path = "[PATH]", uri = "[URI]", key_name = "[KEY_NAME]" },
     ]
 
-    sgx.protected_mrsigner_files = [
-      "[URI]",
-      "[URI]",
-    ]
+    fs.insecure__keys.[KEY_NAME] = "[32-character hex value]"
 
-This syntax specifies the files that are encrypted on disk and transparently
+This syntax allows mounting files that are encrypted on disk and transparently
 decrypted when accessed by Gramine or by application running inside Gramine.
-Protected files guarantee data confidentiality and integrity (tamper
-resistance), as well as file swap protection (a protected file can only be
-accessed when in a specific path).
+Encrypted files guarantee data confidentiality and integrity (tamper
+resistance), as well as file swap protection (an encrypted file can only be
+accessed when in a specific host path).
+
+Encrypted files were previously known as *protected files*, and some Gramine
+tools might still use the old name.
+
+URI can be a file or a directory. If a directory is mounted, all existing
+files/directories within it are recursively treated as encrypted (and are
+expected to be encrypted in the PF format). New files created in an encrypted
+mount are also automatically treated as encrypted.
 
-URI can be a file or a directory. If a directory is specified, all existing
-files/directories within it are registered as protected recursively (and are
-expected to be encrypted in the PF format). New files created in a protected
-directory are automatically treated as protected.
+.. warning::
+   The current implementation assumes that ``type = "encrypted"`` mounts do not
+   overlap on host, i.e. there are no host files reachable through more than one
+   ``type = "encrypted"`` mount. Otherwise, changes made to such files might not
+   be correctly persisted by Gramine.
 
-Note that path size of a protected file is limited to 512 bytes and filename
+Note that path size of an encrypted file is limited to 512 bytes and filename
 size is limited to 260 bytes.
 
-``sgx.insecure__protected_files_key`` specifies the wrap (master) encryption key
-and must be used only for debugging purposes.
+The ``key_name`` mount parameter specifies the name of the encryption key. If
+omitted, it will default to ``"default"``. This feature can be used to mount
+different files or directories with different encryption keys.
 
-.. warning::
-   ``sgx.insecure__protected_files_key`` hard-codes the key in the manifest.
-   This option is thus insecure and must not be used in production environments!
-   Typically, you want to provision the protected files wrap key using SGX
-   local/remote attestation, thus you should not specify the
-   ``sgx.insecure__protected_files_key`` manifest option at all. Instead, use
-   the Secret Provisioning interface (see :doc:`attestation`).
+``fs.insecure__keys.[KEY_NAME]`` can be used to specify the encryption keys
+directly in manifest. This option must be used only for debugging purposes.
 
-There are three types of protected files:
+.. warning::
+   ``sgx.insecure__keys.[KEY_NAME]`` hard-codes the key in the manifest. This
+   option is thus insecure and must not be used in production environments!
+   Typically, you want to provision the encryption keys using SGX
+   local/remote attestation, thus you should not specify any
+   ``sgx.insecure__keys.[KEY_NAME]`` manifest options at all. Instead, use the
+   Secret Provisioning interface (see :doc:`attestation`).
 
-* ``sgx.protected_files`` are encrypted using the wrap (master) encryption key;
-  they are well-suited for input files encrypted by the user and sent to the
-  deployment platform as well as for output files sent back to the user and
-  decrypted at the user side.
+Key names beginning with underscore (``_``) denote special keys provided by
+Gramine:
 
-* ``sgx.protected_mrenclave_files`` are encrypted using the SGX sealing key
-  based on the MRENCLAVE identity of the enclave; they are useful to allow only
-  the same enclave (on the same platform) to unseal files.
+* ``"_sgx_mrenclave"`` (SGX only) is the SGX sealing key based on the MRENCLAVE
+  identity of the enclave. This is useful to allow only the same enclave (on the
+  same platform) to unseal files.
 
-* ``sgx.protected_mrsigner_files`` are encrypted using the SGX sealing key based
-  on the MRSIGNER identity of the enclave; they are useful to allow all enclaves
-  signed with the same key (and on the same platform) to unseal files.
+* ``"_sgx_mrsigner"`` (SGX only) is the SGX sealing key based on the MRSIGNER
+  identity of the enclave. This is useful to allow all enclaves signed with the
+  same key (and on the same platform) to unseal files.
 
 File check policy
 ^^^^^^^^^^^^^^^^^
@@ -876,3 +880,40 @@ Experimental sysfs topology support
     fs.experimental__enable_sysfs_topology = [true|false]
 
 This feature is now enabled by default and the option was removed.
+
+Protected files (deprecated syntax)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    sgx.protected_files = [
+      "[URI]",
+      "[URI]",
+    ]
+
+    sgx.protected_mrenclave_files = [
+      "[URI]",
+      "[URI]",
+    ]
+
+    sgx.protected_mrsigner_files = [
+      "[URI]",
+      "[URI]",
+    ]
+
+This syntax specified the previous SGX-only protected files. It has been
+replaced with ``type = "encrypted"`` mounts (see :ref:`encrypted-files`).
+
+.. warning::
+   Gramine will attempt to convert this syntax to mounted filesystems, but might
+   fail to do so correctly in more complicated cases (e.g. when a single host
+   file belongs to multiple mounts). It is recommended to rewrite all usages of
+   this syntax to ``type = "encrypted"`` mounts.
+
+::
+
+   fs.insecure__protected_files_key = "[32-character hex value]"
+
+This syntax allowed specifying the default encryption key for protected files.
+It has been replaced by ``fs.insecure__keys.[KEY_NAME]]``. Note that both old
+and new syntax are suitable for debugging purposes only.

Documentation/pal/host-abi.rst

@@ -338,5 +338,5 @@ and to obtain an attestation report and quote.
 .. doxygenfunction:: DkAttestationQuote
    :project: pal
 
-.. doxygenfunction:: DkSetProtectedFilesKey
+.. doxygenfunction:: DkGetSpecialKey
    :project: pal

LibOS/shim/include/shim_fs_encrypted.h

@@ -171,4 +171,9 @@ int encrypted_file_rename(struct shim_encrypted_file* enc, const char* new_uri);
 int encrypted_file_get_size(struct shim_encrypted_file* enc, file_off_t* out_size);
 int encrypted_file_set_size(struct shim_encrypted_file* enc, file_off_t size);
 
+int parse_pf_key(const char* key_str, pf_key_t* pf_key);
+
+/* TODO: This function is used only by a feature deprecated in v1.2, remove two versions later. */
+int dump_pf_key(const pf_key_t* pf_key, char* buf, size_t buf_size);
+
 #endif /* SHIM_FS_ENCRYPTED_ */

LibOS/shim/src/fs/chroot/fs.c

@@ -495,74 +495,6 @@ static int chroot_chmod(struct shim_dentry* dent, mode_t perm) {
     return ret;
 }
 
-static int chroot_reopen(struct shim_handle* hdl, PAL_HANDLE* out_palhdl) {
-    PAL_HANDLE palhdl;
-
-    mode_t mode = 0;
-    enum pal_access access = LINUX_OPEN_FLAGS_TO_PAL_ACCESS(hdl->flags);
-    enum pal_create_mode create = PAL_CREATE_NEVER;
-    pal_stream_options_t options = LINUX_OPEN_FLAGS_TO_PAL_OPTIONS(hdl->flags);
-    int ret = DkStreamOpen(hdl->uri, access, mode, create, options, &palhdl);
-    if (ret < 0)
-        return pal_to_unix_errno(ret);
-    *out_palhdl = palhdl;
-    return 0;
-}
-
-/*
- * Prepare the handle to be sent to child process. If the corresponding file still exists on the
- * host, we will not checkpoint its PAL handle, but let the child process open another one.
- *
- * TODO: this is only necessary because PAL handles for protected files cannot be sent to child
- * process (`DkSendHandle`). This workaround limits the damage: inheriting a handle by child process
- * will now fail to work only if it's a handle for a protected file *and* the file has been deleted
- * from host.
- */
-static int chroot_checkout(struct shim_handle* hdl) {
-    assert(hdl->type == TYPE_CHROOT);
-    assert(hdl->pal_handle);
-
-    /* We should be holding `g_dcache_lock` for the whole checkpointing process. */
-    assert(locked(&g_dcache_lock));
-
-    /* We don't take `hdl->lock` because this is actually the handle *copied* for checkpointing (and
-     * the lock isn't even properly initialized). */
-
-    /* First, check if we have not deleted or renamed the file (the dentry contains the same
-     * inode). */
-    bool is_in_dentry = (hdl->dentry->inode == hdl->inode);
-
-    if (is_in_dentry) {
-        /* Then check if the file still exists on host. If so, we assume it can be opened by the
-         * child process, so the PAL handle doesn't need sending. */
-        PAL_STREAM_ATTR attr;
-        if (DkStreamAttributesQuery(hdl->uri, &attr) == 0) {
-            hdl->pal_handle = NULL;
-        }
-    }
-
-    return 0;
-}
-
-static int chroot_checkin(struct shim_handle* hdl) {
-    assert(hdl->type == TYPE_CHROOT);
-
-    /* We don't take `hdl->lock` because this handle is being initialized (during checkpoint
-     * restore). */
-
-    if (!hdl->pal_handle) {
-        PAL_HANDLE palhdl = NULL;
-        int ret = chroot_reopen(hdl, &palhdl);
-        if (ret < 0) {
-            log_warning("%s: failed to open %s: %d", __func__, hdl->uri, ret);
-            return ret;
-        }
-        assert(palhdl);
-        hdl->pal_handle = palhdl;
-    }
-    return 0;
-}
-
 struct shim_fs_ops chroot_fs_ops = {
     .mount      = &chroot_mount,
     .flush      = &chroot_flush,
@@ -576,8 +508,6 @@ struct shim_fs_ops chroot_fs_ops = {
     .hstat      = &generic_inode_hstat,
     .truncate   = &chroot_truncate,
     .poll       = &generic_inode_poll,
-    .checkout   = &chroot_checkout,
-    .checkin    = &chroot_checkin,
 };
 
 struct shim_d_ops chroot_d_ops = {