[RFC] Security and Access Control for Conversational Memory

[HenryL27]

Introduction

As it stands, all users can see all conversations and interactions held in the conversational memory store. This can be an issue, since conversations (especially those using RAG (opensearch-project/ml-commons#1150)) may contain information derived from classified documents.

Goals

• When the security plugin is enabled, a user should only be able to see conversations that they own.

Proposed Solution

1. The .conversational-meta and .conversational-interactions indices will become system indices, so that only the admin role will be able to query it
2. The create APIs will read the user's name from the ThreadContext, and add it to records that get created.
3. The GetConversations API will read the user's name from the ThreadContext, and use that to filter the undelying query.
4. The GetInteractions and DeleteConversation APIs will read the user's name from the ThreadContext, and first check with the ConvoMeta object that the user is the owner of the requested conversation

Alternate Solution

Use the security plugin to create a role with DLS for the indices. Adding user info to the records is still required. But for (Get/Delete)(Conversations/Interactions) the security plugin should handle everything I think. I don't know if there is an automated way to set up a default role with the correct DLS access controls.

Discuss

How should we handle it if a user's permissions change with respect to other data in the cluster? If Alice gets demoted out of access to certain classified documents, what happens to her past conversations? They might have information taken from those documents, but they might also have her travel plans for the conference she's going to next week. Some sort of derived-classification might be necessary? Maybe RAG should write to each interaction the DocID's sent to the LLM, and then some sort of fine-grained access control can be implemented over interactions based on prefixes of docIDs? It gets complicated fast, but I think that's out of scope for this particular RFC/plugin. For now, we'll just focus on per-user security.