05-Adding_Memory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01a8b5c0-87cb-4302-8e3c-dc809d0039fb",
   "metadata": {},
   "source": [
    "# Understanding Memory in LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f73380-6395-4e9f-9c83-3f47a5d7e292",
   "metadata": {},
   "source": [
    "In the previous Notebook, we successfully explored how OpenAI models can enhance the results from Azure Cognitive Search. \n",
    "\n",
    "However, we have yet to discover how to engage in a conversation with the LLM. With [Bing Chat](http://chat.bing.com/), for example, this is possible, as it can understand and reference the previous responses.\n",
    "\n",
    "There is a common misconception that GPT models have memory. This is not true. While they possess knowledge, they do not retain information from previous questions asked to them.\n",
    "\n",
    "In this Notebook, our goal is to illustrate how we can effectively \"endow the LLM with memory\" by employing prompts and context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "733c782e-204c-47d0-8dae-c9df7091ab23",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.memory import ConversationBufferMemory\n",
    "from openai.error import OpenAIError\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.memory import CosmosDBChatMessageHistory\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "#custom libraries that we will use later in the app\n",
    "from common.utils import (\n",
    "    get_search_results,\n",
    "    update_vector_indexes,\n",
    "    model_tokens_limit,\n",
    "    num_tokens_from_docs,\n",
    "    num_tokens_from_string,\n",
    "    get_answer,\n",
    ")\n",
    "\n",
    "from common.prompts import COMBINE_CHAT_PROMPT_TEMPLATE\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "import logging\n",
    "\n",
    "# Get the root logger\n",
    "logger = logging.getLogger()\n",
    "# Set the logging level to a higher level to ignore INFO messages\n",
    "logger.setLevel(logging.WARNING)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6bc63c55-a57d-49a7-b6c7-0f18bca8199e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_BASE\"] = os.environ[\"AZURE_OPENAI_ENDPOINT\"]\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.environ[\"AZURE_OPENAI_API_KEY\"]\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc72b22-11c2-4df0-91b8-033d01829663",
   "metadata": {},
   "source": [
    "### Let's start with the basics\n",
    "Let's use a very simple example to see if the GPT model of Azure OpenAI have memory. We again will be using langchain to simplify our code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3eef5dc9-8b80-4085-980c-865fa41fa1f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"Tell me some use cases for reinforcement learning?\"\n",
    "FOLLOW_UP_QUESTION = \"Give me the main points of our conversation\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a00181d5-bd76-4ce4-a256-75ac5b58c60f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define model\n",
    "MODEL = \"gpt-35-turbo\"\n",
    "COMPLETION_TOKENS = 500\n",
    "# Create an OpenAI instance\n",
    "llm = AzureChatOpenAI(deployment_name=MODEL, temperature=0.5, max_tokens=COMPLETION_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9502d0f1-fddf-40d1-95d2-a1461dcc498a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create a very simple prompt template, just the question as is:\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"question\"],\n",
    "    template=\"{question}\",\n",
    ")\n",
    "\n",
    "chain = LLMChain(llm=llm, prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c5c9903e-15c7-4e05-87a1-58e5a7917ba2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has a wide range of applications across various domains. Here are some use cases for reinforcement learning:\n",
       "\n",
       "1. Game Playing: Reinforcement learning has been successfully applied to games like Chess, Go, and Atari games, where the agent learns to make optimal decisions by playing against itself or learning from human experts.\n",
       "\n",
       "2. Robotics: Reinforcement learning is used to train robots to perform complex tasks, such as grasping objects, walking, or navigating through dynamic environments, by rewarding desired behaviors and penalizing undesired ones.\n",
       "\n",
       "3. Autonomous Vehicles: Reinforcement learning can be used to train self-driving cars to make decisions in real-time, such as lane changing, merging, and navigating intersections, based on the surrounding environment and traffic conditions.\n",
       "\n",
       "4. Recommendation Systems: Reinforcement learning can be employed to build personalized recommendation systems that learn user preferences and provide relevant suggestions for movies, music, products, or advertisements.\n",
       "\n",
       "5. Finance: Reinforcement learning can be used to optimize trading strategies in financial markets by learning to make buy/sell decisions based on historical data and market conditions.\n",
       "\n",
       "6. Healthcare: Reinforcement learning can help in optimizing treatment plans by learning from patient data and medical guidelines to recommend personalized therapies or dosage adjustments.\n",
       "\n",
       "7. Resource Management: Reinforcement learning can be applied to optimize resource allocation and scheduling problems, such as managing energy consumption in smart grids, controlling traffic signals, or optimizing supply chain logistics.\n",
       "\n",
       "8. Natural Language Processing: Reinforcement learning can be used to train chatbots or virtual assistants to interact with users, understand natural language, and provide relevant responses.\n",
       "\n",
       "9. Education: Reinforcement learning can be employed to create intelligent tutoring systems that adapt to individual student needs, providing personalized feedback and guidance in the learning process.\n",
       "\n",
       "10. Healthcare Robotics: Reinforcement learning can be utilized to train robotic systems to assist in healthcare tasks, such as patient lifting, medication delivery, or physical therapy, while ensuring safety and patient comfort.\n",
       "\n",
       "These are just a few examples, and the potential applications of reinforcement learning are vast and expanding as research progresses."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see what the GPT model responds\n",
    "response = chain.run(QUESTION)\n",
    "printmd(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "99acaf3c-ce68-4b87-b24a-6065b15ff9a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I'm sorry, but as an AI language model, I don't have the ability to remember or recall past conversations. Once a conversation ends, the information is not retained. Is there anything specific you would like to discuss or ask about?\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Now let's ask a follow up question\n",
    "chain.run(FOLLOW_UP_QUESTION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e1c143-c95f-4566-a8b4-af8c42f08dd2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "As you can see, it doesn't remember what it just responded, sometimes it responds based only on the system prompt, or just randomly. This proof that the LLM does NOT have memory and that we need to give the memory as a a conversation history as part of the prompt, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0946ce71-6285-432e-b011-9c2dc1ba7b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_prompt = PromptTemplate(\n",
    "    input_variables=[\"history\", \"question\"],\n",
    "    template=\"\"\"\n",
    "                {history}\n",
    "                Human: {question}\n",
    "                AI:\n",
    "            \"\"\"\n",
    "    )\n",
    "chain = LLMChain(llm=llm, prompt=hist_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6d088e51-e5eb-4143-b87d-b2be429eb864",
   "metadata": {},
   "outputs": [],
   "source": [
    "Conversation_history = \"\"\"\n",
    "Human: {question}\n",
    "AI: {response}\n",
    "\"\"\".format(question=QUESTION, response=response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d99e34ad-5539-44dd-b080-3ad05efd2f01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "- Reinforcement learning has a wide range of applications across various domains.\n",
       "- Some use cases of reinforcement learning include game playing, robotics, autonomous vehicles, recommendation systems, finance, healthcare, resource management, natural language processing, education, and healthcare robotics.\n",
       "- Reinforcement learning can be used to train agents to make optimal decisions in games, perform complex tasks in robotics, make real-time decisions in autonomous vehicles, provide personalized recommendations, optimize trading strategies, optimize treatment plans in healthcare, optimize resource allocation and scheduling, train chatbots or virtual assistants, create intelligent tutoring systems, and assist in healthcare tasks.\n",
       "- The potential applications of reinforcement learning are vast and expanding as research progresses."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain.run({\"history\":Conversation_history, \"question\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045e5af6-55d6-4353-b3f6-3275c95db00a",
   "metadata": {},
   "source": [
    "**Bingo!**, so we now know how to create a chatbot using LLMs, we just need to keep the state/history of the conversation and pass it as context every time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafd1694-0077-4aa8-bd01-e9f763ce36a3",
   "metadata": {},
   "source": [
    "## Now that we understand the concept of memory via adding history as a context, let's go back to our GPT Smart Search engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ba257e86-fd90-4a51-a72d-27000913e8c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since Memory adds tokens to the prompt, we would need a better model that allows more space on the prompt\n",
    "MODEL = \"gpt-35-turbo-16k\"\n",
    "COMPLETION_TOKENS = 1000\n",
    "llm = AzureChatOpenAI(deployment_name=MODEL, temperature=0.5, max_tokens=COMPLETION_TOKENS)\n",
    "embedder = OpenAIEmbeddings(deployment=\"text-embedding-ada-002\", chunk_size=1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ef9f459b-e8b8-40b9-a94d-80c079968594",
   "metadata": {},
   "outputs": [],
   "source": [
    "index1_name = \"cogsrch-index-files\"\n",
    "index2_name = \"cogsrch-index-csv\"\n",
    "index3_name = \"cogsrch-index-books-vector\"\n",
    "text_indexes = [index1_name, index2_name]\n",
    "vector_indexes = [index+\"-vector\" for index in text_indexes] + [index3_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b01852c2-6192-496c-adff-4270f9380469",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of results: 5\n",
      "CPU times: user 528 ms, sys: 40.4 ms, total: 568 ms\n",
      "Wall time: 3.02 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Search in text-based indexes first and update vector indexes\n",
    "k=10 # Top k results per each text-based index\n",
    "ordered_results = get_search_results(QUESTION, text_indexes, k=k, reranker_threshold=1, vector_search=False)\n",
    "update_vector_indexes(ordered_search_results=ordered_results, embedder=embedder)\n",
    "\n",
    "# Search in all vector-based indexes available\n",
    "similarity_k = 5 # top results from multi-vector-index similarity search\n",
    "ordered_results = get_search_results(QUESTION, vector_indexes, k=k, vector_search=True,\n",
    "                                        similarity_k=similarity_k,\n",
    "                                        query_vector = embedder.embed_query(QUESTION))\n",
    "print(\"Number of results:\",len(ordered_results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ca500dd8-148c-4d8a-b58b-2df4c957459d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below line if you want to inspect the ordered results\n",
    "# ordered_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9b2a3595-c3b7-4376-b9c5-0db7a42b3ee4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of chunks: 5\n"
     ]
    }
   ],
   "source": [
    "top_docs = []\n",
    "for key,value in ordered_results.items():\n",
    "    location = value[\"location\"] if value[\"location\"] is not None else \"\"\n",
    "    top_docs.append(Document(page_content=value[\"chunk\"], metadata={\"source\": location+os.environ['BLOB_SAS_TOKEN']}))\n",
    "        \n",
    "print(\"Number of chunks:\",len(top_docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c26d7540-feb8-4581-849e-003f4bf2a601",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System prompt token count: 2464\n",
      "Max Completion Token count: 1000\n",
      "Combined docs (context) token count: 1019\n",
      "--------\n",
      "Requested token count: 4483\n",
      "Token limit for gpt-35-turbo-16k : 16384\n",
      "Chain Type selected: stuff\n"
     ]
    }
   ],
   "source": [
    "# Calculate number of tokens of our docs\n",
    "if(len(top_docs)>0):\n",
    "    tokens_limit = model_tokens_limit(MODEL) # this is a custom function we created in common/utils.py\n",
    "    prompt_tokens = num_tokens_from_string(COMBINE_CHAT_PROMPT_TEMPLATE) # this is a custom function we created in common/utils.py\n",
    "    context_tokens = num_tokens_from_docs(top_docs) # this is a custom function we created in common/utils.py\n",
    "    \n",
    "    requested_tokens = prompt_tokens + context_tokens + COMPLETION_TOKENS\n",
    "    \n",
    "    chain_type = \"map_reduce\" if requested_tokens > 0.9 * tokens_limit else \"stuff\"  \n",
    "    \n",
    "    print(\"System prompt token count:\",prompt_tokens)\n",
    "    print(\"Max Completion Token count:\", COMPLETION_TOKENS)\n",
    "    print(\"Combined docs (context) token count:\",context_tokens)\n",
    "    print(\"--------\")\n",
    "    print(\"Requested token count:\",requested_tokens)\n",
    "    print(\"Token limit for\", MODEL, \":\", tokens_limit)\n",
    "    print(\"Chain Type selected:\", chain_type)\n",
    "        \n",
    "else:\n",
    "    print(\"NO RESULTS FROM AZURE SEARCH\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3ce6efa9-2b8f-4810-904d-5986b4ae0372",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning can be applied in various use cases, including:\n",
       "1. Learning prevention strategies for epidemics of infectious diseases, such as pandemic influenza, by automatically learning mitigation policies in complex epidemiological models with a large state space<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[1]</a></sup>.\n",
       "2. Learning sparse reward tasks efficiently by combining self-imitation learning with exploration bonuses, which enhances exploration by producing intrinsic rewards when the agent visits novel states<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[2]</a></sup>.\n",
       "3. Personalized hybrid recommendation algorithms for music based on reinforcement learning, which consider the simulation of the interaction process to capture changes in listeners' preferences sensitively<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[3]</a></sup>.\n",
       "4. Automatic feature engineering in machine learning projects, where a framework called CAFEM (Cross-data Automatic Feature Engineering Machine) is used to optimize feature transformation and improve learning performance<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[4]</a></sup>.\n",
       "5. Job scheduling in data centers, where an Advantage Actor-Critic (A2C) deep reinforcement learning approach called A2cScheduler is used to automatically learn scheduling policies and achieve competitive scheduling performance<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[5]</a></sup>.\n",
       "\n",
       "These references provide more details and information about the respective use cases."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.97 ms, sys: 0 ns, total: 9.97 ms\n",
      "Wall time: 21.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Get the answer\n",
    "response = get_answer(llm=llm, docs=top_docs, query=QUESTION, language=\"English\", chain_type=chain_type)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27501f1b-7db0-4ee3-9cb1-e609254ffa3d",
   "metadata": {},
   "source": [
    "And if we ask the follow up question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5cf5b323-3b9c-479b-8502-acfc4f7915dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The main points of our conversation are about reinforcement learning in various domains. We discussed the use of deep reinforcement learning to learn prevention strategies in the context of pandemic influenza<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[1]</a></sup>. We also talked about the Explore-then-Exploit framework, which combines self-imitation learning with exploration bonuses to improve performance in sparse reward tasks<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[2]</a></sup>. Additionally, we discussed the use of reinforcement learning in personalized music recommendation systems<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[3]</a></sup>. We also touched upon the topic of automatic feature engineering using a framework called CAFEM<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[4]</a></sup>. Lastly, we discussed the A2cScheduler, an Advantage Actor-Critic deep reinforcement learning approach for job scheduling in data centers<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[5]</a></sup>.\n",
       "\n",
       "Anything else I can help you with?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = get_answer(llm=llm, docs=top_docs,  query=FOLLOW_UP_QUESTION, language=\"English\", chain_type=chain_type)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035fa6e6-226c-400f-a504-30255385f43b",
   "metadata": {},
   "source": [
    "You might get a different response from above, but it doesn't matter what response you get, it will be based on the context given, not on previous answers.\n",
    "\n",
    "Until now we just have the same as the prior Notebook 03: results from Azure Search enhanced by OpenAI model, with no memory\n",
    "\n",
    "**Now let's add memory to it:**\n",
    "\n",
    "Reference: https://python.langchain.com/docs/modules/memory/how_to/adding_memory_chain_multiple_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d98b876e-d264-48ae-b5ed-9801d6a9152b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has various use cases across different domains. Here are some examples:\n",
       "\n",
       "1. **Epidemic prevention strategies**: Reinforcement learning can be used to automatically learn prevention strategies for infectious diseases. For example, a study used deep reinforcement learning to learn mitigation policies in complex epidemiological models with a large state space<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[1]</a></sup>.\n",
       "\n",
       "2. **Sparse reward tasks**: Reinforcement learning can be challenging when dealing with tasks that have sparse rewards. Self-imitation learning and exploration bonuses are two approaches that can address this challenge. A recent framework called Explore-then-Exploit (EE) interleaves self-imitation learning with an exploration bonus to enhance both exploitation and exploration in learning tasks<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[2]</a></sup>.\n",
       "\n",
       "3. **Personalized recommendation systems**: Reinforcement learning can be applied to personalized recommendation systems. For example, a personalized hybrid recommendation algorithm for music based on reinforcement learning was proposed. It uses techniques like weighted matrix factorization and convolutional neural networks to learn and extract song feature vectors, and it continuously updates the model based on the preferences of listeners for songs and song transitions<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[3]</a></sup>.\n",
       "\n",
       "4. **Automatic feature engineering**: Feature engineering is a crucial task in machine learning projects. Reinforcement learning can be used to automate feature engineering. For example, a framework called Cross-data Automatic Feature Engineering Machine (CAFEM) formalizes the feature engineering problem as an optimization problem over a Feature Transformation Graph (FTG). It includes a feature engineering learner that learns fine-grained strategies on a single dataset and a cross-data component that speeds up feature engineering learning on unseen datasets<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[4]</a></sup>.\n",
       "\n",
       "5. **Job scheduling in data centers**: Reinforcement learning can be applied to job scheduling in data centers. For example, an Advantage Actor-Critic (A2C) deep reinforcement learning-based approach called A2cScheduler was proposed for job scheduling. It consists of two agents, an actor and a critic, that work together to learn the scheduling policy and reduce estimation errors. The approach showed competitive performance using both simulated workloads and real data from an academic data center<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# memory object, which is neccessary to track the inputs/outputs and hold a conversation.\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\",input_key=\"question\")\n",
    "\n",
    "response = get_answer(llm=llm, docs=top_docs, query=QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                        memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bf28927b-d9ee-4412-bb07-13e055e832a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Here are the main points of our conversation:\n",
       "\n",
       "1. Reinforcement learning has various use cases across different domains.\n",
       "2. Some examples of use cases for reinforcement learning include epidemic prevention strategies, sparse reward tasks, personalized recommendation systems, automatic feature engineering, and job scheduling in data centers.\n",
       "\n",
       "Would you like more information about any of these use cases?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Now we add a follow up question:\n",
    "response = get_answer(llm=llm, docs=top_docs, query=FOLLOW_UP_QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3830b0b8-0ca2-4d0a-9747-f6273368002b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The main points of our conversation are as follows:\n",
       "\n",
       "1. Reinforcement learning has various use cases across different domains.\n",
       "2. Some examples of use cases for reinforcement learning include:\n",
       "   - Epidemic prevention strategies, where deep reinforcement learning is used to learn prevention strategies for infectious diseases<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[1]</a></sup>.\n",
       "   - Sparse reward tasks, where self-imitation learning and exploration bonuses can be used to address the challenge of sparse rewards<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[2]</a></sup>.\n",
       "   - Personalized recommendation systems, where reinforcement learning can be applied to recommend personalized song sequences<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[3]</a></sup>.\n",
       "   - Automatic feature engineering, where reinforcement learning can be used to automate the process of feature engineering<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[4]</a></sup>.\n",
       "   - Job scheduling in data centers, where reinforcement learning can be applied to optimize job scheduling and resource allocation<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[5]</a></sup>.\n",
       "\n",
       "Please let me know if you would like more information about any of these use cases."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Another follow up query\n",
    "response = get_answer(llm=llm, docs=top_docs, query=\"Thank you\", language=\"English\", chain_type=chain_type,  \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "111e732b-3c8c-4df3-8fcb-c3d01e7bec74",
   "metadata": {},
   "source": [
    "You might get a different answer on the above cell, and it is ok, this bot is not yet well configured to answer any question that is not related to its knowledge base, including salutations.\n",
    "\n",
    "Let's check our memory to see that it's keeping the conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1279692c-7eb0-4300-8a66-c7025f02c318",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Human: Tell me some use cases for reinforcement learning?\\nAI: Reinforcement learning has various use cases across different domains. Here are some examples:\\n\\n1. **Epidemic prevention strategies**: Reinforcement learning can be used to automatically learn prevention strategies for infectious diseases. For example, a study used deep reinforcement learning to learn mitigation policies in complex epidemiological models with a large state space<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[1]</a></sup>.\\n\\n2. **Sparse reward tasks**: Reinforcement learning can be challenging when dealing with tasks that have sparse rewards. Self-imitation learning and exploration bonuses are two approaches that can address this challenge. A recent framework called Explore-then-Exploit (EE) interleaves self-imitation learning with an exploration bonus to enhance both exploitation and exploration in learning tasks<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[2]</a></sup>.\\n\\n3. **Personalized recommendation systems**: Reinforcement learning can be applied to personalized recommendation systems. For example, a personalized hybrid recommendation algorithm for music based on reinforcement learning was proposed. It uses techniques like weighted matrix factorization and convolutional neural networks to learn and extract song feature vectors, and it continuously updates the model based on the preferences of listeners for songs and song transitions<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[3]</a></sup>.\\n\\n4. **Automatic feature engineering**: Feature engineering is a crucial task in machine learning projects. Reinforcement learning can be used to automate feature engineering. For example, a framework called Cross-data Automatic Feature Engineering Machine (CAFEM) formalizes the feature engineering problem as an optimization problem over a Feature Transformation Graph (FTG). It includes a feature engineering learner that learns fine-grained strategies on a single dataset and a cross-data component that speeds up feature engineering learning on unseen datasets<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[4]</a></sup>.\\n\\n5. **Job scheduling in data centers**: Reinforcement learning can be applied to job scheduling in data centers. For example, an Advantage Actor-Critic (A2C) deep reinforcement learning-based approach called A2cScheduler was proposed for job scheduling. It consists of two agents, an actor and a critic, that work together to learn the scheduling policy and reduce estimation errors. The approach showed competitive performance using both simulated workloads and real data from an academic data center<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X\\nHuman: Give me the main points of our conversation\\nAI: Here are the main points of our conversation:\\n\\n1. Reinforcement learning has various use cases across different domains.\\n2. Some examples of use cases for reinforcement learning include epidemic prevention strategies, sparse reward tasks, personalized recommendation systems, automatic feature engineering, and job scheduling in data centers.\\n\\nWould you like more information about any of these use cases?\\nHuman: Thank you\\nAI: The main points of our conversation are as follows:\\n\\n1. Reinforcement learning has various use cases across different domains.\\n2. Some examples of use cases for reinforcement learning include:\\n   - Epidemic prevention strategies, where deep reinforcement learning is used to learn prevention strategies for infectious diseases<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[1]</a></sup>.\\n   - Sparse reward tasks, where self-imitation learning and exploration bonuses can be used to address the challenge of sparse rewards<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[2]</a></sup>.\\n   - Personalized recommendation systems, where reinforcement learning can be applied to recommend personalized song sequences<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[3]</a></sup>.\\n   - Automatic feature engineering, where reinforcement learning can be used to automate the process of feature engineering<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[4]</a></sup>.\\n   - Job scheduling in data centers, where reinforcement learning can be applied to optimize job scheduling and resource allocation<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\" target=\"_blank\">[5]</a></sup>.\\n\\nPlease let me know if you would like more information about any of these use cases.'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "memory.buffer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87405173",
   "metadata": {},
   "source": [
    "## Using CosmosDB as persistent memory\n",
    "\n",
    "In previous cell we have added local RAM memory to our chatbot. However, it is not persistent, it gets deleted once the app user's session is terminated. It is necessary then to use a Database for persistent storage of each of the bot user conversations, not only for Analytics and Auditing, but also if we wisg to provide recommendations. \n",
    "\n",
    "Here we will store the conversation history into CosmosDB for future auditing purpose.\n",
    "We will use a class in LangChain use CosmosDBChatMessageHistory, see [HERE](https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cosmos_db.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c7131daa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create CosmosDB instance from langchain cosmos class.\n",
    "cosmos = CosmosDBChatMessageHistory(\n",
    "    cosmos_endpoint=os.environ['AZURE_COSMOSDB_ENDPOINT'],\n",
    "    cosmos_database=os.environ['AZURE_COSMOSDB_NAME'],\n",
    "    cosmos_container=os.environ['AZURE_COSMOSDB_CONTAINER_NAME'],\n",
    "    connection_string=os.environ['AZURE_COMOSDB_CONNECTION_STRING'],\n",
    "    session_id=\"Agent-Test-Session\" + str(random.randint(1, 1000)),\n",
    "    user_id=\"Agent-Test-User\" + str(random.randint(1, 1000))\n",
    "    )\n",
    "\n",
    "# prepare the cosmosdb instance\n",
    "cosmos.prepare_cosmos()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d87cc7c6-5ef1-4492-b133-9f63a392e223",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create or Memory Object\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\",input_key=\"question\",chat_memory=cosmos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "27ceb47a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has various use cases in different domains. Here are some examples:\n",
       "\n",
       "1. **Epidemic Prevention**: Reinforcement learning can be used to automatically learn prevention strategies for infectious diseases. For example, a study used deep reinforcement learning to learn mitigation policies in complex epidemiological models with a large state space<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[1]</a></sup>.\n",
       "\n",
       "2. **Sparse Reward Tasks**: Reinforcement learning is used to tackle tasks with sparse rewards, where efficient exploitation and exploration are required. One approach is self-imitation learning, which encourages exploitation by imitating past good trajectories. Another approach is exploration bonuses, which enhance exploration by providing intrinsic rewards. A novel framework called Explore-then-Exploit (EE) has been introduced, which interleaves self-imitation learning with an exploration bonus to strengthen the effect of these two algorithms<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[2]</a></sup>.\n",
       "\n",
       "3. **Personalized Recommendation Systems**: Reinforcement learning can be used to improve personalized recommendation systems. For example, a personalized hybrid recommendation algorithm for music based on reinforcement learning was proposed. It recommends song sequences that match listeners' preferences better by simulating the interaction process and updating the model continuously based on their preferences<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[3]</a></sup>.\n",
       "\n",
       "4. **Feature Engineering**: Reinforcement learning can be used to automate feature engineering, which is a time-consuming and challenging task in machine learning projects. A framework called Cross-data Automatic Feature Engineering Machine (CAFEM) has been proposed, which formalizes the feature engineering problem as an optimization problem and learns fine-grained feature engineering strategies using reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[4]</a></sup>.\n",
       "\n",
       "5. **Job Scheduling**: Reinforcement learning can be used for efficient job scheduling in data centers. An approach called A2cScheduler, based on Advantage Actor-Critic (A2C) deep reinforcement learning, has been proposed for job scheduling. It consists of two agents, the actor and the critic, which learn the scheduling policy and reduce estimation error, respectively<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[5]</a></sup>.\n",
       "\n",
       "These are just a few examples of the use cases for reinforcement learning. It"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Testing using our Question\n",
    "response = get_answer(llm=llm, docs=top_docs, query=QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                        memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1a5ff826",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Here are the main points of our conversation:\n",
       "\n",
       "1. Reinforcement learning has various use cases in different domains.\n",
       "2. One use case is epidemic prevention, where deep reinforcement learning is used to learn prevention strategies for infectious diseases<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[1]</a></sup>.\n",
       "3. Another use case is tackling tasks with sparse rewards, where self-imitation learning and exploration bonuses are used to enhance exploitation and exploration<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[2]</a></sup>.\n",
       "4. Reinforcement learning can be used to improve personalized recommendation systems, such as a hybrid recommendation algorithm for music based on reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[3]</a></sup>.\n",
       "5. Reinforcement learning can automate feature engineering tasks, such as a framework called CAFEM that learns fine-grained feature engineering strategies using reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[4]</a></sup>.\n",
       "6. Reinforcement learning can be used for efficient job scheduling in data centers, such as the A2cScheduler approach based on Advantage Actor-Critic (A2C) deep reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[5]</a></sup>.\n",
       "\n",
       "These points summarize the main use cases of reinforcement learning that we discussed. Is there anything else I can help you with?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Now we add a follow up question:\n",
    "response = get_answer(llm=llm, docs=top_docs, query=FOLLOW_UP_QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "be1620fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Based on our conversation, here are the main points:\n",
       "\n",
       "1. Reinforcement learning has various use cases in different domains.\n",
       "2. One use case is epidemic prevention, where deep reinforcement learning is used to learn prevention strategies for infectious diseases<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[1]</a></sup>.\n",
       "3. Another use case is tackling tasks with sparse rewards, where self-imitation learning and exploration bonuses are used to enhance exploitation and exploration<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[2]</a></sup>.\n",
       "4. Reinforcement learning can be used to improve personalized recommendation systems, such as a hybrid recommendation algorithm for music based on reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[3]</a></sup>.\n",
       "5. Reinforcement learning can automate feature engineering tasks, such as a framework called CAFEM that learns fine-grained feature engineering strategies using reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[4]</a></sup>.\n",
       "6. Reinforcement learning can be used for efficient job scheduling in data centers, such as the A2cScheduler approach based on Advantage Actor-Critic (A2C) deep reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[5]</a></sup>.\n",
       "\n",
       "These points summarize the main use cases of reinforcement learning that we discussed. Let me know if there's anything else I can assist you with."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Another follow up query\n",
    "response = get_answer(llm=llm, docs=top_docs, query=\"Thank you\", language=\"English\", chain_type=chain_type,  \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc5ac98",
   "metadata": {},
   "source": [
    "Let's check our Azure CosmosDB to see the whole conversation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e1d7688a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[HumanMessage(content='Tell me some use cases for reinforcement learning?', additional_kwargs={}, example=False),\n",
       " AIMessage(content='Reinforcement learning has various use cases in different domains. Here are some examples:\\n\\n1. **Epidemic Prevention**: Reinforcement learning can be used to automatically learn prevention strategies for infectious diseases. For example, a study used deep reinforcement learning to learn mitigation policies in complex epidemiological models with a large state space<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[1]</a></sup>.\\n\\n2. **Sparse Reward Tasks**: Reinforcement learning is used to tackle tasks with sparse rewards, where efficient exploitation and exploration are required. One approach is self-imitation learning, which encourages exploitation by imitating past good trajectories. Another approach is exploration bonuses, which enhance exploration by providing intrinsic rewards. A novel framework called Explore-then-Exploit (EE) has been introduced, which interleaves self-imitation learning with an exploration bonus to strengthen the effect of these two algorithms<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[2]</a></sup>.\\n\\n3. **Personalized Recommendation Systems**: Reinforcement learning can be used to improve personalized recommendation systems. For example, a personalized hybrid recommendation algorithm for music based on reinforcement learning was proposed. It recommends song sequences that match listeners\\' preferences better by simulating the interaction process and updating the model continuously based on their preferences<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[3]</a></sup>.\\n\\n4. **Feature Engineering**: Reinforcement learning can be used to automate feature engineering, which is a time-consuming and challenging task in machine learning projects. A framework called Cross-data Automatic Feature Engineering Machine (CAFEM) has been proposed, which formalizes the feature engineering problem as an optimization problem and learns fine-grained feature engineering strategies using reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[4]</a></sup>.\\n\\n5. **Job Scheduling**: Reinforcement learning can be used for efficient job scheduling in data centers. An approach called A2cScheduler, based on Advantage Actor-Critic (A2C) deep reinforcement learning, has been proposed for job scheduling. It consists of two agents, the actor and the critic, which learn the scheduling policy and reduce estimation error, respectively<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[5]</a></sup>.\\n\\nThese are just a few examples of the use cases for reinforcement learning. It', additional_kwargs={}, example=False),\n",
       " HumanMessage(content='Give me the main points of our conversation', additional_kwargs={}, example=False),\n",
       " AIMessage(content='Here are the main points of our conversation:\\n\\n1. Reinforcement learning has various use cases in different domains.\\n2. One use case is epidemic prevention, where deep reinforcement learning is used to learn prevention strategies for infectious diseases<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[1]</a></sup>.\\n3. Another use case is tackling tasks with sparse rewards, where self-imitation learning and exploration bonuses are used to enhance exploitation and exploration<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[2]</a></sup>.\\n4. Reinforcement learning can be used to improve personalized recommendation systems, such as a hybrid recommendation algorithm for music based on reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[3]</a></sup>.\\n5. Reinforcement learning can automate feature engineering tasks, such as a framework called CAFEM that learns fine-grained feature engineering strategies using reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[4]</a></sup>.\\n6. Reinforcement learning can be used for efficient job scheduling in data centers, such as the A2cScheduler approach based on Advantage Actor-Critic (A2C) deep reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[5]</a></sup>.\\n\\nThese points summarize the main use cases of reinforcement learning that we discussed. Is there anything else I can help you with?', additional_kwargs={}, example=False),\n",
       " HumanMessage(content='Thank you', additional_kwargs={}, example=False),\n",
       " AIMessage(content='Based on our conversation, here are the main points:\\n\\n1. Reinforcement learning has various use cases in different domains.\\n2. One use case is epidemic prevention, where deep reinforcement learning is used to learn prevention strategies for infectious diseases<sup><a href=\"https://arxiv.org/pdf/2003.13676v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[1]</a></sup>.\\n3. Another use case is tackling tasks with sparse rewards, where self-imitation learning and exploration bonuses are used to enhance exploitation and exploration<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206262/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[2]</a></sup>.\\n4. Reinforcement learning can be used to improve personalized recommendation systems, such as a hybrid recommendation algorithm for music based on reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[3]</a></sup>.\\n5. Reinforcement learning can automate feature engineering tasks, such as a framework called CAFEM that learns fine-grained feature engineering strategies using reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[4]</a></sup>.\\n6. Reinforcement learning can be used for efficient job scheduling in data centers, such as the A2cScheduler approach based on Advantage Actor-Critic (A2C) deep reinforcement learning<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2024-10-02T01:02:07Z&st=2023-08-03T17:02:07Z&spr=https&sig=gLxStXFSY6X29OPpPDpBEhoQDdtJNDrMVExNYJ%2BhmBQ%3D\">[5]</a></sup>.\\n\\nThese points summarize the main use cases of reinforcement learning that we discussed. Let me know if there\\'s anything else I can assist you with.', additional_kwargs={}, example=False)]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#load message from cosmosdb\n",
    "cosmos.load_messages()\n",
    "cosmos.messages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e30694-ae2a-47bb-a5c7-db51ecdbba1e",
   "metadata": {},
   "source": [
    "![CosmosDB Memory](./images/cosmos-chathistory.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6789cada-23a3-451a-a91a-0906ceb0bd14",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### Adding memory to our application allows the user to have a conversation, however this feature is not something that comes with the LLM, but instead, memory is something that we must provide to the LLM in form of context of the question.\n",
    "\n",
    "We added persitent memory using CosmosDB.\n",
    "\n",
    "We also can notice that the current chain that we are using is smart, but not that much. Although we have given memory to it, it searches for similar docs everytime, regardless of the input and it struggles to respond to prompts like: Hello, Thank you, Bye, What's your name, What's the weather and any other task that is not search in the knowledge base.\n",
    "\n",
    "\n",
    "## <u>Important Note</u>:<br>\n",
    "As we proceed, while all the code will remain compatible with GPT-3.5 models, we highly recommend transitioning to GPT-4. Here's why:\n",
    "\n",
    "**GPT-3.5-Turbo** can be likened to a 7-year-old child. You can provide it with concise instructions, but it frequently struggles to follow them accurately. Additionally, its limited memory can make sustained conversations challenging.\n",
    "\n",
    "**GPT-3.5-Turbo-16k** resembles the same 7-year-old, but with an increased attention span for longer instructions. However, it still faces difficulties accurately executing them about half the time.\n",
    "\n",
    "**GPT-4** exhibits the capabilities of a 10-12-year-old child. It possesses enhanced reasoning skills and more consistently adheres to instructions. While its memory retention for instructions is moderate, it excels at following them.\n",
    "\n",
    "**GPT-4-32k** is akin to the 10-12-year-old child with an extended memory. It comprehends lengthy sets of instructions and engages in meaningful conversations. Thanks to its robust memory, it offers detailed responses.\n",
    "\n",
    "Understanding this analogy above will become clearer as you complete the final notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c629ebf4-aced-45b7-a6a2-315810d37d48",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We know now how to do a Smart Search Engine that can power a chatbot!! great!\n",
    "\n",
    "But, does this solve all the possible scenarios that a virtual assistant will require?  **What about if the answer to the Smart Search Engine is not related to text, but instead requires to look into tabular data?** The next notebook explains and solves the tabular problem and the concept of Agents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d4d9da4-3918-4da6-b235-a3320f0dcb12",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}