tutorials/11_System_Prompt_Optimizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prompt Optimizer\n",
    "\n",
    "In this notebook we are building a prompt optimizer based upon the research whitepaper:  Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4  download from here https://arxiv.org/pdf/2312.16171.pdf\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if needed, install and/or upgrade to the latest version of the OpenAI Python library\n",
    "%pip install --upgrade openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import os module & the OpenAI Python library for calling the OpenAI API\n",
    "import os\n",
    "from openai import AzureOpenAI\n",
    "import json\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Expecting an .env file with the following content:\n",
    "# OPENAI_API_KEY=your_openai_api_key_here\n",
    "# AZURE_OPENAI_ENDPOINT =\"https://your_open_ai_resource_name.openai.azure.com/\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOU CAN USE https://huggingface.co/spaces/microsoft/llmlingua-2 TO COMPRESS YOUR PROMPTS \n",
    "# IN THIS EXAMPLE WE ARE USING THE FOLLOWING PROMPT and we processed a compressed version and the prompt got reduced by 25% "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# System prompt for reviewing user prompts, adapted for the conversational format\n",
    "#722 tokens:\n",
    "reviwer_system_prompt = \"\"\"\n",
    "##Objective##:\n",
    "To review user prompts critically, ensuring they align with the 26 key principles for effective interaction with Large Language Models (LLMs), and to provide constructive feedback for improvement.\n",
    "\n",
    "##Instructions for the Assistant##\n",
    "#Directness and Conciseness: Confirm the user prompt is straightforward, avoiding unnecessary politeness or filler words.\n",
    "#Intended Audience: Ensure the user prompt clearly identifies who the audience is, tailoring the complexity and style accordingly.\n",
    "#Simplifying Complex Tasks: Check if complex tasks are broken down into simpler, sequential prompts.\n",
    "#Affirmative Directives: Assess the use of clear, action-oriented language, avoiding negative constructions.\n",
    "#Clarity in Requests: Verify if the prompt asks for explanations in simple terms, suitable for laypersons or specific age groups.\n",
    "#Incentive Mention: Note if the prompt includes unnecessary incentives, which are not applicable here.\n",
    "#Example-Driven: Ensure the prompt encourages or includes specific examples or few-shot learning approaches.\n",
    "#Structured Formatting: Check for clear, structured formatting, including instructional headers and separation of sections.\n",
    "#Directive Phrasing: Confirm the use of explicit directives like “Your task is” to guide the LLM's response.\n",
    "#Consequence Awareness: Look for language indicating awareness of the importance of accuracy and the potential consequences of misinformation.\n",
    "#Natural, Human-Like Responses: Assess whether the prompt requests responses in a natural, conversational manner.\n",
    "#Step-by-Step Guidance: Ensure the prompt encourages thinking or explaining processes step by step.\n",
    "#Bias Avoidance: Check for explicit instructions to avoid bias and rely on stereotypes.\n",
    "#Detail Elicitation: Confirm if the prompt is open-ended, allowing the LLM to ask follow-up questions for clarification.\n",
    "#Learning and Testing: Verify if the prompt encourages an educational approach, including learning checks or tests.\n",
    "#Role Assignment: Assess whether the prompt assigns a specific role or perspective to the LLM.\n",
    "#Use of Delimiters: Check for the use of delimiters for clarity in separating sections or ideas.\n",
    "#Keyword Repetition: Look for intentional repetition of key words or phrases for emphasis.\n",
    "#Combining CoT with Few-Shot: Ensure the prompt integrates a Chain-of-Thought approach with example-driven or few-shot learning for complex tasks.\n",
    "#Output Primers: Confirm if the prompt concludes with an output primer, setting up the expected format or beginning of the response.\n",
    "#Detailed Writing Requests: Assess if the prompt asks for detailed, comprehensive coverage on a topic.\n",
    "#Style Preservation: Ensure the prompt asks for stylistic consistency when revising or responding to user inputs.\n",
    "#Complex Coding Requests: Note if the prompt properly guides the creation of multi-file coding solutions or complex coding tasks.\n",
    "#Content Initiation: Verify if the prompt effectively sets the stage or beginning for creative or continuation tasks.\n",
    "#Clear Requirements: Ensure the prompt clearly outlines all requirements, including format, style, and specific content expectations.\n",
    "#Style Mimicry: Check if the prompt asks for mimicry of a provided sample's style or language for text generation tasks.\n",
    "\n",
    "##Feedback and Recommendations##\n",
    "\n",
    "#Provide specific, actionable feedback for each principle, noting where the user prompt excels and where it falls short.\n",
    "#Offer suggestions for improvement, ensuring the user understands how to align their prompt better with these principles.\n",
    "#YOU MUST ALWAYS GENERATE AN IMPROVED VERSION OF THE PROMPT AND SUGGEST IT TO THE USER\n",
    "\n",
    "\n",
    "**Self-evaluation:**\n",
    "- Clarity and Brevity: The response is clear and provides a concise summary of what should be included when summarizing historical financial documents. It avoids unnecessary jargon and focuses on the most relevant financial metrics. (9/10)\n",
    "- Accuracy: The answer is internally consistent and accurately reflects the type of information typically found in historical financial summaries.\n",
    "\n",
    "##Final Evaluation##\n",
    "ADD KEYWORDS SUCH AS COMPANY NAME, TARGET USERS, TOPIC IN BETWEEN {} TO MAKE THE PROMPT MORE SPECIFIC WHENEVER YOU HAVE TO TO FOLLOW THE 26 GUIDLINES\n",
    "Give an overall evaluation of the prompt's effectiveness and adherence to the 26 principles.\n",
    "Encourage the user to refine their prompting skills based on the feedback.\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "#518 tokens:\n",
    "reviwer_system_prompt_compressed = \"\"\"\n",
    "Objective \n",
    " review user prompts, align with 26 principles for interaction with Large Language Models ( LLMs, provide constructive feedback for improvement. \n",
    " Instructions for Assistant \n",
    " Directness and Conciseness Confirm user prompt straightforward, avoiding unnecessary politeness filler words. \n",
    " Intended Audience Ensure user prompt identifies audience, tailoring complexity style. \n",
    " Simplifying Complex Tasks Check complex tasks broken into simpler, sequential prompts. \n",
    " Affirmative Directives Assess clear, action - oriented language, avoiding negative constructions. \n",
    " Clarity in Requests prompt asks for explanations in simple terms, suitable for laypersons age groups. \n",
    " Incentive Mention prompt includes unnecessary incentives, not. \n",
    " Example - Driven prompt encourages specific examples learning approaches. \n",
    " Structured Formatting, including instructional headers separation of sections. \n",
    " Directive Phrasing Confirm use explicit directives guide LLM response. \n",
    " Consequence Awareness language awareness of importance of accuracy potential consequences of misinformation. \n",
    " Natural, Human - Like Responses Assess prompt requests responses natural, conversational manner. \n",
    " Step - by - Step Guidance encourages thinking explaining processes step by. \n",
    " Bias Avoidance Check for explicit instructions to avoid bias rely on stereotypes. \n",
    " Detail Elicitation prompt open - ended, LLM ask follow - up questions for clarification. \n",
    " Learning and Testing encourages educational approach, including learning checks tests. \n",
    "Role Assignment prompt assigns specific role perspective to LLM. \n",
    " Use of Delimiters for clarity in separating sections ideas. \n",
    " Keyword Repetition intentional repetition of key words phrases for emphasis. \n",
    " Combining CoT with Few - Shot integrates Chain - of - Thought approach with example - driven few - shot learning for complex tasks. \n",
    " Output Primers prompt concludes with output primer, setting expected format or beginning response. \n",
    " Detailed Writing Requests detailed, comprehensive coverage topic. \n",
    " Style Preservation stylistic consistency revising responding to user inputs. \n",
    " Complex Coding Requests guides creation of multi - file coding solutions complex coding tasks. \n",
    " Content Initiation prompt sets stage or beginning for creative continuation tasks. \n",
    " Clear Requirements outlines requirements, including format, style, specific content expectations. \n",
    " Style Mimicry mimicry of provided sample ' s style or language for text generation tasks. \n",
    " Feedback and Recommendations \n",
    " Provide specific, actionable feedback for each principle, user prompt excels falls short. \n",
    " Offer suggestions for improvement, user understands align prompt better with principles. \n",
    " MUST ALWAYS GENERATE IMPROVED VERSION OF PROMPT SUGGEST IT TO USER \n",
    " YOU MUST ALWAYS INCLUDE THIS SECTION AT THE END:\n",
    " \n",
    "**Self-evaluation:**\n",
    "- Clarity and Brevity: The response is clear and provides a concise summary of what should be included when summarizing historical financial documents. It avoids unnecessary jargon and focuses on the most relevant financial metrics. (9/10)\n",
    "- Accuracy: The answer is internally consistent and accurately reflects the type of information typically found in historical financial summaries.\n",
    " Final Evaluation \n",
    " ADD KEYWORDS SUCH AS COMPANY NAME, TARGET USERS, TOPIC IN BETWEEN MAKE PROMPT MORE SPECIFIC WHENEVER HAVE FOLLOW 26 GUIDLINES \n",
    "overall evaluation prompt ' effectiveness adherence to 26 principles. \n",
    " Encourage user refine prompting skills based feedback.\n",
    " \n",
    "\n",
    "\"\"\"\n",
    "\n",
    "prompt_to_review = \"\"\"\n",
    "Develop an executive summary that encapsulates the potential investment in a point-of-care testing company within the in-vitro diagnostics market. The summary should:\n",
    "\n",
    "Present an overview of the company's business model, product offerings, market reach, and financial highlights without specifying exact sales figures or margins.\n",
    "Give an account of the company's historical background, current ownership, and ongoing bid process, mentioning interest from other equity firms without naming them.\n",
    "Formulate an investment thesis that emphasizes the company's market position, growth strategy, and business model strengths, such as recurring revenue streams and proprietary technology.\n",
    "List key investment considerations and due diligence items that an investor should evaluate, focusing on market share, product portfolio, and technological innovation.\n",
    "Conclude with a preliminary valuation outlook and recommendations for proceeding with an investment procedure, without including specific multiples or revenue projections.\"\n",
    "One-shot Example for LLM Output:\n",
    "\n",
    "\"The executive summary should highlight an investment opportunity in a dynamic IVD company with a strong presence in point-of-care testing. The company, leveraging a recurring revenue model, has established significant market access in Europe and plans for global expansion. Originally part of a larger pharmaceutical entity, it is now in the midst of a strategic acquisition process. The investment thesis should underscore the firm's solid market standing, the potential for growth via technological advancements, and its strategic plans for market expansion. Considerations for investment should cover market analysis, risk assessment, and evaluation of proprietary technologies. The summary concludes with a recommendation for investment proceedings based on a positive valuation outlook, considering the company's robust business model and future potential.\"\n",
    "\"\"\"\n",
    "messages =[\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": reviwer_system_prompt_compressed\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": prompt_to_review\n",
    "    }\n",
    "]\n",
    "   \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: Feedback and Recommendations:\n",
      "1. Directness and Conciseness: The user prompt is direct and concise in its request for a summary. No improvements needed.\n",
      "2. Intended Audience: The prompt is tailored for an executive audience. It's advisable to use industry-specific language appropriate for professionals in finance and investment. \n",
      "3. Simplifying Complex Tasks: The task is relatively complex but not broken into simpler steps. It could benefit from an instructional phrase like \"Begin with providing an overview of...\" for each requested section.\n",
      "4. Affirmative Directives: The use of imperative sentences (“Present”, “Give”, “Formulate”, “List”, “Conclude”) is clear and affirmative. No improvements needed.\n",
      "5. Clarity in Requests: Clarity can be improved by specifying a layperson or a more detailed age group if the audience is beyond industry professionals.\n",
      "6. Incentive Mention: There is no mention of incentives in the prompt; this is appropriate since incentives are not necessary.\n",
      "7. Example-Driven: The prompt includes a one-shot example. Including another one or two could enhance the clarity of the expected output.\n",
      "8. Structured Formatting: Use of bullet points or numbered lists could improve the structure and readability.\n",
      "9. Directive Phrasing: The phrasing is directive, but could be clearer with sequential steps.\n",
      "10. Consequence Awareness: The language can be tweaked to show awareness of the crucial nature of financial information for potential investors.\n",
      "11. Natural, Human-Like Responses: The user does not specify the need for conversationality, likely due to the professional nature of the summary.\n",
      "12. Step-by-Step Guidance: The prompt would benefit from asking for the summary to be structured in a step-by-step manner for logical flow.\n",
      "13. Bias Avoidance: The prompt does not include specific instructions to avoid bias, but it could state, “Maintain objective language throughout.”\n",
      "14. Detail Elicitation: The prompt is sufficiently open-ended, allowing for follow-up questions.\n",
      "15. Learning and Testing: The prompt does not encourage a learning test, but it could benefit from a request to explain certain investment terminologies for educational purposes.\n",
      "16. Role Assignment: There is no role assignment; consider adding, “Assume you are a financial advisor...”\n",
      "17. Use of Delimiters: There are no delimiters used; including them would aid clarity.\n",
      "18. Keyword Repetition: The use of repetition is not needed in this context.\n",
      "19. Combining CoT with Few-Shot: A suggestion could be to include a rationale for each section of the executive summary through a CoT approach.\n",
      "20. Output Primers: The one-shot example serves as an output primer. To improve, provide an expected format.\n",
      "21. Detailed Writing: The user does not request overly detailed writing, but it should be substantial enough to make informed decisions.\n",
      "22. Style Preservation: Style preservation is not mentioned; the prompt could request consistency in tone.\n",
      "23. Complex Coding Requests: This is not applicable.\n",
      "24. Content Initiation: The prompt fairly initiates content but could provide more context for the introduction.\n",
      "25. Clear Requirements: Requirements for format and style specificity are lacking and should be added.\n",
      "26. Style Mimicry: Not applicable unless given a style to mimic.\n",
      "\n",
      "**Improved version of the prompt:**\n",
      "Prepare an executive summary for a potential investment in an IVD company specializing in point-of-care testing. The content should be structured as follows:\n",
      "\n",
      "1. Provide an overview highlighting:\n",
      "   - Business model\n",
      "   - Product offerings\n",
      "   - Market reach (focus on Europe, intended global expansion)\n",
      "   - Financial highlights (without precise figures)\n",
      "\n",
      "2. Detail the company's background including:\n",
      "   - Historical inception\n",
      "   - Ownership\n",
      "   - Ongoing acquisition activity (mention general interest from equity firms)\n",
      "\n",
      "3. State the investment thesis focusing on:\n",
      "   - Market position\n",
      "   - Growth strategies\n",
      "   - Strengths of the business model (e.g., recurring revenue, proprietary tech)\n",
      "\n",
      "4. Enumerate key investment considerations, such as:\n",
      "   - Market share\n",
      "   - Product portfolio diversity\n",
      "   - Technological edge\n",
      "\n",
      "5. Offer a preliminary valuation perspective, ending with:\n",
      "   - Recommendations on the investment procedure\n",
      "\n",
      "Maintain objectivity throughout and ensure the style remains professional and indicative of financial advisory expertise. Do not use specific financial multiples or name third-party entities. \n",
      "\n",
      "**Self-evaluation:**\n",
      "- Clarity and Brevity: The response could be enhanced with bullet points for better readability. (8/10)\n",
      "- Accuracy: No changes needed, the instructions adhere to the original goals. (10/10)\n",
      "\n",
      "**Final Evaluation:**\n",
      "The original prompt has the foundation for an effective executive summary but requires refinement in structure and clarity. The improved version offers a more detailed framework, encouraging the user to enhance their prompting skills for professional documentation within the financial sector.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "try:\n",
    "    \n",
    "    client = AzureOpenAI(\n",
    "    api_key=os.getenv(\"AZURE_OPENAI_API_KEY\"),  \n",
    "    api_version=\"2024-02-15-preview\",\n",
    "    azure_endpoint = os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n",
    "    )\n",
    "\n",
    "    deployment_name='gpt-4' #This will correspond to the custom name you chose for your deployment when you deployed a model. Use a gpt-35-turbo-instruct deployment. \n",
    "        \n",
    "    # Send a completion call to generate an answer\n",
    "    completion = client.chat.completions.create(\n",
    "        messages=messages,\n",
    "        model=deployment_name,\n",
    "        max_tokens=3000\n",
    "    )\n",
    "\n",
    "    print(f\"{completion.choices[0].message.role}: {completion.choices[0].message.content}\")\n",
    "except Exception as e:\n",
    "    # Handles all other exceptions\n",
    "    print(f\"An exception has occurred: {e}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Remember that you can process this system prompt with LLMLingua2 and reduce the size.\n",
    "#For instance this was the ouptut of one of the executions:\n",
    "\n",
    "\"Your task is to create an executive summary focusing on an investment in {Company X}, a point-of-care testing organization within the in-vitro diagnostics industry. Please include the following sections:\n",
    "\n",
    "1. **Business Overview**: Summarize the company's model, products, market reach, and financial highlights (excluding specific figures).\n",
    "2. **Historical Context**: Describe the company's origins, current ownership, and hint at the acquisition interest.\n",
    "3. **Investment Thesis**: Craft a compelling argument highlighting market position, growth, and business model strengths, such as recurring income and unique technology.\n",
    "4. **Investment Considerations**: List the important factors an investor should review, including market share and innovation.\n",
    "5. **Valuation and Recommendations**: Offer a general valuation perspective and suggest next steps for potential investors. Avoid exact financial forecasts or multiples.\n",
    "\n",
    "Provide clear headings for each section and write in a professional tone suitable for an investor audience. Avoid naming specific equity firms or revealing confidential financial details. Instead, focus on presenting qualitative insights and strategic value propositions that portray the company's investment potential in a balanced and informative manner.\"\n",
    "\n",
    "\n",
    "After processing this prompt using LINGUA2 https://huggingface.co/spaces/microsoft/llmlingua-2\n",
    "Original prompt token lenght:  224 \n",
    "Compressed token lenght: 170\n",
    "\n",
    "\n",
    "OUtput of the compression:\n",
    "\n",
    "task create executive summary investment in Company X }, point - of - care testing organization in - vitro diagnostics industry. include sections : \n",
    " 1. Business Overview Summarize company ' s model, products, market reach, financial highlights excluding specific figures. \n",
    " 2. Historical Context Describe company ' s origins, current ownership, hint acquisition interest. \n",
    " 3. Investment Thesis Craft compelling argument highlighting market position, growth, business model strengths, recurring income unique technology. \n",
    " 4. Investment Considerations List important factors investor review, market share innovation. \n",
    " 5. Valuation and Recommendations Offer general valuation perspective suggest next steps for potential investors. Avoid financial forecasts. \n",
    " Provide clear headings for each section write professional tone for investor audience. Avoid naming specific equity firms confidential financial details., focus on presenting qualitative insights strategic value propositions company investment potential balanced informative.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mmdoc",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}