examples/evaluate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Skythought Evals: Evaluation for LLM Reasoning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation and Setup\n",
    "\n",
    "You can install the latest release from PyPI, or install from source:\n",
    "\n",
    "#### Installing from PyPI\n",
    "\n",
    "```shell\n",
    "pip install skythought\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Installing from source\n",
    "\n",
    "For installing from source, we recommend using uv for package management (For uv installation, refer to the [official guide](https://docs.astral.sh/uv/getting-started/installation)).\n",
    "\n",
    "```shell\n",
    "# Clone the repository\n",
    "git clone https://github.com/NovaSky-AI/SkyThought.git\n",
    "cd SkyThought\n",
    "\n",
    "# Create and activate a virtual environment (using uv here)\n",
    "uv venv --python 3.10\n",
    "source .venv/bin/activate\n",
    "\n",
    "# Install the package in editable mode\n",
    "uv pip install -e .\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're evaluating OpenAI models, make sure to setup the appropriate env vars:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment if needed\n",
    "# export OPENAI_API_KEY=your_openai_api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the CLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!skythought --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the following:\n",
    "\n",
    "<p align=\"center\"><img src=\"../assets/cli.png\" width=\"50%\"></p>\n",
    "\n",
    "We provide the following commands:\n",
    "- `skythought evaluate` : Evaluate a model on a given task. This is the main entrypoint for those interested in evaluation\n",
    "- `skythought generate`: Generate model outputs for a pre-configured task. This is useful in data curation i.e in cases where you might post-process the generations before scoring. Our evaluation library supports training datasets such as NUMINA, APPS and TACO. \n",
    "- `skythought score`: Score saved generations for a given task. This is again useful in the case of data curation where standalone scoring might be preferred. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `evaluate` \n",
    "\n",
    "Given below are some example commands: \n",
    "\n",
    "1. Quick Start\n",
    "\n",
    "```bash\n",
    "skythought evaluate \\\n",
    "--task aime24 \\ \n",
    "--model  NovaSky-AI/Sky-T1-32B-Preview \\\n",
    "--backend vllm \\\n",
    "--batch-size 128\n",
    "```\n",
    "\n",
    "2. Customized\n",
    "\n",
    "```bash\n",
    "skythought evaluate \\\n",
    "    --task aime24 \\\n",
    "    --model  NovaSky-AI/Sky-T1-32B-Flash \\\n",
    "    --backend vllm \\\n",
    "    --backend-args tensor_parallel_size=8,revision=0dccf55,dtype=float32 \\\n",
    "    --sampling-params max_tokens=4096,temperature=0.1 \\\n",
    "    # use a pre-configured system prompt\n",
    "    --system-prompt-name prime_rl \\\n",
    "    --result-dir ./ \\\n",
    "    --batch-size 128\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Concepts\n",
    "\n",
    "-  Task: A task is an evaluation dataset. We use the `task` argument to retrieve the corresponding configuration file from our pre-configured benchmarks (To see the available tasks, use `skythought evaluate --help`) \n",
    "-  Model: A Model consists of the model ID and templating configuration. This configuration optionally contains the system prompt and an assistant prefill message. We use the `model` argument to retrieve pre-configured templating parameters (system prompt, assistant prefill, etc) for the model, if available. You can find the list of pre-configured models [here](https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/evals/models/model_configs.yaml). If a model is not available, then we use no system prompt (i.e we default to the system prompt in the chat template, if specified). You can use the `--system-prompt-name` flag to use one of the pre-configured system prompts in the library. To see the available system prompts, use `skythought evaluate --help`. You can also pass the full system prompt via CLI with the `--system-prompt` option. \n",
    "- Backend: The Backend is concerned with how the LLM instance is created and queried. We support a variety of backends via the `backend` argument. \n",
    "    - The `openai` backend can be used to query OpenAI-compatible endpoints. Example: `--backend openai --backend-args base_url=https://api.openai.com`\n",
    "    - The `vllm` backend instantiates a local model instance with [vLLM](docs.vllm.ai) for efficient inference. \n",
    "    - The `ray` backend leverages [Ray Data](https://docs.ray.io/en/latest/data/data.html) on top of vLLM for scaling inference to multiple replicas on single node or a multi-node Ray cluster. This is the recommended backend for high throughput. \n",
    "The Backend also consists of configuration at instantiation (`--backend-args`) and during generation (`--sampling-params` to control temperature, max_tokens, etc, as well as `--n` for number of generations per problem). \n",
    "\n",
    "\n",
    "During evaluation, the flow is straightforward: \n",
    "1. Load dataset and create conversations based on the Task and Model specified by the user\n",
    "2. Generate model responses from the Backend based on the provided sampling parameters\n",
    "3. Score model responses based on the Task \n",
    "4. Output final results\n",
    "\n",
    "<p align=\"center\"><img src=\"../assets/flow.png\" width=\"65%\"></p>\n",
    "\n",
    "Once finished, the results should be saved in a folder in `result-dir` :\n",
    "\n",
    "```bash\n",
    "result-dir/\n",
    "├── NovaSky-AI_Sky-T1-32B-Flash_aime24_myHash\n",
    "│   ├── results.json\n",
    "│   └── summary.json\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more details - such as which configurations are best for performance, how to perform multi-node inference, etc refer to the [README](../skythought/evals/README.md)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.21"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}