Looking for the JS/TS library? Check out AgentsJS

✨ 1.0 release ✨

This README reflects the upcoming 1.0 release. For documentation on the current 0.x release, see the 0.x branch

What is Agents?

The Agents framework enables you to build voice AI agents that can see, hear, and speak in realtime. It provides a fully open-source platform for creating server-side agentic applications.

Features

• Flexible integrations: A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.
• Integrated job scheduling: Built-in task scheduling and distribution with dispatch APIs to connect end users to agents.
• Extensive WebRTC clients: Build client applications using LiveKit's open-source SDK ecosystem, supporting nearly all major platforms.
• Telephony integration: Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.
• Exchange data with clients: Use RPCs and other Data APIs to seamlessly exchange data with clients.
• Open-source: Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers.

Installation

To install the core Agents library, along with plugins for popular model providers:

pip install "livekit-agents[openai,silero,deepgram,cartesia,turn-detector]~=1.0rc"

Docs and guides

Documentation on the framework and how to use it can be found here

Core concepts

• Agent: An LLM-based application with defined instructions.
• AgentSession: A container for agents that manages interactions with end users.
• entrypoint: The starting point for an interactive session, similar to a request handler in a web server.

Usage examples

Simple voice agent

---

from livekit.agents import (
    Agent,
    AgentSession,
    JobContext,
    RunContext,
    WorkerOptions,
    cli,
    function_tool,
)
from livekit.plugins import deepgram, openai, silero

@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""

    return {"weather": "sunny", "temperature": 70}


async def entrypoint(ctx: JobContext):
    await ctx.connect()

    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
        tools=[lookup_weather],
    )
    session = AgentSession(
        vad=silero.VAD.load(),
        # any combination of STT, LLM, TTS, or realtime API can be used
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(voice="ash"),
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

You'll need the following environment variables for this example:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET
• DEEPGRAM_API_KEY
• OPENAI_API_KEY

Multi-agent handoff

---

This code snippet is abbreviated. For the full example, see multi_agent.py

...
class IntroAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=f"You are a story teller. Your goal is to gather a few pieces of information from the user to make the story personalized and engaging."
            "Ask the user for their name and where they are from"
        )

    async def on_enter(self):
        self.session.generate_reply(instructions="greet the user and gather information")

    @function_tool
    async def information_gathered(
        self,
        context: RunContext,
        name: str,
        location: str,
    ):
        """Called when the user has provided the information needed to make the story personalized and engaging.

        Args:
            name: The name of the user
            location: The location of the user
        """

        context.userdata.name = name
        context.userdata.location = location

        story_agent = StoryAgent(name, location)
        return story_agent, "Let's start the story!"


class StoryAgent(Agent):
    def __init__(self, name: str, location: str) -> None:
        super().__init__(
            instructions=f"You are a storyteller. Use the user's information in order to make the story personalized."
            f"The user's name is {name}, from {location}"
            # override the default model, switching to Realtime API from standard LLMs
            llm=openai.realtime.RealtimeModel(voice="echo"),
            chat_ctx=chat_ctx,
        )

    async def on_enter(self):
        self.session.generate_reply()


async def entrypoint(ctx: JobContext):
    await ctx.connect()

    userdata = StoryData()
    session = AgentSession[StoryData](
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(voice="echo"),
        userdata=userdata,
    )

    await session.start(
        agent=IntroAgent(),
        room=ctx.room,
    )
...

Additional examples

---

We've built additional examples including:

• multi-user agent with push to talk
• using parallel function calls
• restaurant ordering and reservations
• avatar integration
• simple video publisher

Running your agent

python myagent.py dev

This starts the agent server, and your agent will wait for user connections. Each process can efficiently host multiple concurrent agents.

You can connect to your agent using any app built with LiveKit's client SDKs or telephony integration. Alternatively, use the Agents Playground to test your agents.

Contributing

The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in LiveKit's Slack community.

LiveKit Ecosystem
LiveKit SDKs	Browser · iOS/macOS/visionOS · Android · Flutter · React Native · Rust · Node.js · Python · Unity · Unity (WebGL)
Server APIs	Node.js · Golang · Ruby · Java/Kotlin · Python · Rust · PHP (community) · .NET (community)
UI Components	React · Android Compose · SwiftUI
Agents Frameworks	Python · Node.js · Playground
Services	LiveKit server · Egress · Ingress · SIP
Resources	Docs · Example apps · Cloud · Self-hosting · CLI