Python: Realtime API

[markwallace-microsoft]

Milestones

• ADR reviewed and decision agreed: Python: .Net: ADR for realtime #10355
• Preview implementation for Python: Python: Introducing Realtime Clients for OpenAI and Azure OpenAI #10127
• Getting started samples
• Learn site documentation updated
• Blog post

Description

Integrate a real-time audio API within the Semantic Kernel to allow seamless interaction with OpenAI GPT, Gemini, and Anthropic models. This API will enable applications to send live audio streams for processing and receive responses in real time, facilitating enhanced conversational agents and multimedia applications. Especially targeting contact center scenarios

Scenarios

• As a developer, I can integrate the real-time audio API into my SK application to interact with through voice commands including plugin and filter use.
• As a developer, I can configure the audio input and output settings for optimal real-time performance.

Requirements:

API Development:

• Design API endpoints capable of receiving live audio streams and connecting them to realtime audio endpoints
• Enable the API to support multiple audio formats (e.g., PCM, WAV, MP3).
• Allow customization of kernel parameters (e.g., temperature, response length).

Integration Support:

• Provide seamless integration capabilities with the Semantic Kernel's existing features like plugins and filters.
• Ensure compatibility with major programming languages and frameworks (e.g., Python, C#, Java).
• Documentation and Samples:
• Provide comprehensive API documentation, including usage guidelines, parameter descriptions, and example use cases.
• Create sample projects demonstrating integration with OpenAI, Gemini, and Anthropic models.