About FastAPI and Streaming #2047

Akimio521 · 2024-06-08T10:06:06Z

Akimio521
Jun 8, 2024

I have built an API interface using FastAPI and g4f for other devices to call. The current problem is with streaming output. I don't know how to modify my Python program to meet the official API's requirements for streaming output (the front-end can get data from my FastAPI interface normally after streaming output is closed).

@app.post("/v1/chat/completions")
async def openai_api(request: Request):
    json = await request.json()
    response = client_openai.chat.completions.create(
        model=json.get("model"),
        messages=json.get("messages"),
        stream=json.get("stream"),
    )
    if json.get("stream"):  # 流式输出
        for chunk in response:
            if chunk.choices[0].delta.content is not None:
                print(chunk.choices[0].delta.content, end="")
        return response # It is not a right way to return a streaming response
    else:
        print(response)
        return response # It is a right way to return response

Looking forward to your reply

gamelist1990 · 2024-06-22T06:05:13Z

gamelist1990
Jun 22, 2024

If it's okay in my way, I can suggest it.

6 replies

gamelist1990 Jun 22, 2024

@Akimio521
See my sample code.

Server side code:

import os
import re
import uuid
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, StreamingResponse
import time

from fastapi.staticfiles import StaticFiles

import g4f
from g4f.client.async_client import AsyncClient
from g4f.cookies import read_cookie_files

app = FastAPI()
app.mount("/home", StaticFiles(directory="home"), name="home")
cookies_dir = os.path.join(os.path.dirname(__file__), "har")




async def chat_with_OpenAI(prompt: str, streaming: bool = False):
   

  

    try:
        client = AsyncClient(
            provider=g4f.Provider.OpenaiChat,
            api_key=read_cookie_files(cookies_dir),  # 正しい関数名に修正
        )
        if streaming:
            buffer = ""
            async for chunk in client.chat.completions.create(
                model="auto",
                messages=[{"role": "user", "content":  prompt}],
                stream=True,
            ):
                if chunk.choices[0].delta.content:
                    buffer += chunk.choices[0].delta.content
                    if len(buffer) > 1: # YIELD when buffer exceeds 100 characters
                        buffer = ""
            if buffer: # If there are any strings remaining in the buffer, yield
                yield buffer
        else:
            # 非同期処理
            response = await client.chat.completions.create(
                model="auto",
                messages=conversation_history,
                Stream=True,
            )
            if response.choices[0].delta.content:
                    yield response.choices[0].delta.content  # yield をループ内で実行
    except Exception as e:
        logging.error(f"Error occurred: {str(e)}")
        yield "An error has occurred with your OpenAI provider. If this happens repeatedly, please use another provider"  # エラーメッセージをyieldします


async def process_chat_stream(provider: str, user_id: str, prompt: str, system: str = AI_prompt):
    streaming = True
    if provider == 'OpenAI':
        async for response in chat_with_OpenAI(user_id, prompt, system, streaming):
            yield f"data: {response}\n\n"  # SSE フォーマットで送信
    else:
        yield JSONResponse(content={"error": "Invalid provider specified"}, status_code=400)


@app.post("/stream")
async def stream(request: Request):
    data = await request.json()
    provider = data.get('provider', "OpenAI")
    prompt = data.get('prompt')


    if not prompt:
        return JSONResponse(content={"response": "No question asked"}, status_code=200)

    # 文字数制限を設ける
    if len(prompt) > 300:
        return JSONResponse(content={"response": "Please keep it within 300 characters"}, status_code=400)

    return StreamingResponse(process_chat_stream(provider, user_id, prompt, system),
                             media_type="text/event-stream")  # media_type を設定

Client Side Code：

<!DOCTYPE html>
<html>
<head>
    <title>Chat with AI</title>
</head>
<body>
    <h1>Chat with AI</h1>
    <form id="chat-form">
        <label for="prompt">Prompt:</label><br>
        <input type="text" id="prompt" name="prompt"><br>
        <input type="submit" value="Submit">
    </form>
    <div id="response"></div>

    <script>
    document.getElementById('chat-form').addEventListener('submit', function(e) {
        e.preventDefault();
        const prompt = document.getElementById('prompt').value;
        const responseDiv = document.getElementById('response');
        responseDiv.innerHTML = '';

        fetch('/stream', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({
                'prompt': prompt,
                'provider': 'OpenAI',
                'system': 'None'
            })
        }).then(response => {
            const reader = response.body.getReader();
            const decoder = new TextDecoder('utf-8');

            return reader.read().then(function processText({ done, value }) {
                if (done) {
                    return;
                }
                let result = decoder.decode(value);
                result = result.replace(/^data:\s*/gm, '').trim();  // 'data:' とその後の空白を全て削除
                responseDiv.innerHTML += result;
                return reader.read().then(processText);
            });
        }).catch(error => {
            console.error(error);
        });
    });
    </script>
</body>
</html>

Streaming with this code is working fine in my environment!

Akimio521 Jun 22, 2024
Author

Thanks for your help.I would try it in my environment.

gamelist1990 Jun 22, 2024

This code has a problem that it is difficult to convert to html properly on the client side when AI outputs markdown format code, so we are looking for a solution.

gamelist1990 Jun 22, 2024

@Akimio521 Here's my webapi's experimental streaming process, and if it works, it will look something like this

https://streamable.com/34i7dt

I can't upload the video for some reason, so you'll have to watch it from the url.

gamelist1990 Jun 23, 2024

@Akimio521 Trim(); is not recommended, as it will cause problems with missing whitespace and AI layouts.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

About FastAPI and Streaming #2047

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

About FastAPI and Streaming #2047

Akimio521 Jun 8, 2024

Replies: 1 comment · 6 replies

gamelist1990 Jun 22, 2024

gamelist1990 Jun 22, 2024

Akimio521 Jun 22, 2024 Author

gamelist1990 Jun 22, 2024

gamelist1990 Jun 22, 2024

gamelist1990 Jun 23, 2024

Akimio521
Jun 8, 2024

Replies: 1 comment 6 replies

gamelist1990
Jun 22, 2024

Akimio521 Jun 22, 2024
Author