Performing controlled actions while streaming

[cacoos]

I read the documentation, but I can't find anything related to executing actions while streaming, at least in React (NextJS).

For example, let's say I want to call my /api/completion endpoint from a component to update a specific state in a controlled way. What is the best approach for this?

It seems the best option is to call fetch to the API and handle SSE manually without the SDK, so I can perform actions per chunk received.

I noticed that onChunk is a property for streamText on the server but not available on the client. Why is that?

There are some workarounds, like the one in this pull request: #3252, but it's like something is off.

Any help would be much appreciated!

[brennanmceachran]

Are you using useChat? Assuming so because of the mentioned PR.

One way people have been doing this is with writeData on the dataStream. See here for docs. The vercel ai-chatbot example is also riddled with little tips along this path. You could add an onChunk listener on streamText and annotate the message or send.

Can you share more about your use case? I might have other approaches depending on what actions you're trying to perform per chunk

[cacoos]

Hey Brennan! Thanks for your answer.

This is more about UI updates than server updates.

Here’s the use case:

Let’s say you have a canvas app where an AI can draw on it. You call useCompletion (or useObject) to stream a sequence of actions as a textStream, like this:

• (User): Hey! Can you draw a horse?
• (AI): Sure! I'll start with the body: <draw>{x: 0, y: 0, width: 10, height: 10}</draw><paint>{...}</paint>...

I want to parse that stream and execute actions on the canvas. For example, <draw>{x: 0, y: 0, width: 10, height: 10}</draw> would draw something on the canvas, <paint>{...}</paint> would paint, and so on.

One option is to just use useEffect with messages, completion, or data as dependencies, but that doesn’t feel right. I’d prefer to act specifically when each token is received.

Is my example clear enough? Let me know :)

Thank you!

[brennanmceachran]

How married are you to the inline xml style of canvas actions? I ask because I know drawing is hard for llms. So if you've figured out something that works I don't want to mess with it -- in this case I'd go with the useEffect.

Otherwise I'd consider thinking about draw, paint, etc as toolcalls. You'll be able to coerce the {...} into a schema that makes sense for that tool and respond with tool invocations w/ results (draw success vs. error).

You'd offload the complexity for parsing, streaming, etc to the llm & ai-sdk at the expense of small complexity of migrating to useChat to handle the messages. But then you'd also get things like onToolCall on the client allowing you to immediately respond to draw and paint within the canvas while streaming.

[cacoos]

Thank you Brennan for the help!

That unicorn-drawing project seems fun :) My case isn’t exactly painting, but it’s something related.

Yes, I’m aware of the tools, but I’m opting for full-text responses because of the flexibility they offer.

I’m curious about the use case for v0.dev. They open a document when the AI streams code, and once the code is previewable, the view switches to the "Preview" tab. How do you handle that purely based on state? useEffect?

[brennanmceachran]

They've open sourced a template for that (it uses tool calls, which then use further ai calls)
https://github.com/vercel/ai-chatbot

My link at the very top of the thread shows the function calls to "createDocument" which might give you a few ideas.

[cacoos]

Oh, thanks! Seems like tools is the best way then.

Appreciate the help :)