Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JS] Interrupts: Streaming doesn't populate interrupt metadata #1816

Open
mbleigh opened this issue Feb 4, 2025 · 0 comments
Open

[JS] Interrupts: Streaming doesn't populate interrupt metadata #1816

mbleigh opened this issue Feb 4, 2025 · 0 comments
Labels
bug Something isn't working js

Comments

@mbleigh
Copy link
Collaborator

mbleigh commented Feb 4, 2025

Since interrupts occur during the tool loop, and the tool loop happens after generation is complete, if a developer is constructing message history only from reading streaming data they will not receive the interrupt metadata that is expected to resume. We have a few options:

  1. Relax the constraint that a resume needs to correspond to a toolRequest with interrupt metadata - any toolRequest is fine.
  2. Proactively begin tool execution the moment a tool request is generated and pause streaming until the tool response is ready.
  3. Add a mechanism for the chunk wire protocol to signal a patch to an existing chunk.
  4. Ignore the problem and document that you are expected to treat repsonse.messages as canonical and streaming is just a way to show progress.

I can see pros and cons to all of these. (3) is probably the most robust but it makes chunk handling even more complex than it already is. Maybe that's fine though? If it was something like:

{
  index: 1,
  role: "tool",
  content: [], // send an empty content arrray because we're not appending
  replaceContent: [{
    index: 2,
    replaceWith: {toolRequest: {...}, metadata: {...}},
  }]
}

Anyway, definitely not going to solve this before 1.0 so I think (4) is gonna have to be the short-term solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working js
Projects
Status: No status
Development

No branches or pull requests

1 participant