How do I implement tool call streaming?

how do i do tool call streaming

Now I have all the information needed. Here's a comprehensive answer:

##Tool Call Streaming in AI SDK

Tool call streaming is enabled by default in AI SDK 5.0. This feature streams tool inputs as they're being generated, providing a better user experience by showing what the model is "thinking" in real-time.

###How It Works

When streaming is enabled, tool parts in messages go through several states:

###Server-Side Setup

// app/api/chat/route.ts
import { streamText, tool, convertToModelMessages } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';

const tools = {
  getWeatherInformation: tool({
    description: 'Get weather for a city',
    inputSchema: z.object({ city: z.string() }),
    // Optional: callbacks for streaming events
    onInputStart: () => {
      console.log('Tool input streaming started');
    },
    onInputDelta: ({ inputTextDelta }) => {
      console.log('Input delta:', inputTextDelta);
    },
    onInputAvailable: ({ input }) => {
      console.log('Full input available:', input);
    },
    async execute({ city }) {
      return { weather: 'sunny', temperature: 72 };
    },
  }),
};

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o'),
    messages: await convertToModelMessages(messages),
    tools,
    // Tool call streaming is enabled by default in v5
  });

  return result.toUIMessageStreamResponse();
}

###Client-Side Rendering

// app/page.tsx
import { useChat } from '@ai-sdk/react';

export default function Chat() {
  const { messages, sendMessage, status } = useChat();

  return (
    <div>
      {messages?.map(message => (
        <div key={message.id}>
          {message.parts.map((part, index) => {
            switch (part.type) {
              case 'text':
                return <span key={index}>{part.text}</span>;

              case 'tool-getWeatherInformation':
                switch (part.state) {
                  // Show streaming input as it's generated
                  case 'input-streaming':
                    return (
                      <pre key={index}>
                        Streaming: {JSON.stringify(part.input, null, 2)}
                      </pre>
                    );
                  
                  // Input is complete, tool is executing
                  case 'input-available':
                    return (
                      <div key={index}>
                        Getting weather for {part.input.city}...
                      </div>
                    );
                  
                  // Tool finished successfully
                  case 'output-available':
                    return (
                      <div key={index}>
                        Weather in {part.input.city}: {part.output.weather}
                      </div>
                    );
                  
                  // Tool execution failed
                  case 'output-error':
                    return (
                      <div key={index} className="text-red-500">
                        Error: {part.errorText}
                      </div>
                    );
                }
            }
          })}
        </div>
      ))}
    </div>
  );
}

###Stream Protocol Events

Under the hood, the data stream sends these events for tool calls:

1. tool-input-start - Indicates streaming has begun
2. tool-input-delta - Contains incremental chunks of input
3. tool-input-available - Full input is ready
4. tool-output-available - Result after execution

###Key Points

1. Enabled by default - No configuration needed in AI SDK 5.0
2. Provider support varies - Check provider docs (Anthropic, OpenAI support it well)
3. Typed tool parts - Use part.type like 'tool-yourToolName' to match specific tools
4. Graceful degradation - If streaming isn't supported, you'll just see input-available directly

See the full working example in examples/next-openai/app/use-chat-tools for a complete implementation.

isn't yield a thing?

Now I have everything. Here's the complete answer about yield in tool execution:

Yes, absolutely! yield is the key feature for streaming tool outputs (as opposed to tool inputs). You can use an async generator function (async *execute) to stream intermediate/preliminary results during tool execution.

##Streaming Tool Outputs with yield

Use async *execute instead of async execute to yield multiple values:

import { tool } from 'ai';
import { z } from 'zod';

const weatherTool = tool({
  description: 'Get the weather in a location',
  inputSchema: z.object({ city: z.string() }),
  
  // Use async generator function with *
  async *execute({ city }) {
    // First yield: preliminary result (e.g., loading state)
    yield { state: 'loading' as const, message: `Fetching weather for ${city}...` };

    // Simulate API call
    await new Promise(resolve => setTimeout(resolve, 3000));

    // Final yield: the actual result (last value is the final result)
    yield {
      state: 'ready' as const,
      temperature: 72,
      weather: 'sunny',
    };
  },
});

###How It Works

1. All yields except the last are treated as preliminary results (preliminary: true)
2. The last yield is the final result sent to the model
3. Each preliminary result is streamed to the client in real-time

###Client-Side Rendering

On the client, you check part.output to see the yielded values, and optionally part.preliminary to know if it's an intermediate result:

case 'tool-getWeather':
  switch (part.state) {
    case 'output-available':
      // Check the output's state (your custom field)
      if (part.output.state === 'loading') {
        return <div>Loading: {part.output.message}</div>;
      }
      return <div>Weather: {part.output.weather}, {part.output.temperature}°F</div>;
  }

Or using the preliminary flag:

case 'output-available':
  const isPreliminary = part.preliminary ?? false;
  return (
    <div>
      {isPreliminary ? 'Loading...' : `Result: ${JSON.stringify(part.output)}`}
    </div>
  );

###Full Example

From examples/next-openai/tool/weather-tool.ts:

export const weatherTool = tool({
  description: 'Get the weather in a location',
  inputSchema: z.object({ city: z.string() }),
  async *execute() {
    yield { state: 'loading' as const };

    // Add randomized delay
    await new Promise(resolve =>
      setTimeout(resolve, 1000 + Math.floor(Math.random() * 4000)),
    );

    yield {
      state: 'ready' as const,
      temperature: 72,
      weather: randomWeather(),
    };
  },
});

###Summary: Two Types of Tool Streaming