ChatStream Chat Completion

Stream Chat Completion

Chat completion can sometimes take a long time to finish, especially when the response is big. In such cases, it is useful to stream the chat completion to the client in real-time. This allows the client to display the new message as it is being generated by the model, rather than have users wait for it to finish.

http://localhost:3000
User: How is it going?
Assistant: All good, how may I help you?
Why is the sky blue?
Send Message

Client

Let's create a React component that imports the useChat hook from the ai/react module. The useChat hook will call the /api/chat endpoint when the user sends a message. The endpoint will generate the assistant's response based on the conversation history and stream it to the client.

pages/index.tsx
import { useChat } from 'ai/react';
export default function Page() {
const { messages, input, setInput, append } = useChat();
return (
<div>
<input
value={input}
onChange={event => {
setInput(event.target.value);
}}
onKeyDown={async event => {
if (event.key === 'Enter') {
append({ content: input, role: 'user' });
}
}}
/>
{messages.map((message, index) => (
<div key={index}>{message.content}</div>
))}
</div>
);
}

Server

Next, let's create the /api/chat endpoint that generates the assistant's response based on the conversation history.

app/api/chat/route.ts
import { CoreMessage, streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
export async function POST(req: Request) {
const { messages }: { messages: CoreMessage[] } = await req.json();
const result = await streamText({
model: openai('gpt-4'),
system: 'You are a helpful assistant.',
messages,
});
return result.toAIStreamResponse();
}

View Example on GitHub