Stream Text with Chat Prompt
Chat completion can sometimes take a long time to finish, especially when the response is big. In such cases, it is useful to stream the chat completion to the client in real-time. This allows the client to display the new message as it is being generated by the model, rather than have users wait for it to finish.
http://localhost:3000
User: How is it going?
Assistant: All good, how may I help you?
Why is the sky blue?
Send Message
Client
Let's create a simple conversation between a user and a model, and place a button that will call continueConversation
.
app/page.tsx
'use client';
import { useState } from 'react';import { Message, continueConversation } from './actions';import { readStreamableValue } from 'ai/rsc';
// Allow streaming responses up to 30 secondsexport const maxDuration = 30;
export default function Home() { const [conversation, setConversation] = useState<Message[]>([]); const [input, setInput] = useState<string>('');
return ( <div> <div> {conversation.map((message, index) => ( <div key={index}> {message.role}: {message.content} </div> ))} </div>
<div> <input type="text" value={input} onChange={event => { setInput(event.target.value); }} /> <button onClick={async () => { const { messages, newMessage } = await continueConversation([ ...conversation, { role: 'user', content: input }, ]);
let textContent = '';
for await (const delta of readStreamableValue(newMessage)) { textContent = `${textContent}${delta}`;
setConversation([ ...messages, { role: 'assistant', content: textContent }, ]); } }} > Send Message </button> </div> </div> );}
Server
Now, let's implement the continueConversation
function that will insert the user's message into the conversation and stream back the new message.
app/actions.ts
'use server';
import { streamText } from 'ai';import { openai } from '@ai-sdk/openai';import { createStreamableValue } from 'ai/rsc';
export interface Message { role: 'user' | 'assistant'; content: string;}
export async function continueConversation(history: Message[]) { 'use server';
const stream = createStreamableValue();
(async () => { const { textStream } = streamText({ model: openai('gpt-3.5-turbo'), system: "You are a dude that doesn't drop character until the DVD commentary.", messages: history, });
for await (const text of textStream) { stream.update(text); }
stream.done(); })();
return { messages: history, newMessage: stream.value, };}