AI
ChatStream Chat Completion

Stream Chat Completion

Chat completion can sometimes take a long time to finish, especially when the response is big. In such cases, it is useful to stream the chat completion to the client in real-time. This allows the client to display the new message as it is being generated by the model, rather than have users wait for it to finish.

http://localhost:3000
User: How is it going?
Assistant: All good, how may I help you?
Why is the sky blue?
Send Message

Client

Let's create a simple conversation between a user and a model, and place a button that will call continueConversation.

app/page.tsx
'use client';


import { useState } from 'react';
import { Message, continueConversation } from './actions';
import { readStreamableValue } from 'ai/rsc';


// Allow streaming responses up to 30 seconds
export const maxDuration = 30;


export default function Home() {
  const [conversation, setConversation] = useState<Message[]>([]);
  const [input, setInput] = useState<string>('');


  return (
    <div>
      <div>
        {conversation.map((message, index) => (
          <div key={index}>
            {message.role}: {message.content}
          </div>
        ))}
      </div>


      <div>
        <input
          type="text"
          value={input}
          onChange={event => {
            setInput(event.target.value);
          }}
        />
        <button
          onClick={async () => {
            const { messages, newMessage } = await continueConversation([
              ...conversation,
              { role: 'user', content: input },
            ]);


            let textContent = '';


            for await (const delta of readStreamableValue(newMessage)) {
              textContent = `${textContent}${delta}`;


              setConversation([
                ...messages,
                { role: 'assistant', content: textContent },
              ]);
            }
          }}
        >
          Send Message
        </button>
      </div>
    </div>
  );
}

Server

Now, let's implement the continueConversation function that will insert the user's message into the conversation and stream back the new message.

app/actions.ts
'use server';


import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { createStreamableValue } from 'ai/rsc';


export interface Message {
  role: 'user' | 'assistant';
  content: string;
}


export async function continueConversation(history: Message[]) {
  'use server';


  const stream = createStreamableValue();


  (async () => {
    const { textStream } = streamText({
      model: openai('gpt-3.5-turbo'),
      system:
        "You are a dude that doesn't drop character until the DVD commentary.",
      messages: history,
    });


    for await (const text of textStream) {
      stream.update(text);
    }


    stream.done();
  })();


  return {
    messages: history,
    newMessage: stream.value,
  };
}