FoundationsPrompts

Prompts

Prompts are instructions that you give a large language model (LLM) to tell it what to do. It's like when you ask someone for directions; the clearer your question, the better the directions you'll get.

Many LLM providers offer complex interfaces for specifying prompts. They involve different roles and message types. While these interfaces are powerful, they can be hard to use and understand.

In order to simplify prompting across compatible providers, the Vercel AI SDK offers two categories of prompts: text prompts and message prompts.

Text Prompts

Text prompts are strings. They are ideal for simple generation use cases, e.g. repeatedly generating content for variants of the same prompt text.

You can set text prompts using the prompt property made available by AI SDK functions like generateText or streamUI. You can structure the text in any way and inject variables, e.g. using a template literal.

const result = await generateText({
  model: yourModel,
  prompt: 'Invent a new holiday and describe its traditions.',
});

You can also use template literals to provide dynamic data to your prompt.

const result = await generateText({
  model: yourModel,
  prompt:
    `I am planning a trip to ${destination} for ${lengthOfStay} days. ` +
    `Please suggest the best tourist activities for me to do.`,
});

Message Prompts

A message prompt is an array of user, assistant, and tool messages. They are great for chat interfaces and more complex, multi-modal prompts.

Each message has a role and a content property. The content can either be text (for user and assistant messages), or an array of relevant parts (data) for that message type.

const result = await streamUI({
  model: yourModel,
  messages: [
    { role: 'user', content: 'Hi!' },
    { role: 'assistant', content: 'Hello, how can I help?' },
    { role: 'user', content: 'Where can I buy the best Currywurst in Berlin?' },
  ],
});

Not all language models support all message and content types. For example, some models might not be capable of handling multi-modal inputs or tool messages. Learn more about the capabilities of select models.

Multi-modal refers to interacting with a model across different data types such as text, image, or audio data.

Instead of sending a text in the content property, you can send an array of parts that include text and other data types. Currently image and text parts are supported.

For models that support multi-modal inputs, user messages can include images. An image can be one of the following:

base64-encoded image:
- string with base-64 encoded content
- data URL string, e.g. data:image/png;base64,...
binary image:
- ArrayBuffer
- Uint8Array
- Buffer
URL:
- http(s) URL string, e.g. https://example.com/image.png
- URL object, e.g. new URL('https://example.com/image.png')

It is possible to mix text and multiple images.

Not all models support all types of multi-modal inputs. Check the model's capabilities before using this feature.

Example: Binary image (Buffer)

const result = await generateText({
  model,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Describe the image in detail.' },
        {
          type: 'image',
          image: fs.readFileSync('./data/comic-cat.png'),
        },
      ],
    },
  ],
});

Example: Base-64 encoded image (string)

const result = await generateText({
  model: yourModel,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Describe the image in detail.' },
        {
          type: 'image',
          image: fs.readFileSync('./data/comic-cat.png').toString('base64'),
        },
      ],
    },
  ],
});

Example: Image URL (string)

const result = await generateText({
  model: yourModel,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Describe the image in detail.' },
        {
          type: 'image',
          image:
            'https://github.com/vercel/ai/blob/main/examples/ai-core/data/comic-cat.png?raw=true',
        },
      ],
    },
  ],
});

Tool messages

Tools (also known as function calling) are programs that you can provide an LLM to extend it's built-in functionality. This can be anything from calling an external API to calling functions within your UI. Learn more about Tools in the next section.

For models that support tool calls, assistant messages can contain tool call parts, and tool messages can contain tool result parts. A single assistant message can call multiple tools, and a single tool message can contain multiple tool results.

const result = await generateText({
  model: yourModel,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'How many calories are in this block of cheese?',
        },
        { type: 'image', image: fs.readFileSync('./data/roquefort.jpg') },
      ],
    },
    {
      role: 'assistant',
      content: [
        {
          type: 'tool-call',
          toolCallId: '12345',
          toolName: 'get-nutrition-data',
          args: { cheese: 'Roquefort' },
        },
        // there could be more tool calls here (parallel calling)
      ],
    },
    {
      role: 'tool',
      content: [
        {
          type: 'tool-result',
          toolCallId: '12345', // needs to match the tool call id
          toolName: 'get-nutrition-data',
          result: {
            name: 'Cheese, roquefort',
            calories: 369,
            fat: 31,
            protein: 22,
          },
        },
        // there could be more tool results here (parallel calling)
      ],
    },
  ],
});

System Messages

System messages are the initial set of instructions given to models that help guide and constrain the models' behaviors and responses. You can set system prompts using the system property. System messages work with both the prompt and the messages properties.

const result = await generateText({
  model: yourModel,
  system:
    `You help planning travel itineraries. ` +
    `Respond to the users' request with a list ` +
    `of the best stops to make in their destination.`,
  prompt:
    `I am planning a trip to ${destination} for ${lengthOfStay} days. ` +
    `Please suggest the best tourist activities for me to do.`,
});