Prompts
Prompts are instructions that you give a large language model (LLM) to tell it what to do. It's like when you ask someone for directions; the clearer your question, the better the directions you'll get.
Many LLM providers offer complex interfaces for specifying prompts. They involve different roles and message types. While these interfaces are powerful, they can be hard to use and understand.
In order to simplify prompting across compatible providers, the AI SDK offers two categories of prompts: text prompts and message prompts.
Text Prompts
Text prompts are strings. They are ideal for simple generation use cases, e.g. repeatedly generating content for variants of the same prompt text.
You can set text prompts using the prompt
property made available by AI SDK functions like generateText
or streamUI
.
You can structure the text in any way and inject variables, e.g. using a template literal.
const result = await generateText({ model: yourModel, prompt: 'Invent a new holiday and describe its traditions.',});
You can also use template literals to provide dynamic data to your prompt.
const result = await generateText({ model: yourModel, prompt: `I am planning a trip to ${destination} for ${lengthOfStay} days. ` + `Please suggest the best tourist activities for me to do.`,});
Message Prompts
A message prompt is an array of user, assistant, and tool messages. They are great for chat interfaces and more complex, multi-modal prompts.
Each message has a role
and a content
property. The content can either be text (for user and assistant messages), or an array of relevant parts (data) for that message type.
const result = await streamUI({ model: yourModel, messages: [ { role: 'user', content: 'Hi!' }, { role: 'assistant', content: 'Hello, how can I help?' }, { role: 'user', content: 'Where can I buy the best Currywurst in Berlin?' }, ],});
Not all language models support all message and content types. For example, some models might not be capable of handling multi-modal inputs or tool messages. Learn more about the capabilities of select models.
Multi-modal messages
Multi-modal refers to interacting with a model across different data types such as text, image, or audio data.
Instead of sending a text in the content
property, you can send an array of parts that include text and other data types.
Currently image and text parts are supported.
For models that support multi-modal inputs, user messages can include images. An image
can be one of the following:
- base64-encoded image:
string
with base-64 encoded content- data URL
string
, e.g.data:image/png;base64,...
- binary image:
ArrayBuffer
Uint8Array
Buffer
- URL:
- http(s) URL
string
, e.g.https://example.com/image.png
URL
object, e.g.new URL('https://example.com/image.png')
- http(s) URL
It is possible to mix text and multiple images.
Not all models support all types of multi-modal inputs. Check the model's capabilities before using this feature.
Example: Binary image (Buffer)
const result = await generateText({ model, messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe the image in detail.' }, { type: 'image', image: fs.readFileSync('./data/comic-cat.png'), }, ], }, ],});
Example: Base-64 encoded image (string)
const result = await generateText({ model: yourModel, messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe the image in detail.' }, { type: 'image', image: fs.readFileSync('./data/comic-cat.png').toString('base64'), }, ], }, ],});
Example: Image URL (string)
const result = await generateText({ model: yourModel, messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe the image in detail.' }, { type: 'image', image: 'https://github.com/vercel/ai/blob/main/examples/ai-core/data/comic-cat.png?raw=true', }, ], }, ],});
Tool messages
Tools (also known as function calling) are programs that you can provide an LLM to extend it's built-in functionality. This can be anything from calling an external API to calling functions within your UI. Learn more about Tools in the next section.
For models that support tool calls, assistant messages can contain tool call parts, and tool messages can contain tool result parts. A single assistant message can call multiple tools, and a single tool message can contain multiple tool results.
const result = await generateText({ model: yourModel, messages: [ { role: 'user', content: [ { type: 'text', text: 'How many calories are in this block of cheese?', }, { type: 'image', image: fs.readFileSync('./data/roquefort.jpg') }, ], }, { role: 'assistant', content: [ { type: 'tool-call', toolCallId: '12345', toolName: 'get-nutrition-data', args: { cheese: 'Roquefort' }, }, // there could be more tool calls here (parallel calling) ], }, { role: 'tool', content: [ { type: 'tool-result', toolCallId: '12345', // needs to match the tool call id toolName: 'get-nutrition-data', result: { name: 'Cheese, roquefort', calories: 369, fat: 31, protein: 22, }, }, // there could be more tool results here (parallel calling) ], }, ],});
System Messages
System messages are the initial set of instructions given to models that help guide and constrain the models' behaviors and responses.
You can set system prompts using the system
property.
System messages work with both the prompt
and the messages
properties.
const result = await generateText({ model: yourModel, system: `You help planning travel itineraries. ` + `Respond to the users' request with a list ` + `of the best stops to make in their destination.`, prompt: `I am planning a trip to ${destination} for ${lengthOfStay} days. ` + `Please suggest the best tourist activities for me to do.`,});