ChromeAI

jeasonstudio/chrome-ai is a community provider that uses Chrome Built-in AI to provide language model support for the Vercel AI SDK.

Setup

The ChromeAI provider is available in the chrome-ai module. You can install it with:

pnpm install chrome-ai

Enabling AI in Chrome

Chrome built-in AI is a preview feature, you need to use chrome version 127 or greater, now in dev or canary channel, may release on stable chanel at Jul 17, 2024.

After then, you should turn on these flags:

• chrome://flags/#prompt-api-for-gemini-nano: Enabled
• chrome://flags/#optimization-guide-on-device-model: Enabled BypassPrefRequirement
• chrome://components/: Click Optimization Guide On Device Model to download the model.

Language Models

The chromeai provider instance is a function that you can invoke to create a language model:

import { chromeai } from 'chrome-ai';

const model = chromeai();

It automatically selects the correct model id. You can also pass additional settings in the second argument:

import { chromeai } from 'chrome-ai';

const model = chromeai('generic', {
  // additional settings
  temperature: 0.5,
  topK: 5,
});

You can use the following optional settings to customize:

• modelId 'text' | 'generic'

  Used to distinguish models of Gemini Nano, there is no difference in the current version.

• temperature number

  The value is passed through to the provider. The range depends on the provider and model. For most providers, 0 means almost deterministic results, and higher values mean more randomness.

• topK number

  Only sample from the top K options for each subsequent token.

  Used to remove "long tail" low probability responses. Recommended for advanced use cases only. You usually only need to use temperature.

Examples

You can use Chrome built-in language models to generate text with the generateText or streamText function:

import { generateText } from 'ai';
import { chromeai } from 'chrome-ai';

const { text } = await generateText({
  model: chromeai(),
  prompt: 'Who are you?',
});

console.log(text); //  I am a large language model, trained by Google.

import { streamText } from 'ai';
import { chromeai } from 'chrome-ai';

const { textStream } = await streamText({
  model: chromeai(),
  prompt: 'Who are you?',
});

let result = '';
for await (const textPart of textStream) {
  result = textPart;
}

console.log(result);
//  I am a large language model, trained by Google.

Chrome built-in language models can also be used in the generateObject function:

import { generateObject } from 'ai';
import { chromeai } from 'chrome-ai';
import { z } from 'zod';

const { object } = await generateObject({
  model: chromeai('text'),
  schema: z.object({
    recipe: z.object({
      name: z.string(),
      ingredients: z.array(
        z.object({
          name: z.string(),
          amount: z.string(),
        }),
      ),
      steps: z.array(z.string()),
    }),
  }),
  prompt: 'Generate a lasagna recipe.',
});

console.log(object);
// { recipe: {...} }