> ## Documentation Index
> Fetch the complete documentation index at: https://vlyai-1c28d863.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# AI chatbot

> Build a streaming AI chatbot — conversation history, tool calling, multi-model selection — using OpenAI or Anthropic. The recipe everyone asks for.

export const PromptBlock = ({children, variant}) => <div className={`vly-prompt ${variant || ""}`}>{children}</div>;

export const Prerequisites = ({items}) => <Note>
    <strong>Before you start.</strong>
    <ul>
      {(items || ["A vly.ai account (free tier is enough)", "A modern browser (Chrome, Firefox, Safari, Edge — all current versions)", "About 10 minutes"]).map((item, i) => <li key={i}>{item}</li>)}
    </ul>
  </Note>;

An AI chatbot is the most-requested vly build. This recipe covers the complete pattern: streaming responses, persistent conversation history, tool calling (so the bot can take actions on your data), and multi-provider support.

**Time:** \~45 minutes. **Difficulty:** Intermediate.

## What you'll build

* A chat UI with a conversation list and a message input.
* Streaming responses (tokens appear as the model generates them).
* Persistent history (refresh the page; conversations are still there).
* Tool calling — the bot can search your tasks, create a task, or send an email.
* Provider switching (OpenAI vs. Anthropic) without code changes.

## Prerequisites

<Prerequisites
  items={[
"A vly project",
"An [OpenAI](/integrations/openai) account with API access (or [Anthropic](/integrations/anthropic))",
"Familiarity with [Convex actions](/convex/actions) (the recipe uses them heavily)",
"About 45 minutes"
]}
/>

## Step 1 — Project foundation

Use **Plan mode** for the first prompt.

<PromptBlock>
  **Goal**: a streaming AI chatbot. Users have multiple conversations; the bot replies in real time as tokens arrive.

  **User**: Authenticated users only. Each conversation belongs to one user.

  **Data**:

  * `conversations`: `{ userId, title, createdAt, updatedAt }`. Index by\_user\_and\_updated.
  * `messages`: `{ conversationId, role: 'user' | 'assistant' | 'system', content, createdAt }`. Index by\_conversation.

  **Behavior**:

  * Sidebar lists the user's conversations (newest first), with a "New conversation" button.
  * Main pane shows the messages of the active conversation.
  * Message input at the bottom — submit sends the message and triggers a streaming reply.
  * New conversations get an auto-generated title from the first message (a short summary).
  * Use OpenAI gpt-4o-mini as the default model.

  **Constraints**:

  * Streaming: assistant tokens appear in real time as the model generates them.
  * Use Convex's reactive sync to drive the streaming UI (write tokens to the message row; the client subscription auto-updates).
  * Conversation titles are AI-generated after the first user message.
  * Limit conversation history sent to the model: most recent 20 messages.
</PromptBlock>

Submit. Review the plan. Approve.

vly will prompt for `OPENAI_API_KEY`. Paste it.

Build takes \~90 seconds.

## Step 2 — Test the basic flow

1. Open the preview, sign in.
2. Click **New conversation**.
3. Type "What's the capital of France?" and submit.
4. Watch tokens stream in. The message should populate live, not wait for the full response.

If streaming feels jerky or doesn't appear, check the network panel — the issue is usually that the action is buffering instead of writing tokens incrementally.

## Step 3 — Add tool calling

Now make the bot take actions. We'll let it search and create tasks.

<PromptBlock>
  **Goal**: the bot can search and create tasks for the user via tool calling.

  **Behavior**:

  * Add two tools to the OpenAI call:
    * `search_tasks(query: string)` — searches the user's tasks by title and returns matches.
    * `create_task(title: string, dueAt?: string)` — creates a task for the current user.
  * Tool calls execute the actual Convex query/mutation, then the result is fed back to the model for the next turn.
  * When the bot uses a tool, show a small "🔧 Searching tasks..." or "🔧 Creating task..." indicator in the conversation.
  * Multiple tool calls can chain; the bot can search, then create based on what it found.

  **Constraints**:

  * All tool calls happen server-side in the action; the client never sees the tool call payload.
  * Tool calls must respect the user's auth context (search only their tasks; create tasks owned by them).
</PromptBlock>

If you don't have a `tasks` table from a prior recipe, vly will create one (it's a small ask).

After build:

* "Find my tasks about the Q2 report" — bot calls `search_tasks`, returns results.
* "Make a task to follow up next Tuesday" — bot calls `create_task`, confirms.

The bot composes naturally with the data tools. You can add more (`send_email`, `update_task`, etc.) by extending the same pattern.

## Step 4 — Provider switching

Let users pick between OpenAI and Anthropic models.

<PromptBlock>
  **Goal**: users can pick between OpenAI gpt-4o, OpenAI gpt-4o-mini, Anthropic claude-sonnet-4-6, and Anthropic claude-haiku-4-5 for each conversation.

  **Data**:

  * Add `model` field to `conversations` table. Default 'gpt-4o-mini'.

  **Behavior**:

  * Model dropdown above the message input.
  * Selected model is saved per-conversation.
  * The action routes to OpenAI or Anthropic SDK based on the model prefix.

  **Constraints**:

  * Tool calling must work for both providers (each has its own format).
  * Streaming behavior is the same for both.
</PromptBlock>

vly may need an Anthropic API key — paste it when prompted.

After build, switching the dropdown changes which provider runs the next message.

## Step 5 — Polish

Common refinements:

<Tabs>
  <Tab title="Markdown rendering">
    ```text theme={null}
    Render assistant messages as markdown (use react-markdown). Support code blocks
    with syntax highlighting via shiki. Support inline LaTeX with KaTeX.
    ```
  </Tab>

  <Tab title="Copy code blocks">
    ```text theme={null}
    Each code block in an assistant message has a copy-to-clipboard button in the
    top-right corner.
    ```
  </Tab>

  <Tab title="Stop button">
    ```text theme={null}
    While the bot is generating, the send button becomes a stop button. Clicking it
    aborts the in-flight request and saves whatever has been generated so far.
    ```
  </Tab>

  <Tab title="Regenerate">
    ```text theme={null}
    Each assistant message has a "Regenerate" button. Clicking it removes that message
    and regenerates from the previous user message.
    ```
  </Tab>

  <Tab title="Search past conversations">
    ```text theme={null}
    Add a search bar at the top of the sidebar that filters conversations by title or
    by content of any message.
    ```
  </Tab>

  <Tab title="Token / cost tracking">
    ```text theme={null}
    Track input + output tokens per message. Show a small cost estimate per
    conversation in the sidebar. Use OpenAI / Anthropic pricing as of the current model.
    ```
  </Tab>
</Tabs>

## Going further

<CardGroup cols={2}>
  <Card title="Multi-modal (images)" icon="image">
    Allow image attachments. Use GPT-4o or Claude vision. Convex file storage for the upload.
  </Card>

  <Card title="Voice in/out" icon="mic" href="/recipes/voice-ai-app">
    Combine with the voice AI recipe — push-to-talk in, TTS out.
  </Card>

  <Card title="RAG over your docs" icon="brain">
    Embed your docs with [vector search](/convex/vector-search). Add a `search_docs` tool that returns relevant chunks.
  </Card>

  <Card title="Memory across conversations" icon="archive">
    Summarize old conversations and prepend the summary as context to new ones.
  </Card>
</CardGroup>

## Common pitfalls

<AccordionGroup>
  <Accordion title="Tokens not streaming smoothly" icon="alert-triangle">
    The action is buffering. Make sure the streaming loop writes to the database on every chunk (not at the end). Convex's reactive sync handles the client side.
  </Accordion>

  <Accordion title="Tool calls timeout" icon="alert-triangle">
    Convex actions have a 10-min timeout. Long tool chains shouldn't hit this, but if they do, split into multiple actions and use scheduled functions.
  </Accordion>

  <Accordion title="Model returns invalid JSON in tool calls" icon="alert-triangle">
    OpenAI's structured output mode (`response_format: { type: "json_object" }`) helps. For tool calling, the SDK handles parsing — just catch errors and re-prompt the model.
  </Accordion>

  <Accordion title="Costs ballooning" icon="alert-triangle">
    Default to `gpt-4o-mini` for most conversations. Cap conversation history sent to the model. Cache repeated tool results.
  </Accordion>
</AccordionGroup>

## Related

<CardGroup cols={3}>
  <Card title="OpenAI integration" icon="sparkles" href="/integrations/openai">
    The full OpenAI integration reference.
  </Card>

  <Card title="Anthropic integration" icon="brain" href="/integrations/anthropic">
    Same surface, Claude models.
  </Card>

  <Card title="Voice AI app" icon="mic" href="/recipes/voice-ai-app">
    The voice variant.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.