OpenAI

Add OpenAI’s models — GPT-4o, GPT-4o mini, embeddings, image generation, audio — to your vly app. The default LLM provider for AI features.

OpenAI is the most-used AI integration in vly. The integration supports chat completions, embeddings (for vector search), image generation, and audio (Whisper STT and TTS).

Prerequisites

Setup

1. Get your OpenAI API key

   1. Sign in at platform.openai.com.
   2. Click API keys → Create new secret key.
   3. Name it vly-yourapp (or similar).
   4. Copy the key (starts with sk-...). You won’t see it again.
2. Connect OpenAI in vly

   In a prompt:

   Add OpenAI for AI features. I'll provide my API key.
   

   vly opens the integration setup dialog. Paste your key and confirm. The key is stored encrypted as OPENAI_API_KEY.

3. Use it in a feature

   When a task is created, generate a one-line summary of the task
   description using OpenAI's gpt-4o-mini model. Save it in a new
   `summary` field on the task.
   

   vly creates the action, calls the API, and writes the result back via a mutation. See the code patterns below for what gets generated.

Configuration

OPENAI_API_KEY · string · required

Your OpenAI API key. Server-only. Starts with sk-....

OPENAI_ORG_ID · string

Optional. Your OpenAI organization ID, if you want to scope usage to a specific org.

OPENAI_MODEL · string · default: "gpt-4o-mini"

Default model for generic AI features. Override per-call. Common values: gpt-4o, gpt-4o-mini, gpt-4-turbo.

Code patterns

Chat completion

convex/ai.ts

import { action } from "./_generated/server";
import { v } from "convex/values";
import OpenAI from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! });

export const summarize = action({
  args: { text: v.string() },
  handler: async (_ctx, { text }) => {
    const response = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "Summarize the following in one sentence." },
        { role: "user",   content: text },
      ],
      max_tokens: 100,
    });
    return response.choices[0].message.content ?? "";
  },
});

Streaming chat

For chatbot UIs, stream tokens to the client as they arrive:

convex/ai.ts

import { action } from "./_generated/server";

export const streamChat = action({
  args: { messages: v.array(v.object({
    role: v.union(v.literal("user"), v.literal("assistant"), v.literal("system")),
    content: v.string(),
  })) },
  handler: async (ctx, { messages }) => {
    const stream = await openai.chat.completions.create({
      model: "gpt-4o",
      messages,
      stream: true,
    });

    let buffer = "";
    for await (const chunk of stream) {
      const delta = chunk.choices[0]?.delta?.content ?? "";
      buffer += delta;
      // Update a Convex document with the running buffer so the
      // client's reactive query auto-updates.
      await ctx.runMutation(api.chats.appendStream, {
        chatId: messages[0].chatId,
        content: buffer,
      });
    }
    return buffer;
  },
});

The client reads via useQuery(api.chats.get, { id }) and sees tokens stream in real time.

Embeddings (for vector search)

convex/ai.ts

export const embed = action({
  args: { text: v.string() },
  handler: async (_ctx, { text }) => {
    const response = await openai.embeddings.create({
      model: "text-embedding-3-small",
      input: text,
    });
    return response.data[0].embedding;
  },
});

Pair with Convex vector search for semantic queries.

Image generation (DALL-E)

convex/ai.ts

export const generateImage = action({
  args: { prompt: v.string() },
  handler: async (_ctx, { prompt }) => {
    const response = await openai.images.generate({
      model: "dall-e-3",
      prompt,
      n: 1,
      size: "1024x1024",
    });
    return response.data[0].url;
  },
});

For persistent storage, download the image URL and upload to Convex file storage.

Audio (Whisper STT)

convex/ai.ts

export const transcribe = action({
  args: { audioUrl: v.string() },
  handler: async (_ctx, { audioUrl }) => {
    const audio = await fetch(audioUrl).then((r) => r.arrayBuffer());
    const file  = new File([audio], "audio.mp3", { type: "audio/mpeg" });

    const response = await openai.audio.transcriptions.create({
      file,
      model: "whisper-1",
    });
    return response.text;
  },
});

Tool / function calling

For agents that can call your own functions:

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages,
  tools: [{
    type: "function",
    function: {
      name: "search_tasks",
      description: "Search tasks by title or tag",
      parameters: {
        type: "object",
        properties: {
          query: { type: "string", description: "Search text" },
        },
        required: ["query"],
      },
    },
  }],
});

if (response.choices[0].message.tool_calls) {
  // Run the requested function and feed the result back
  // ...
}

Use this for AI chatbots that take actions.

Cost management

OpenAI bills per token. Control costs by:

Use the smallest model that works

gpt-4o-mini is 15× cheaper than gpt-4o and good enough for most summarization, extraction, and simple chat tasks.

Cap tokens

Set max_tokens on every call. Prevents runaway responses and runaway bills.

Cache responses

For deterministic prompts (summarize the same text), cache the result in a Convex table keyed by content hash.

Set a usage limit in OpenAI

OpenAI dashboard → Settings → Limits. Set a hard monthly cap; calls fail when exceeded (better than a surprise bill).

Choosing a model

Task	Recommended	Notes
Quick chat / chatbot	gpt-4o-mini	Fast, cheap, good enough for 80% of conversational use
High-quality reasoning	gpt-4o	When you need careful logic — code review, complex extraction
Embeddings	text-embedding-3-small	1536 dims, cheap, accurate
Image generation	dall-e-3	Best quality, slower than DALL-E 2
Speech-to-text	whisper-1	Multi-lingual, robust to noise
Text-to-speech	tts-1	6 voices, natural-sounding

Comparison with other providers

Capability	OpenAI	Anthropic	Google AI
Chat / completion	✓	✓	✓
Embeddings	✓	✗	✓
Image generation	✓ (DALL-E)	✗	✓ (Imagen)
Audio (STT)	✓ (Whisper)	✗	✓
Audio (TTS)	✓	✗	✓
Function calling	✓	✓	✓
Vision (image input)	✓	✓	✓
Pricing	$$	$$	$

For chatbots, Anthropic’s Claude is often a better choice for nuanced conversations. For everything else, OpenAI is the default.

Troubleshooting

'You exceeded your current quota'

Your OpenAI account has no credit, or you hit your usage limit. Add credit at platform.openai.com/billing.

'The model gpt-4o does not exist or you do not have access'

Some models require tier-1+ access. Check Tier requirements. Most accounts auto-graduate after $5 of usage.

Streaming response 'cuts off'

max_tokens is set too low, or the model hit its context window. Increase max_tokens, or for very long inputs, use a model with a larger context window.

Slow response time

Some prompts are slow regardless of model. Add a loading state in the UI; for chatbots, switch to streaming so users see partial output.

Related

AI chatbot recipe

Complete walkthrough — streaming, history, tool calls.

Image generator recipe

User-facing image generation with DALL-E.

Vector search

Semantic search with OpenAI embeddings + Convex.