useModel API

The useModel API is the primary interface for interacting with AI models in elizaOS. It provides a unified, type-safe way to access different model types from various providers, with automatic provider selection, error handling, and performance optimization.

Overview

The useModel API abstracts the complexity of working with different AI providers, allowing you to focus on your application logic rather than provider-specific implementation details.

Key Features:

Type-safe model interactions
Automatic provider selection
Parameter validation
Error handling and logging
Performance monitoring
Provider failover support

Basic Usage

Simple Text Generation

import { ModelType } from "@elizaos/core";

// Basic text generation
const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Explain artificial intelligence in simple terms",
  temperature: 0.7,
  maxTokens: 500,
});

console.log(response); // "Artificial intelligence (AI) is..."

API Signature

async useModel<T extends ModelTypeName, R = ModelResultMap[T]>(
  modelType: T,
  params: Omit<ModelParamsMap[T], 'runtime'> | any,
  provider?: string
): Promise<R>

Parameters:

modelType: The type of model to use (e.g., ModelType.TEXT_LARGE)
params: Model-specific parameters (automatically typed based on model type)
provider: Optional provider override (defaults to automatic selection)

Model Types and Parameters

Text Generation Models

TEXT_LARGE / TEXT_SMALL

For complex reasoning and general text generation:

const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Write a product description for a smart home device",
  temperature: 0.8, // Creativity level (0.0-1.0)
  maxTokens: 1000, // Maximum response length
  stopSequences: ["\n\n"], // Stop generation at these sequences
  frequencyPenalty: 0.1, // Reduce repetition
  presencePenalty: 0.1, // Encourage diverse topics
});

TEXT_REASONING_LARGE / TEXT_REASONING_SMALL

For logical reasoning and problem-solving:

const solution = await runtime.useModel(ModelType.TEXT_REASONING_LARGE, {
  prompt: "Solve this logic puzzle: If all roses are flowers...",
  temperature: 0.1, // Low temperature for logical consistency
  maxTokens: 800,
});

TEXT_COMPLETION

For text completion tasks:

const completion = await runtime.useModel(ModelType.TEXT_COMPLETION, {
  prompt: "The key to successful project management is",
  temperature: 0.6,
  maxTokens: 300,
});

Embedding Models

TEXT_EMBEDDING

Convert text to vector embeddings:

// Simple text embedding
const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: "This is a sample text for embedding",
});

console.log(embedding); // [0.1, -0.2, 0.5, ...] (number array)

// Alternative parameter formats
const embedding2 = await runtime.useModel(ModelType.TEXT_EMBEDDING, "Direct text string");

const embedding3 = await runtime.useModel(ModelType.TEXT_EMBEDDING, null);

Tokenization Models

TEXT_TOKENIZER_ENCODE

Convert text to tokens:

const tokens = await runtime.useModel(ModelType.TEXT_TOKENIZER_ENCODE, {
  prompt: "Hello, world!",
  modelType: ModelType.TEXT_TOKENIZER_ENCODE,
});

console.log(tokens); // [15496, 11, 1917, 0] (number array)

TEXT_TOKENIZER_DECODE

Convert tokens back to text:

const text = await runtime.useModel(ModelType.TEXT_TOKENIZER_DECODE, {
  tokens: [15496, 11, 1917, 0],
  modelType: ModelType.TEXT_TOKENIZER_DECODE,
});

console.log(text); // "Hello, world!"

Image Models

IMAGE

Generate images from text descriptions:

const images = await runtime.useModel(ModelType.IMAGE, {
  prompt: "A serene mountain landscape at sunset",
  size: "1024x1024",
  count: 1,
});

console.log(images); // [{ url: "https://..." }]

IMAGE_DESCRIPTION

Describe images with text:

// Using image URL
const description = await runtime.useModel(ModelType.IMAGE_DESCRIPTION, {
  imageUrl: "https://example.com/image.jpg",
  prompt: "Describe this image in detail",
});

// Using direct URL
const description2 = await runtime.useModel(
  ModelType.IMAGE_DESCRIPTION,
  "https://example.com/image.jpg"
);

console.log(description); // { title: "Mountain Scene", description: "A beautiful..." }

Audio Models

TRANSCRIPTION

Convert audio to text:

// Using audio URL
const transcript = await runtime.useModel(ModelType.TRANSCRIPTION, {
  audioUrl: "https://example.com/audio.mp3",
  prompt: "This is a meeting recording",
});

// Using audio buffer
const audioBuffer = await fs.readFile("audio.wav");
const transcript2 = await runtime.useModel(ModelType.TRANSCRIPTION, audioBuffer);

console.log(transcript); // "Hello everyone, welcome to the meeting..."

TEXT_TO_SPEECH

Convert text to audio:

const audio = await runtime.useModel(ModelType.TEXT_TO_SPEECH, {
  text: "Hello, this is a test of text-to-speech",
  voice: "alloy",
  speed: 1.0,
});

// Direct text input
const audio2 = await runtime.useModel(ModelType.TEXT_TO_SPEECH, "Hello, this is a test");

Object Generation Models

OBJECT_SMALL / OBJECT_LARGE

Generate structured objects:

const userProfile = await runtime.useModel(ModelType.OBJECT_LARGE, {
  prompt: "Generate a user profile for a software developer",
  schema: {
    type: "object",
    properties: {
      name: { type: "string" },
      age: { type: "number" },
      skills: {
        type: "array",
        items: { type: "string" },
      },
      experience: { type: "number" },
    },
    required: ["name", "age", "skills"],
  },
  temperature: 0.7,
});

console.log(userProfile);
// { name: "Alex Johnson", age: 28, skills: ["JavaScript", "Python"], ... }

Advanced Usage

Provider Selection

Automatic Provider Selection

By default, models are selected based on priority and availability:

// Uses the highest priority provider that supports TEXT_LARGE
const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Generate a response",
});

Explicit Provider Selection

Force a specific provider:

// Force OpenAI provider
const response = await runtime.useModel(
  ModelType.TEXT_LARGE,
  {
    prompt: "Generate a response",
  },
  "openai"
);

// Force Anthropic provider
const response2 = await runtime.useModel(
  ModelType.TEXT_LARGE,
  {
    prompt: "Generate a response",
  },
  "anthropic"
);

Parameter Validation

The API provides compile-time type checking:

// TypeScript will catch parameter mismatches
const response = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  prompt: "This will cause a type error", // Error: 'prompt' doesn't exist on TextEmbeddingParams
  text: "This is correct", // Correct parameter
});

Error Handling

Basic Error Handling

try {
  const response = await runtime.useModel(ModelType.TEXT_LARGE, {
    prompt: "Generate a response",
    temperature: 0.7,
  });

  console.log(response);
} catch (error) {
  console.error("Model generation failed:", error.message);

  // Handle specific error types
  if (error.message.includes("API key")) {
    console.error("Check your API key configuration");
  } else if (error.message.includes("rate limit")) {
    console.error("Rate limit exceeded, try again later");
  }
}

Fallback Strategies

async function generateWithFallback(runtime, prompt) {
  const providers = ["openai", "anthropic", "ollama"];

  for (const provider of providers) {
    try {
      return await runtime.useModel(
        ModelType.TEXT_LARGE,
        {
          prompt,
          temperature: 0.7,
        },
        provider
      );
    } catch (error) {
      console.warn(`Provider ${provider} failed:`, error.message);

      // Continue to next provider
      if (provider === providers[providers.length - 1]) {
        throw new Error("All providers failed");
      }
    }
  }
}

Performance Optimization

Response Caching

const cache = new Map();

async function getCachedResponse(prompt) {
  const cacheKey = `text_large:${prompt}`;

  if (cache.has(cacheKey)) {
    return cache.get(cacheKey);
  }

  const response = await runtime.useModel(ModelType.TEXT_LARGE, {
    prompt,
    temperature: 0.7,
  });

  cache.set(cacheKey, response);
  return response;
}

Batch Processing

async function processBatch(prompts) {
  const promises = prompts.map((prompt) =>
    runtime.useModel(ModelType.TEXT_LARGE, {
      prompt,
      temperature: 0.7,
    })
  );

  return await Promise.all(promises);
}

Model Integration Patterns

Memory Integration

Use models with memory management:

// Generate embedding for memory storage
const memory = {
  content: { text: "User asked about pricing" },
  entityId: userId,
  roomId: roomId,
  worldId: worldId,
};

const memoryWithEmbedding = await runtime.addEmbeddingToMemory(memory);

Action Integration

Use models within actions:

const analyzeAction = {
  name: "ANALYZE_DATA",
  handler: async (runtime, message, state) => {
    const analysis = await runtime.useModel(ModelType.TEXT_LARGE, {
      prompt: `Analyze this data: ${message.content.text}`,
      temperature: 0.3,
      maxTokens: 1000,
    });

    return {
      text: analysis,
      action: "ANALYZE_DATA",
    };
  },
};

Provider Integration

Use models within providers:

const analysisProvider = {
  name: "ANALYSIS_PROVIDER",
  get: async (runtime, message, state) => {
    const analysis = await runtime.useModel(ModelType.TEXT_LARGE, {
      prompt: `Provide analysis context for: ${message.content.text}`,
      temperature: 0.5,
      maxTokens: 500,
    });

    return {
      text: analysis,
      values: { hasAnalysis: true },
      data: { analysisResult: analysis },
    };
  },
};

Real-World Examples

Conversational Agent

const conversationHandler = async (runtime, message, state) => {
  // Generate contextual response
  const response = await runtime.useModel(ModelType.TEXT_LARGE, {
    prompt: `
      Previous conversation: ${state.recentMessages}
      User: ${message.content.text}
      
      Respond as a helpful assistant:
    `,
    temperature: 0.8,
    maxTokens: 800,
    stopSequences: ["\nUser:", "\nAssistant:"],
  });

  return response;
};

Content Moderation

const moderateContent = async (runtime, content) => {
  const moderation = await runtime.useModel(ModelType.TEXT_LARGE, {
    prompt: `
      Analyze this content for inappropriate material:
      "${content}"
      
      Respond with: SAFE, INAPPROPRIATE, or HARMFUL
    `,
    temperature: 0.1,
    maxTokens: 50,
  });

  return moderation.trim().toUpperCase();
};

Code Analysis

const analyzeCode = async (runtime, code) => {
  const analysis = await runtime.useModel(ModelType.TEXT_REASONING_LARGE, {
    prompt: `
      Analyze this code for potential issues:
      
      \`\`\`
      ${code}
      \`\`\`
      
      Provide:
      1. Potential bugs
      2. Security issues
      3. Performance problems
      4. Best practice violations
    `,
    temperature: 0.2,
    maxTokens: 1500,
  });

  return analysis;
};

Document Summary

const summarizeDocument = async (runtime, document) => {
  // First, create embedding for the document
  const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: document,
  });

  // Generate summary
  const summary = await runtime.useModel(ModelType.TEXT_LARGE, {
    prompt: `
      Summarize this document in 3 bullet points:
      
      ${document}
      
      Summary:
    `,
    temperature: 0.3,
    maxTokens: 300,
  });

  return {
    summary,
    embedding,
  };
};

Best Practices

1. Parameter Optimization

// Use appropriate parameters for your use case
const creativeResponse = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Write a creative story",
  temperature: 0.9, // High creativity
  maxTokens: 2000,
  presencePenalty: 0.2, // Encourage diverse topics
});

const factualResponse = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Explain quantum physics",
  temperature: 0.1, // Low creativity for accuracy
  maxTokens: 1000,
  frequencyPenalty: 0.1, // Reduce repetition
});

2. Model Selection

// Choose appropriate model sizes
const quickResponse = await runtime.useModel(ModelType.TEXT_SMALL, {
  prompt: "What's the weather like?",
});

const complexAnalysis = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Analyze the economic implications of this policy",
});

3. Error Recovery

async function robustModelCall(runtime, params) {
  const maxRetries = 3;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await runtime.useModel(ModelType.TEXT_LARGE, params);
    } catch (error) {
      if (attempt === maxRetries) {
        throw error;
      }

      // Exponential backoff
      await new Promise((resolve) => setTimeout(resolve, Math.pow(2, attempt) * 1000));
    }
  }
}

4. Monitoring and Logging

const monitoredModelCall = async (runtime, modelType, params) => {
  const startTime = Date.now();

  try {
    const result = await runtime.useModel(modelType, params);
    const duration = Date.now() - startTime;

    runtime.logger.info("Model call succeeded", {
      modelType,
      duration,
      inputLength: params.prompt?.length || 0,
      outputLength: result.length || 0,
    });

    return result;
  } catch (error) {
    const duration = Date.now() - startTime;

    runtime.logger.error("Model call failed", {
      modelType,
      duration,
      error: error.message,
    });

    throw error;
  }
};

Troubleshooting

Common Issues

No model handler found

// Check if model type is supported
const handler = runtime.getModel(ModelType.TEXT_LARGE);
if (!handler) {
  console.error("No provider supports TEXT_LARGE");
}

Type errors

// Ensure parameters match model type
const response = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: "Correct parameter", // Use 'text', not 'prompt'
});

Provider-specific errors

// Check provider availability
try {
  const response = await runtime.useModel(
    ModelType.TEXT_LARGE,
    {
      prompt: "Test",
    },
    "openai"
  );
} catch (error) {
  if (error.message.includes("API key")) {
    console.error("OpenAI API key not configured");
  }
}

Debug Information

Enable detailed logging:

// Set environment variable for debug logging
process.env.LOG_LEVEL = "debug";

// The runtime will log model calls automatically
const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Test prompt",
});

The useModel API provides a powerful, flexible interface for AI model integration in elizaOS. By understanding the different model types, parameters, and patterns, you can build sophisticated AI-powered applications that leverage the best capabilities of each model type and provider.

useModel API

On this page