elizaOS
Core Concepts/AI & Language Models

Model Providers

Working with different AI model providers in elizaOS v1.2.0 with enhanced plugin-based architecture

Model providers in elizaOS v1.2.0 abstract the complexity of working with different AI services, allowing you to seamlessly switch between providers or use multiple providers simultaneously. Each provider implements a standardized interface while handling the specifics of their respective APIs, with enhanced plugin-based architecture for better flexibility and control.

Overview

The model provider system enables:

  • Unified API: Single interface for all AI models
  • Runtime switching: Change providers without code modifications
  • Multi-provider support: Use different providers for different model types
  • Automatic fallbacks: Graceful degradation when providers are unavailable
  • Priority-based selection: Control which provider is used for specific tasks
  • Plugin-based architecture: Enhanced modularity and extensibility (v1.2.0)
  • Model capability detection: Automatic detection of provider capabilities (v1.2.0)
  • Enhanced routing: Smart routing based on model requirements (v1.2.0)

v1.2.0 Plugin-Based Architecture

In v1.2.0, model providers are implemented as plugins that register model handlers with the runtime. This provides better modularity and allows for more sophisticated model routing.

Model Handler Registration

Plugins now register model handlers for specific model types:

export const openaiPlugin: Plugin = {
  name: "openai",
  description: "OpenAI model integration",
  version: "1.2.0",

  // Register model handlers
  models: {
    [ModelType.TEXT_LARGE]: async (runtime: IAgentRuntime, params) => {
      const client = createOpenAIClient(runtime);
      const response = await client.chat.completions.create({
        model: "gpt-4",
        messages: [{ role: "user", content: params.prompt }],
        temperature: params.temperature || 0.7,
        max_tokens: params.maxTokens || 1500,
      });
      return response.choices[0].message.content;
    },

    [ModelType.TEXT_EMBEDDING]: async (runtime: IAgentRuntime, params) => {
      const client = createOpenAIClient(runtime);
      const response = await client.embeddings.create({
        model: "text-embedding-3-large",
        input: params.text,
      });
      return response.data[0].embedding;
    },

    [ModelType.IMAGE_GENERATION]: async (runtime: IAgentRuntime, params) => {
      const client = createOpenAIClient(runtime);
      const response = await client.images.generate({
        model: "dall-e-3",
        prompt: params.prompt,
        size: params.size || "1024x1024",
      });
      return response.data[0].url;
    },
  },

  // Plugin configuration
  config: {
    OPENAI_API_KEY: process.env.OPENAI_API_KEY,
    OPENAI_API_BASE: process.env.OPENAI_API_BASE,
  },
};

Enhanced Model Routing

The runtime now supports sophisticated model routing based on capabilities:

// Runtime automatically selects best handler
const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Explain quantum computing",
  temperature: 0.7,
  maxTokens: 1000,
  // Additional context for smart routing
  context: {
    requestId: uuidv4(),
    runId: runtime.runId,
    requiresStreaming: false,
    prioritizeLatency: true,
  },
});

Model Capabilities Detection

Plugins can declare their capabilities for better routing:

export const enhancedOpenAIPlugin: Plugin = {
  name: "openai",
  models: {
    [ModelType.TEXT_LARGE]: async (runtime, params) => {
      // Handler implementation
    },
  },

  // Declare capabilities (new in v1.2.0)
  capabilities: {
    [ModelType.TEXT_LARGE]: {
      supportsStreaming: true,
      maxTokens: 4096,
      supportedFormats: ["text", "json"],
      costPerToken: 0.00003,
      averageLatency: 2000, // ms
      reliabilityScore: 0.99,
    },
  },
};

Priority-Based Selection

Multiple plugins can register handlers for the same model type, with priority-based selection:

// High priority OpenAI plugin
runtime.registerModel(
  ModelType.TEXT_LARGE,
  openaiHandler,
  "openai",
  1000 // High priority
);

// Lower priority Anthropic plugin
runtime.registerModel(
  ModelType.TEXT_LARGE,
  anthropicHandler,
  "anthropic",
  900 // Lower priority
);

// Runtime selects highest priority handler
const response = await runtime.useModel(ModelType.TEXT_LARGE, params);

Supported Model Providers

OpenAI

The OpenAI provider offers access to GPT models, embeddings, and DALL-E image generation.

Installation:

npm install @elizaos/plugin-openai

Configuration:

// Character file
{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-openai"]
}

// Environment variables
OPENAI_API_KEY=your-api-key

**Environment Variables:**
```bash
OPENAI_API_KEY=sk-...
OPENAI_API_BASE=https://api.openai.com/v1  # Optional

Supported Models:

  • Text Generation: GPT-4, GPT-4 Turbo, GPT-3.5-turbo
  • Embeddings: text-embedding-3-large, text-embedding-3-small
  • Image Generation: DALL-E 3, DALL-E 2
  • Audio: Whisper (transcription), TTS (text-to-speech)

Usage Example:

// Automatic model selection based on modelProvider
const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Explain quantum computing",
  temperature: 0.7,
  maxTokens: 1000,
});

Anthropic (Claude)

The Anthropic provider provides access to Claude models known for their safety and reasoning capabilities.

Installation:

npm install @elizaos/plugin-anthropic

Configuration:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-anthropic"]
}

// Environment variables
ANTHROPIC_API_KEY=your-api-key

**Environment Variables:**
```bash
ANTHROPIC_API_KEY=sk-ant-...

Supported Models:

  • Text Generation: Claude-3 Opus, Claude-3 Sonnet, Claude-3 Haiku
  • Vision: Claude-3 Vision (image description)

Usage Example:

const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Analyze this code for potential issues",
  temperature: 0.3,
  maxTokens: 2000,
});

Google Gemini

The Google Gemini provider offers access to Google's multimodal AI models.

Installation:

npm install @elizaos/plugin-google-genai

Configuration:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-google-genai"]
}

// Environment variables
GOOGLE_GENERATIVE_AI_API_KEY=your-api-key

**Environment Variables:**
```bash
GOOGLE_GENERATIVE_AI_API_KEY=AI...

Supported Models:

  • Text Generation: Gemini Pro, Gemini Flash
  • Embeddings: text-embedding-004
  • Vision: Gemini Pro Vision

Usage Example:

const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Write a creative story about AI",
  temperature: 0.8,
  maxTokens: 1500,
});

Ollama (Local Models)

The Ollama provider enables running open-source models locally.

Installation:

npm install @elizaos/plugin-ollama

Prerequisites:

  1. Install Ollama: https://ollama.ai
  2. Pull models: ollama pull llama3.1

Configuration:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-ollama"]
}

// Environment variables
OLLAMA_API_ENDPOINT=http://localhost:11434
OLLAMA_MODEL=llama3.1

**Environment Variables:**
```bash
OLLAMA_API_ENDPOINT=http://localhost:11434
OLLAMA_MODEL=llama3.1

Supported Models:

  • Text Generation: Llama 3.1, Mistral, CodeLlama, Phi-3
  • Embeddings: nomic-embed-text, all-minilm
  • Vision: llava, moondream

Usage Example:

const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Explain machine learning concepts",
  temperature: 0.6,
  maxTokens: 1000,
});

Heurist AI

The Heurist provider offers decentralized AI inference.

Installation:

npm install @elizaos/plugin-heurist

Configuration:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-heurist"]
}

// Environment variables
HEURIST_API_KEY=your-api-key

### OpenRouter

The OpenRouter provider gives access to multiple models through a unified API.

**Installation:**
```bash
npm install @elizaos/plugin-openrouter

Configuration:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-openrouter"]
}

// Environment variables
OPENROUTER_API_KEY=your-api-key

**Environment Variables:**
```bash
OPENROUTER_API_KEY=sk-or-...

Local AI

The Local AI provider enables running AI models completely offline.

Installation:

npm install @elizaos/plugin-local-ai

Configuration:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-local-ai"]
}

// Environment variables
USE_LOCAL_AI=true

Usage Example:

const response = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Generate a response using local AI",
  temperature: 0.7,
  maxTokens: 1000,
});

Provider Selection Strategies

Primary Provider

The ElizaOS runtime automatically selects the best available provider based on plugin order and priority:

{
  "name": "MyAgent",
  "plugins": ["@elizaos/plugin-openai"]  // Single provider
}

Multi-Provider Setup

Use different providers for different capabilities:

{
  "name": "MyAgent",
  "plugins": [
    "@elizaos/plugin-sql",        // Always first
    "@elizaos/plugin-anthropic",  // Text generation
    "@elizaos/plugin-openai",     // Embeddings and image generation
    "@elizaos/plugin-ollama"      // Local models for privacy
  ]
}

Priority-Based Selection

Control provider selection through plugin priority:

// Higher priority plugins are selected first
const plugins = [
  { name: "@elizaos/plugin-openai", priority: 1000 },
  { name: "@elizaos/plugin-anthropic", priority: 900 },
  { name: "@elizaos/plugin-ollama", priority: 500 },
];

Conditional Provider Loading

Load providers based on environment variables:

const getPlugins = () => {
  const plugins = ["@elizaos/plugin-sql"];

  // Add providers based on available API keys
  if (process.env.OPENAI_API_KEY) {
    plugins.push("@elizaos/plugin-openai");
  }
  if (process.env.ANTHROPIC_API_KEY) {
    plugins.push("@elizaos/plugin-anthropic");
  }
  if (process.env.OLLAMA_API_ENDPOINT) {
    plugins.push("@elizaos/plugin-ollama");
  }

  return plugins;
};

export const character = {
  name: "MyAgent",
  plugins: getPlugins(),
};

Advanced Configuration

Model-Specific Settings

Configure individual model parameters:

{
  "name": "MyAgent",
  "modelProvider": "openai",
  "settings": {
    "modelConfig": {
      "temperature": 0.7,
      "maxOutputTokens": 2000,
      "frequency_penalty": 0.1,
      "presence_penalty": 0.1
    }
  }
}

Custom Endpoints

Use custom API endpoints:

{
  "name": "MyAgent",
  "modelProvider": "openai",
  "modelEndpointOverride": "https://custom-api.example.com/v1",
  "settings": {
    "secrets": {
      "OPENAI_API_KEY": "custom-key"
    }
  }
}

Provider-Specific Configuration

Different providers may have unique configuration options:

{
  "name": "MyAgent",
  "settings": {
    // OpenAI specific
    "openai": {
      "organization": "org-123",
      "project": "proj-456"
    },

    // Anthropic specific
    "anthropic": {
      "version": "2023-06-01"
    },

    // Ollama specific
    "ollama": {
      "host": "localhost:11434",
      "keep_alive": "5m"
    }
  }
}

Runtime Provider Management

Dynamic Provider Selection

Select providers programmatically with enhanced context:

// In your action or provider
const response = await runtime.useModel(
  ModelType.TEXT_LARGE,
  {
    prompt: "Generate response",
    temperature: 0.7,
    context: {
      requestId: uuidv4(),
      runId: runtime.runId,
      actionContext: runtime.currentActionContext,
    },
  },
  "anthropic"
); // Force specific provider

Provider Availability Check

Check if a provider is available with enhanced capabilities:

const textLargeHandler = runtime.getModel(ModelType.TEXT_LARGE);
if (textLargeHandler) {
  const capabilities = runtime.getModelCapabilities(ModelType.TEXT_LARGE);

  console.log(`Provider: ${textLargeHandler.provider}`);
  console.log(`Max tokens: ${capabilities.maxTokens}`);
  console.log(`Supports streaming: ${capabilities.supportsStreaming}`);

  const response = await runtime.useModel(ModelType.TEXT_LARGE, {
    prompt: "Generate response",
    maxTokens: Math.min(1000, capabilities.maxTokens),
  });
}

Enhanced Fallback Mechanisms

Implement automatic fallbacks with capability-aware selection:

async function generateWithFallback(runtime, prompt, requirements = {}) {
  const modelType = ModelType.TEXT_LARGE;
  const availableHandlers = runtime.getModelHandlers(modelType);

  // Sort by capability score
  const sortedHandlers = availableHandlers.sort((a, b) => {
    const scoreA = calculateCapabilityScore(a.capabilities, requirements);
    const scoreB = calculateCapabilityScore(b.capabilities, requirements);
    return scoreB - scoreA;
  });

  for (const handler of sortedHandlers) {
    try {
      return await runtime.useModel(
        modelType,
        {
          prompt,
          temperature: 0.7,
          context: {
            requestId: uuidv4(),
            runId: runtime.runId,
            preferredProvider: handler.provider,
          },
        },
        handler.provider
      );
    } catch (error) {
      console.warn(`Provider ${handler.provider} failed:`, error.message);

      // Log failure for monitoring
      await runtime.emitEvent("model_provider_failed", {
        provider: handler.provider,
        modelType,
        error: error.message,
        runId: runtime.runId,
      });

      continue;
    }
  }

  throw new Error("All providers failed");
}

function calculateCapabilityScore(capabilities, requirements) {
  let score = 0;

  if (requirements.prioritizeLatency && capabilities.averageLatency < 3000) {
    score += 10;
  }

  if (requirements.prioritizeCost && capabilities.costPerToken < 0.0001) {
    score += 5;
  }

  if (requirements.requiresStreaming && capabilities.supportsStreaming) {
    score += 15;
  }

  // Reliability bonus
  score += capabilities.reliabilityScore * 10;

  return score;
}

Model Performance Monitoring

Track model performance across providers:

class ModelPerformanceMonitor {
  private metrics: Map<string, ModelMetrics> = new Map();

  async trackModelCall(
    provider: string,
    modelType: ModelType,
    startTime: number,
    endTime: number,
    success: boolean,
    tokenCount?: number
  ) {
    const key = `${provider}:${modelType}`;
    const existing = this.metrics.get(key) || {
      provider,
      modelType,
      totalCalls: 0,
      successfulCalls: 0,
      averageLatency: 0,
      totalTokens: 0,
    };

    existing.totalCalls++;
    if (success) {
      existing.successfulCalls++;
    }

    const latency = endTime - startTime;
    existing.averageLatency = (existing.averageLatency + latency) / 2;

    if (tokenCount) {
      existing.totalTokens += tokenCount;
    }

    this.metrics.set(key, existing);
  }

  getMetrics(): ModelMetrics[] {
    return Array.from(this.metrics.values());
  }

  getBestProvider(modelType: ModelType): string | null {
    const providers = Array.from(this.metrics.values())
      .filter((m) => m.modelType === modelType)
      .sort((a, b) => {
        const scoreA = (a.successfulCalls / a.totalCalls) * 100 - a.averageLatency;
        const scoreB = (b.successfulCalls / b.totalCalls) * 100 - b.averageLatency;
        return scoreB - scoreA;
      });

    return providers[0]?.provider || null;
  }
}

Provider Comparison

ProviderStrengthsBest ForLimitations
OpenAIMost advanced models, comprehensive APIProduction apps, complex reasoningCost, API dependency
AnthropicSafety-focused, excellent reasoningContent analysis, code reviewLimited model variety
Google GeminiFast, multimodal, competitive pricingReal-time apps, mixed mediaNewer ecosystem
OllamaPrivacy, no API costs, customizableLocal deployment, privacy-sensitiveRequires local resources
HeuristDecentralized, cost-effectiveBlockchain apps, distributed systemsLimited model selection
OpenRouterAccess to many models, unified APIModel comparison, flexibilityThird-party dependency
Local AIComplete privacy, no API callsOffline deployment, sensitive dataSetup complexity, resource intensive

Cost Optimization

Model Size Selection

Choose appropriate model sizes:

// Use smaller models for simple tasks
const simpleResponse = await runtime.useModel(ModelType.TEXT_SMALL, {
  prompt: "Summarize in one sentence",
});

// Use larger models for complex reasoning
const complexResponse = await runtime.useModel(ModelType.TEXT_LARGE, {
  prompt: "Analyze this complex problem",
});

Provider Cost Comparison

Monitor usage across providers:

const costTracker = {
  openai: { requests: 0, tokens: 0 },
  anthropic: { requests: 0, tokens: 0 },
  ollama: { requests: 0, tokens: 0 },
};

// Track usage in your application
function trackUsage(provider, tokens) {
  costTracker[provider].requests++;
  costTracker[provider].tokens += tokens;
}

Best Practices

1. Provider Redundancy

Always have backup providers:

const primaryProviders = ["openai", "anthropic"];
const fallbackProviders = ["ollama", "local-ai"];

2. Environment-Specific Configuration

Use different providers for different environments:

const getProviderConfig = () => {
  if (process.env.NODE_ENV === "production") {
    return {
      modelProvider: "openai",
      plugins: ["@elizaos/plugin-openai", "@elizaos/plugin-anthropic"],
    };
  } else {
    return {
      modelProvider: "ollama",
      plugins: ["@elizaos/plugin-ollama"],
    };
  }
};

3. Model Type Specialization

Use different providers for different model types:

// OpenAI for embeddings
const embedding = await runtime.useModel(
  ModelType.TEXT_EMBEDDING,
  {
    text: "Sample text",
  },
  "openai"
);

// Anthropic for reasoning
const reasoning = await runtime.useModel(
  ModelType.TEXT_LARGE,
  {
    prompt: "Analyze this data",
  },
  "anthropic"
);

// Ollama for privacy-sensitive tasks
const privateTask = await runtime.useModel(
  ModelType.TEXT_LARGE,
  {
    prompt: "Process confidential data",
  },
  "ollama"
);

4. Configuration Validation

Validate provider configurations on startup:

export const validateProviderConfig = (character) => {
  const requiredVars = {
    openai: ["OPENAI_API_KEY"],
    anthropic: ["ANTHROPIC_API_KEY"],
    google: ["GOOGLE_GENERATIVE_AI_API_KEY"],
  };

  const provider = character.modelProvider;
  const required = requiredVars[provider];

  if (required) {
    for (const envVar of required) {
      if (!process.env[envVar]) {
        throw new Error(`${envVar} is required for ${provider} provider`);
      }
    }
  }
};

Troubleshooting

Common Issues

  1. Provider not available:

    • Check plugin installation
    • Verify API keys
    • Ensure provider is in plugins array

  2. Model not supported:

    • Check provider capabilities
    • Verify model type constants
    • Check provider documentation

  3. API key issues:

    • Verify environment variables
    • Check API key format
    • Confirm API key permissions

Debug Information

Enable debug logging:

// Set environment variable
LOG_LEVEL = debug;

// Or in code
runtime.logger.debug("Provider selection:", {
  modelType: ModelType.TEXT_LARGE,
  availableProviders: Object.keys(runtime.models),
});

The model provider system in elizaOS v1.2.0 offers enhanced flexibility and reliability for AI model integration. By understanding the different providers and their capabilities, you can build robust agents that leverage the best models for each task while maintaining fallback options for reliability.

v1.2.0 Key Improvements

Plugin-Based Architecture

  • Modular Design: Providers are now plugins that can be independently developed and maintained
  • Dynamic Registration: Model handlers are registered at runtime with priority-based selection
  • Enhanced Capabilities: Providers can declare their capabilities for smart routing

Advanced Model Routing

  • Capability-Aware Selection: Runtime automatically selects the best provider based on requirements
  • Context Tracking: All model calls include rich context for debugging and monitoring
  • Performance Monitoring: Built-in performance tracking and optimization

Enhanced Debugging

  • Run ID Tracking: All model calls are tracked with unique run IDs
  • Action Context: Model calls include action context for better debugging
  • Event Emission: Model provider failures and successes are emitted as events

Better Fallback Mechanisms

  • Smart Fallbacks: Fallbacks are based on capability scores rather than fixed order
  • Automatic Retry: Built-in retry logic with exponential backoff
  • Health Monitoring: Continuous monitoring of provider health and performance

These improvements make the model provider system more robust, observable, and efficient while maintaining the ease of use that makes elizaOS agents powerful and flexible.

LLM Plugins

Language model plugins and integrations for elizaOS

useModel API

Using the useModel API for AI model interactions in elizaOS