Embedding Setup

Setting up and using embeddings in elizaOS for semantic search and memory management

Embeddings are vector representations of text that capture semantic meaning, enabling powerful features like semantic search, memory retrieval, and content similarity analysis. elizaOS provides comprehensive embedding support through its model provider system.

Overview

Embeddings in elizaOS are used for:

Memory search: Finding relevant memories based on semantic similarity
Knowledge retrieval: Searching through knowledge bases and documents
Content similarity: Comparing and clustering text content
Semantic understanding: Powering intelligent responses and recommendations

Embedding Models

Supported Embedding Providers

elizaOS supports multiple embedding providers through its plugin system:

OpenAI Embeddings

// Installation
npm install @elizaos/plugin-openai

// Configuration
{
  "plugins": ["@elizaos/plugin-openai"]
}

// Environment variables
OPENAI_API_KEY=sk-...


**Available Models:**
- `text-embedding-3-large` (3072 dimensions)
- `text-embedding-3-small` (1536 dimensions)
- `text-embedding-ada-002` (1536 dimensions)

#### Google Embeddings
```typescript
// Installation
npm install @elizaos/plugin-google-genai

// Configuration
{
  "plugins": ["@elizaos/plugin-google-genai"]
}

// Environment variables
GOOGLE_GENERATIVE_AI_API_KEY=AI...


**Available Models:**
- `text-embedding-004` (768 dimensions)
- `text-embedding-gecko` (768 dimensions)

#### Ollama Local Embeddings
```typescript
// Installation
npm install @elizaos/plugin-ollama

// Prerequisites
ollama pull nomic-embed-text

// Configuration
{
  "plugins": ["@elizaos/plugin-ollama"]
}

// Environment variables
OLLAMA_API_ENDPOINT=http://localhost:11434
OLLAMA_EMBEDDING_MODEL=nomic-embed-text


**Available Models:**
- `nomic-embed-text` (768 dimensions)
- `all-minilm` (384 dimensions)
- `bge-large` (1024 dimensions)

## Basic Usage

### Generating Embeddings

```typescript
import { ModelType } from '@elizaos/core';

// Generate embedding for a text
const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: "This is a sample text to embed"
});

console.log(embedding); // [0.123, -0.456, 0.789, ...]
console.log(embedding.length); // 1536 (for OpenAI ada-002)

Alternative Parameter Formats

// Using object parameter
const embedding1 = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: "Sample text",
});

// Using string parameter
const embedding2 = await runtime.useModel(ModelType.TEXT_EMBEDDING, "Sample text");

// Using null (for dimension testing)
const embedding3 = await runtime.useModel(ModelType.TEXT_EMBEDDING, null);

Memory Integration

Adding Embeddings to Memory

elizaOS automatically handles embedding generation for memory storage:

// Create memory with automatic embedding
const memory = {
  content: { text: "User asked about the weather today" },
  entityId: userId,
  roomId: roomId,
  worldId: worldId,
};

// This automatically generates and attaches embeddings
const memoryWithEmbedding = await runtime.addEmbeddingToMemory(memory);

Manual Memory Embedding

// Create memory object
const memory = {
  id: uuidv4(),
  content: { text: "Important conversation about project plans" },
  entityId: userId,
  roomId: roomId,
  worldId: worldId,
};

// Generate embedding
const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: memory.content.text,
});

// Attach embedding to memory
memory.embedding = embedding;

// Store in database
await runtime.createMemory(memory, "messages");

Semantic Search

Searching Memories

// Search for memories similar to a query
const searchQuery = "What did we discuss about the project?";

// Generate embedding for search query
const queryEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: searchQuery,
});

// Search memories using embedding
const similarMemories = await runtime.searchMemories({
  embedding: queryEmbedding,
  match_threshold: 0.8, // Similarity threshold (0-1)
  count: 10, // Number of results
  roomId: roomId, // Optional: limit to specific room
  tableName: "messages", // Memory table to search
});

console.log(similarMemories);

Advanced Search Parameters

const searchResults = await runtime.searchMemories({
  embedding: queryEmbedding,
  match_threshold: 0.7,
  count: 20,
  roomId: roomId,
  unique: true, // Only return unique memories
  worldId: worldId, // Optional: limit to specific world
  entityId: userId, // Optional: limit to specific entity
  tableName: "messages",
});

Embedding Configuration

Model Selection

Choose embedding models based on your needs:

// High-quality embeddings (OpenAI)
const character = {
  name: "MyAgent",
  plugins: ["@elizaos/plugin-openai"],
};

// Environment variables for OpenAI
// OPENAI_API_KEY=sk-...
// OPENAI_EMBEDDING_MODEL=text-embedding-3-large

// Fast local embeddings (Ollama)
const localCharacter = {
  name: "LocalAgent",
  plugins: ["@elizaos/plugin-ollama"],
};

// Environment variables for Ollama
// OLLAMA_API_ENDPOINT=http://localhost:11434
// OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Dimension Management

elizaOS automatically manages embedding dimensions:

// The runtime automatically detects and sets embedding dimensions
// during initialization
await runtime.initialize();

// This ensures database compatibility with your chosen model

Performance Optimization

Embedding Caching

// Cache embeddings to avoid redundant API calls
const embeddingCache = new Map();

async function getCachedEmbedding(text: string) {
  const cacheKey = `embed:${text}`;

  if (embeddingCache.has(cacheKey)) {
    return embeddingCache.get(cacheKey);
  }

  const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: text,
  });

  embeddingCache.set(cacheKey, embedding);
  return embedding;
}

Batch Processing

// Process multiple texts efficiently
async function embedMultipleTexts(texts: string[]) {
  const promises = texts.map((text) => runtime.useModel(ModelType.TEXT_EMBEDDING, { text }));

  return await Promise.all(promises);
}

// Usage
const texts = ["First document to embed", "Second document to embed", "Third document to embed"];

const embeddings = await embedMultipleTexts(texts);

Knowledge Base Integration

RAG (Retrieval-Augmented Generation)

// Create knowledge base with embeddings
const createKnowledgeBase = async (documents: string[]) => {
  const knowledgeItems = [];

  for (const doc of documents) {
    const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
      text: doc,
    });

    knowledgeItems.push({
      id: uuidv4(),
      content: { text: doc },
      embedding: embedding,
      agentId: runtime.agentId,
    });
  }

  // Store in knowledge base
  for (const item of knowledgeItems) {
    await runtime.createKnowledge(item);
  }
};

// Search knowledge base
const searchKnowledge = async (query: string) => {
  const queryEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: query,
  });

  return await runtime.searchKnowledge({
    agentId: runtime.agentId,
    embedding: queryEmbedding,
    match_threshold: 0.7,
    match_count: 5,
  });
};

Document Processing

// Process and embed documents
const processDocument = async (document: string) => {
  // Split document into chunks
  const chunks = splitIntoChunks(document, 1000); // 1000 char chunks

  const embeddedChunks = [];

  for (const chunk of chunks) {
    const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
      text: chunk,
    });

    embeddedChunks.push({
      text: chunk,
      embedding: embedding,
    });
  }

  return embeddedChunks;
};

// Utility function to split text
function splitIntoChunks(text: string, chunkSize: number): string[] {
  const chunks = [];
  for (let i = 0; i < text.length; i += chunkSize) {
    chunks.push(text.slice(i, i + chunkSize));
  }
  return chunks;
}

Similarity Calculations

Cosine Similarity

// Calculate cosine similarity between embeddings
function cosineSimilarity(a: number[], b: number[]): number {
  const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));

  return dotProduct / (magnitudeA * magnitudeB);
}

// Compare two texts
const compareTexts = async (text1: string, text2: string) => {
  const embedding1 = await runtime.useModel(ModelType.TEXT_EMBEDDING, { text: text1 });
  const embedding2 = await runtime.useModel(ModelType.TEXT_EMBEDDING, { text: text2 });

  const similarity = cosineSimilarity(embedding1, embedding2);
  return similarity;
};

Similarity Thresholds

// Common similarity thresholds
const SIMILARITY_THRESHOLDS = {
  VERY_SIMILAR: 0.9, // Nearly identical
  SIMILAR: 0.8, // Clearly related
  SOMEWHAT_SIMILAR: 0.7, // Possibly related
  DIFFERENT: 0.6, // Probably different
  VERY_DIFFERENT: 0.5, // Definitely different
};

// Classify similarity
function classifySimilarity(similarity: number): string {
  if (similarity >= SIMILARITY_THRESHOLDS.VERY_SIMILAR) return "very_similar";
  if (similarity >= SIMILARITY_THRESHOLDS.SIMILAR) return "similar";
  if (similarity >= SIMILARITY_THRESHOLDS.SOMEWHAT_SIMILAR) return "somewhat_similar";
  if (similarity >= SIMILARITY_THRESHOLDS.DIFFERENT) return "different";
  return "very_different";
}

Advanced Features

Embedding Reranking

// Rerank search results using BM25 + embedding hybrid search
const rerankResults = async (query: string, results: any[]) => {
  // First, get embedding-based results
  const queryEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: query,
  });

  const embeddingResults = await runtime.searchMemories({
    embedding: queryEmbedding,
    match_threshold: 0.5,
    count: 50,
    tableName: "messages",
  });

  // Then rerank using BM25 for better relevance
  const rerankedResults = await runtime.rerankMemories(query, embeddingResults);

  return rerankedResults.slice(0, 10); // Top 10 results
};

// Combine text and image embeddings (conceptual)
const createMultiModalEmbedding = async (text: string, imageUrl?: string) => {
  const textEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: text,
  });

  let combinedEmbedding = textEmbedding;

  if (imageUrl) {
    // Image description as proxy for image embedding
    const imageDescription = await runtime.useModel(ModelType.IMAGE_DESCRIPTION, {
      imageUrl: imageUrl,
    });

    const imageTextEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
      text: imageDescription.description,
    });

    // Combine embeddings (weighted average)
    combinedEmbedding = textEmbedding.map((val, i) => val * 0.7 + imageTextEmbedding[i] * 0.3);
  }

  return combinedEmbedding;
};

Best Practices

1. Model Selection

// Choose embedding models based on use case
const configs = {
  // High accuracy, higher cost
  production: {
    provider: "openai",
    model: "text-embedding-3-large",
    dimensions: 3072,
  },

  // Balanced performance
  balanced: {
    provider: "openai",
    model: "text-embedding-3-small",
    dimensions: 1536,
  },

  // Privacy-focused, local
  local: {
    provider: "ollama",
    model: "nomic-embed-text",
    dimensions: 768,
  },
};

2. Text Preprocessing

// Preprocess text for better embeddings
function preprocessText(text: string): string {
  return text
    .toLowerCase() // Normalize case
    .replace(/\s+/g, " ") // Normalize whitespace
    .replace(/[^\w\s]/g, "") // Remove special chars
    .trim(); // Remove leading/trailing spaces
}

// Use preprocessed text for embeddings
const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: preprocessText(inputText),
});

3. Embedding Validation

// Validate embedding quality
function validateEmbedding(embedding: number[]): boolean {
  // Check if embedding is not null or empty
  if (!embedding || embedding.length === 0) {
    return false;
  }

  // Check for NaN values
  if (embedding.some((val) => isNaN(val))) {
    return false;
  }

  // Check magnitude (should not be zero)
  const magnitude = Math.sqrt(embedding.reduce((sum, val) => sum + val * val, 0));
  return magnitude > 0;
}

4. Error Handling

// Robust embedding generation with fallbacks
async function generateEmbeddingWithFallback(text: string) {
  const providers = ["openai", "google", "ollama"];

  for (const provider of providers) {
    try {
      const embedding = await runtime.useModel(
        ModelType.TEXT_EMBEDDING,
        {
          text: text,
        },
        provider
      );

      if (validateEmbedding(embedding)) {
        return embedding;
      }
    } catch (error) {
      console.warn(`Embedding failed with ${provider}:`, error.message);
    }
  }

  throw new Error("All embedding providers failed");
}

Troubleshooting

Common Issues

Embedding dimension mismatch

// Ensure consistent embedding dimensions
const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
  text: "test",
});
console.log("Embedding dimensions:", embedding.length);

Memory search returns no results

// Check similarity threshold
const results = await runtime.searchMemories({
  embedding: queryEmbedding,
  match_threshold: 0.5, // Lower threshold for more results
  count: 10,
  tableName: "messages",
});

Slow embedding generation

// Use caching for repeated text
const cache = new Map();

async function cachedEmbedding(text: string) {
  if (cache.has(text)) {
    return cache.get(text);
  }

  const embedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: text,
  });

  cache.set(text, embedding);
  return embedding;
}

Debug Information

// Debug embedding setup
const debugEmbeddings = async () => {
  try {
    // Test embedding generation
    const testEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
      text: "test",
    });

    console.log("Embedding provider working:", testEmbedding.length > 0);
    console.log("Embedding dimensions:", testEmbedding.length);

    // Test similarity search
    const searchResults = await runtime.searchMemories({
      embedding: testEmbedding,
      match_threshold: 0.5,
      count: 1,
      tableName: "messages",
    });

    console.log("Search working:", searchResults.length >= 0);
  } catch (error) {
    console.error("Embedding debug failed:", error);
  }
};

Migration Guide

From Text-Based to Embedding-Based Search

// Old text-based search
const oldSearch = async (query: string) => {
  return await runtime.getMemories({
    tableName: "messages",
    roomId: roomId,
    count: 10,
  });
};

// New embedding-based search
const newSearch = async (query: string) => {
  const queryEmbedding = await runtime.useModel(ModelType.TEXT_EMBEDDING, {
    text: query,
  });

  return await runtime.searchMemories({
    embedding: queryEmbedding,
    match_threshold: 0.7,
    count: 10,
    roomId: roomId,
    tableName: "messages",
  });
};

Embeddings provide powerful semantic understanding capabilities to elizaOS agents. By properly configuring and utilizing embeddings, you can create more intelligent, context-aware agents that understand the meaning behind text rather than just matching keywords.

Embedding Setup

On this page