Tools

Memory Lancedb Pro V4

Name: Memory Lancedb Pro V4
Rating: 3.5 (1 reviews)
Author: canxia-hub

By canxia-hub 👁 17 views ▲ 0 votes

Capability-first LanceDB memory plugin for OpenClaw — hybrid retrieval (FTS+Vector+Lexical), Wiki knowledge graph + vector search, 17 tools, dreaming engine, incremental build

Homepage GitHub

Install

npm install

#

Configuration Example

{
  "plugins": {
    "entries": {
      "memory-lancedb-pro": {
        "config": {
          "dbPath": "path/to/lancedb",
          "tableName": "memories",
          "embeddingDimension": 2560,
          "connectionMode": "embedded",
          "retrieval": {
            "hybrid": true,
            "rerank": false
          },
          "vault": {
            "path": "path/to/wiki"
          },
          "context": {
            "includeCompiledDigestPrompt": false  // default: false
          },
          "hostInterop": {
            "enableArtifacts": true,
            "enableEvents": true
          },
          "dreaming": {
            "enabled": false  // default: false
          }
        }
      }
    }
  }
}

README

# Memory LanceDB Pro v4

Capability-first LanceDB memory plugin for OpenClaw — hybrid retrieval, Wiki knowledge graph with vector search, dreaming engine, and incremental build.

**Production-proven**: 767+ memories, 142 wiki pages vector-indexed, 4350 graph nodes / 5435 edges, 218/218 vitest passing.

## Quick Facts

| Item | Detail |
|------|--------|
| Version | 4.0.0 |
| OpenClaw | >=2026.5.6 |
| Tools | 18 (10 memory + 8 wiki) |
| Tests | 218 (9 test files) |
| DB | LanceDB 0.33 embedded |
| Embedding | 2560-dim (DashScope tongyi-embedding-vision-flash) |
| Tables | `memories` + `wiki_pages` (fully isolated) |
| Retrieval | Hybrid: FTS (BM25) + Vector (cosine) + Lexical |
| Wiki | Graph + Vector Search + Incremental Build |

---

## Architecture

```
dist/
  index.js                  ← Plugin entry (definePluginEntry)
  config/                   ← Config resolve + schema + backend config
  store/
    lancedb-store.js        ← LanceDB 0.33 store (BITMAP + FTS indexes)
    asset-store.js          ← Multimodal asset indexing
    migrations.js           ← Schema migration manager
    scope-manager.js        ← Memory scope normalization
  retrieval/
    embedder.js             ← DashScope multimodal embedding (2560-dim)
    hybrid-retriever.js     ← FTS + Vector + Lexical three-way fusion
    search-manager.js       ← Public search interface (MemorySearchResult)
    rerank.js               ← External rerank provider (optional)
  tools/
    register.js             ← Tool registration + context init
    store.js                ← memory_store
    recall.js               ← memory_recall
    list.js                 ← memory_list
    promote.js              ← memory_promote
    update.js               ← memory_update
    archive.js              ← memory_archive
    diagnostics.js          ← memory_stats + memory_debug
    wiki-tools.js           ← 8 wiki tools registration
  wiki/
    wiki-store.js           ← Wiki vault CRUD
    wiki-graph.js           ← Graph build (full + incremental) + query
    wiki-extractor.js       ← Markdown content extractor
    wiki-index.js           ← Category index builder
    wiki-doctor.js          ← Vault health checker
    wiki-sync-links.js      ← Backlink synchronizer
    wiki-supplement.js      ← Corpus + prompt supplements (3-way fusion)
    digest-compiler.js      ← Compiled digest generator
    wiki-vector-index.js    ← P1: Vector index for wiki pages
    build-manifest.js       ← P2: File fingerprint tracking
    obsidian-cli.js         ← Obsidian CLI probe (optional)
  capture/
    sanitization.js         ← Envelope sludge detection
    policy.js               ← Recall query normalization
    prompt-defense.js       ← Prompt injection detection + escape
  dreaming/
    engine.js               ← Dreaming 3-phase sweep engine
    config.js               ← Dreaming config + source aliases
  hooks/
    auto-memory.js          ← Auto-recall hooks (agent_end)
  interop/
    memory-capability.js    ← Memory capability runtime
    public-artifacts.js     ← Public artifacts provider
    host-events.js          ← Host events manager
  doctor/
    cli.js                  ← Doctor CLI (schema validation + cleanup)
  state/
    plugin-state.js         ← openKeyedStore persistence
  migration/
    index.js                ← Legacy v0→v2 migration
```

---

## Tools (18)

### Memory Tools (10)

| Tool | Description |
|------|-------------|
| `memory_store` | Store a memory with embedding |
| `memory_recall` | Hybrid search (FTS + Vector + Lexical) |
| `memory_list` | List memories with filters |
| `memory_update` | Update an existing memory |
| `memory_archive` | Archive (soft-delete) a memory |
| `memory_promote` | Promote memory to durable/core |
| `memory_stats` | Memory statistics |
| `memory_debug` | Retrieval pipeline debug trace |
| `memory_migrate_legacy` | Migrate from legacy database |
| `memory_forget` | Permanently delete a memory |

### Wiki Tools (8)

| Tool | Description |
|------|-------------|
| `wiki_status` | Vault mode, health, Obsidian CLI availability |
| `wiki_new` | Create new wiki entry with front matter |
| `wiki_get` | Read wiki entry by path or lookup term |
| `wiki_query` | Query knowledge graph by keyword |
| `wiki_build` | Build graph (incremental by default, `force:true` for full) |
| `wiki_doctor` | Lint vault: broken links, stale graph, health |
| `wiki_index` | Rebuild category indexes + main INDEX.md |
| `wiki_sync_links` | Synchronize backlinks across all entries |

---

## Feature Matrix

### Enabled by Default

| Feature | Description |
|---------|-------------|
| **Hybrid Retrieval** | Three-way fusion: FTS (BM25, icu tokenizer) + Vector (cosine, native LanceDB) + Lexical (fallback). Sigmoid normalization for BM25 scores. |
| **Wiki Vector Search** | Independent `wiki_pages` table with full-page embedding. Three-way fusion: vector x0.5 + keyword x0.3 + graph x0.2. Graceful degradation when embedding unavailable. |
| **Wiki Incremental Build** | File fingerprint tracking (`build-manifest.json`, mtime + contentHash). Only re-extracts changed files. Dangling edge cleanup. No-change detection skips rebuild. |
| **Wiki Corpus Supplement** | Wiki search integrated into `memory_recall` pipeline via corpus supplement interface. |
| **autoRecall** | Automatic memory recall on agent turn end, with prompt injection defense and envelope sludge filtering. |
| **access_count** | Recall access tracking with write-back. Feeds dreaming deep promotion quality scoring. |
| **Dreaming Sweep** | 3-phase: light archive + deep promote + REM create. Scheduled via cron. |
| **Doctor CLI** | Schema validation + contamination cleanup (`openclaw doctor`). |
| **Host Interop** | Public artifacts provider + host events manager. |
| **Plugin State** | openKeyedStore persistence (fire-and-forget, honest degradation). |
| **Legacy Migration** | Automatic v0→v2 schema upgrade with skip-existing idempotency. |

### Disabled by Default

| Feature | Description | Reason |
|---------|-------------|--------|
| **includeCompiledDigestPrompt** | Compiled wiki digest injected into prompt | Offline assessment pending. Enable via config `context.includeCompiledDigestPrompt: true`. |
| **Semantic Edge Inference** | LLM-powered semantic edges in wiki graph | Not implemented in TypeScript (graceful degradation). Use Python wiki_ops.py for semantic builds. |
| **Dreaming Engine** | Background dreaming process | Engine created but not started. Enable via config `dreaming.enabled: true`. |
| **Rerank** | External rerank provider | Requires additional config (`retrieval.rerankProvider`, `retrieval.rerankModel`, `retrieval.rerankBaseUrl`). |
| **Obsidian CLI** | Obsidian vault integration | Probe available but Obsidian not installed on this host. Auto-detects if installed. |

### Removed in v4

| Feature | Reason |
|---------|--------|
| **autoCapture / customTriggers** | Permanently removed per user decision. Redundant with agent-initiated `memory_store` / `/remember`. Capture library modules retained for recall path (policy/sanitization/prompt-defense). |

---

## Retrieval Pipeline

```
memory_recall("query")
  ├── FTS Search (BM25, icu tokenizer, sigmoid normalization)
  ├── Vector Search (cosine, native LanceDB vectorSearch)
  ├── Lexical Search (fallback, term frequency + exact match bonus)
  ├── Three-way Fusion (configurable weights)
  ├── Rerank (optional, external provider)
  └── Wiki Corpus Supplement
        ├── Vector Search (wiki_pages table, cosine)
        ├── Keyword Match (title + body)
        ├── Graph Query (graph.json node matching)
        └── Fusion: vector×0.5 + keyword×0.3 + graph×0.2
```

## Wiki Build Pipeline

```
wiki_build()                    ← Incremental by default
  ├── Load build-manifest.json
  ├── Scan vault fingerprints (mtime + contentHash)
  ├── Detect changes (added / modified / deleted / unchanged)
  ├── If no changes → skip rebuild
  ├── If changes:
  │     ├── Re-extract changed files only
  │     ├── Remove deleted/modified file nodes/edges
  │     ├── Merge with unchanged graph
  │     ├── Clean dangling edges
  │     ├── Deduplicate nodes
  │     ├── Detect communities
  │     ├── Export graph.json + GRAPH_REPORT.md + graph.html
  │     ├── Update digest
  │     └── Update vector index (wiki_pages)
  └── Save manifest

wiki_build({ force: true })     ← Full rebuild
  ├── Extract all files
  ├── Build graph from scratch
  ├── Clean dangling edges
  ├── Detect communities
  ├── Export all outputs
  ├── Compile digest
  ├── Rebuild vector index
  └── Save manifest
```

## Storage

| Table | Purpose | Schema |
|-------|---------|--------|
| `memories` | User memories | id, scope, content, embedding[2560], category, importance, createdAt, updatedAt, metadata |
| `wiki_pages` | Wiki vector index | id, path, title, content, embedding[2560], category, tags, updatedAt, metadata |
| `memory_assets` | Multimodal assets | id, memoryId, modality, mimeType, storagePath, ... |

**Indexes**: BITMAP on `scope`/`category` (memories), BITMAP on `category` (wiki_pages), FTS on `content` (memories, icu tokenizer).

## Development

```bash
# Install
npm install

# Type check
npm run typecheck

# Test (218 tests)
npm test

# Integration tests
node scripts/test-wiki-vector-index.mjs
node scripts/test-wiki-incremental-build.mjs
```

## Configuration

```jsonc
{
  "plugins": {
    "entries": {
      "memory-lancedb-pro": {
        "config": {
          "dbPath": "path/to/lancedb",
          "tableName": "memories",
          "embeddingDimension": 2560,
          "connectionMode": "embedded",
          "retrieval": {
            "hybrid": true,
            "rerank": false
          },
          "vault": {
            "path": "path/to/wiki"
          },
          "context": {
            "includeCompiledDigestPrompt": false  // default: false
          },
          "hostInterop": {
            "enableArtifacts": true,
            "enableEvents": true
          },
          "dreami

... (truncated)

tools