Tools
Memory Lancedb Pro
By Enhanced LanceDB memory plugin for OpenClaw โ Hybrid Retrieval (Vector + BM25), Cross-Encoder Rerank, Multi-Scope Isolation, Management CLI
Install
npm install
```Configuration Example
{
"plugins": {
"load": {
"paths": ["plugins/memory-lancedb-pro"]
},
"entries": {
"memory-lancedb-pro": {
"enabled": true,
"config": {
"embedding": {
"apiKey": "${JINA_API_KEY}",
"model": "jina-embeddings-v5-text-small",
"baseURL": "https://api.jina.ai/v1",
"dimensions": 1024,
"taskQuery": "retrieval.query",
"taskPassage": "retrieval.passage",
"normalized": true
}
}
}
},
"slots": {
"memory": "memory-lancedb-pro"
}
}
}README
<div align="center">
# ๐ง memory-lancedb-pro
**Enhanced Long-Term Memory Plugin for [OpenClaw](https://github.com/openclaw/openclaw)**
Hybrid Retrieval (Vector + BM25) ยท Cross-Encoder Rerank ยท Multi-Scope Isolation ยท Management CLI
[](https://github.com/openclaw/openclaw)
[](https://lancedb.com)
[](LICENSE)
**English** | [็ฎไฝไธญๆ](README_CN.md)
</div>
---
## ๐บ Video Tutorial
> **Watch the full walkthrough โ covers installation, configuration, and how hybrid retrieval works under the hood.**
[](https://youtu.be/MtukF1C8epQ)
๐ **https://youtu.be/MtukF1C8epQ**
[](https://www.bilibili.com/video/BV1zUf2BGEgn/)
๐ **https://www.bilibili.com/video/BV1zUf2BGEgn/**
---
## Why This Plugin?
The built-in `memory-lancedb` plugin in OpenClaw provides basic vector search. **memory-lancedb-pro** takes it much further:
| Feature | Built-in `memory-lancedb` | **memory-lancedb-pro** |
|---------|--------------------------|----------------------|
| Vector search | โ
| โ
|
| BM25 full-text search | โ | โ
|
| Hybrid fusion (Vector + BM25) | โ | โ
|
| Cross-encoder rerank (Jina) | โ | โ
|
| Recency boost | โ | โ
|
| Time decay | โ | โ
|
| Length normalization | โ | โ
|
| MMR diversity | โ | โ
|
| Multi-scope isolation | โ | โ
|
| Noise filtering | โ | โ
|
| Adaptive retrieval | โ | โ
|
| Management CLI | โ | โ
|
| Session memory | โ | โ
|
| Task-aware embeddings | โ | โ
|
| Any OpenAI-compatible embedding | Limited | โ
(OpenAI, Gemini, Jina, Ollama, etc.) |
---
## Architecture
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ index.ts (Entry Point) โ
โ Plugin Registration ยท Config Parsing ยท Lifecycle Hooks โ
โโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โ โ โ โ
โโโโโโผโโโโ โโโโโโผโโโโ โโโโโผโโโโโ โโโโผโโโโโโโโโโโ
โ store โ โembedderโ โretrieverโ โ scopes โ
โ .ts โ โ .ts โ โ .ts โ โ .ts โ
โโโโโโโโโโ โโโโโโโโโโ โโโโโโโโโโ โโโโโโโโโโโโโโโ
โ โ
โโโโโโผโโโโ โโโโโโโผโโโโโโโโโโโ
โmigrate โ โnoise-filter.ts โ
โ .ts โ โadaptive- โ
โโโโโโโโโโ โretrieval.ts โ
โโโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโ โโโโโโโโโโโโ
โ tools.ts โ โ cli.ts โ
โ (Agent API) โ โ (CLI) โ
โโโโโโโโโโโโโโโ โโโโโโโโโโโโ
```
### File Reference
| File | Purpose |
|------|---------|
| `index.ts` | Plugin entry point. Registers with OpenClaw Plugin API, parses config, mounts `before_agent_start` (auto-recall), `agent_end` (auto-capture), and `command:new` (session memory) hooks |
| `openclaw.plugin.json` | Plugin metadata + full JSON Schema config declaration (with `uiHints`) |
| `package.json` | NPM package info. Depends on `@lancedb/lancedb`, `openai`, `@sinclair/typebox` |
| `cli.ts` | CLI commands: `memory list/search/stats/delete/delete-bulk/export/import/reembed/migrate` |
| `src/store.ts` | LanceDB storage layer. Table creation / FTS indexing / Vector search / BM25 search / CRUD / bulk delete / stats |
| `src/embedder.ts` | Embedding abstraction. Compatible with any OpenAI-API provider (OpenAI, Gemini, Jina, Ollama, etc.). Supports task-aware embedding (`taskQuery`/`taskPassage`) |
| `src/retriever.ts` | Hybrid retrieval engine. Vector + BM25 โ RRF fusion โ Jina Cross-Encoder Rerank โ Recency Boost โ Importance Weight โ Length Norm โ Time Decay โ Hard Min Score โ Noise Filter โ MMR Diversity |
| `src/scopes.ts` | Multi-scope access control. Supports `global`, `agent:<id>`, `custom:<name>`, `project:<id>`, `user:<id>` |
| `src/tools.ts` | Agent tool definitions: `memory_recall`, `memory_store`, `memory_forget` (core) + `memory_stats`, `memory_list` (management) |
| `src/noise-filter.ts` | Noise filter. Filters out agent refusals, meta-questions, greetings, and low-quality content |
| `src/adaptive-retrieval.ts` | Adaptive retrieval. Determines whether a query needs memory retrieval (skips greetings, slash commands, simple confirmations, emoji) |
| `src/migrate.ts` | Migration tool. Migrates data from the built-in `memory-lancedb` plugin to Pro |
---
## Core Features
### 1. Hybrid Retrieval
```
Query โ embedQuery() โโ
โโโ RRF Fusion โ Rerank โ Recency Boost โ Importance Weight โ Filter
Query โ BM25 FTS โโโโโโ
```
- **Vector Search**: Semantic similarity via LanceDB ANN (cosine distance)
- **BM25 Full-Text Search**: Exact keyword matching via LanceDB FTS index
- **Fusion Strategy**: Vector score as base, BM25 hits get a 15% boost (tuned beyond traditional RRF)
- **Configurable Weights**: `vectorWeight`, `bm25Weight`, `minScore`
### 2. Cross-Encoder Reranking
- **Jina Reranker API**: `jina-reranker-v2-base-multilingual` (5s timeout protection)
- **Hybrid Scoring**: 60% cross-encoder score + 40% original fused score
- **Graceful Degradation**: Falls back to cosine similarity reranking on API failure
### 3. Multi-Stage Scoring Pipeline
| Stage | Formula | Effect |
|-------|---------|--------|
| **Recency Boost** | `exp(-ageDays / halfLife) * weight` | Newer memories score higher (default: 14-day half-life, 0.10 weight) |
| **Importance Weight** | `score *= (0.7 + 0.3 * importance)` | importance=1.0 โ ร1.0, importance=0.5 โ ร0.85 |
| **Length Normalization** | `score *= 1 / (1 + 0.5 * log2(len/anchor))` | Prevents long entries from dominating (anchor: 500 chars) |
| **Time Decay** | `score *= 0.5 + 0.5 * exp(-ageDays / halfLife)` | Old entries gradually lose weight, floor at 0.5ร (60-day half-life) |
| **Hard Min Score** | Discard if `score < threshold` | Removes irrelevant results (default: 0.35) |
| **MMR Diversity** | Cosine similarity > 0.85 โ demoted | Prevents near-duplicate results |
### 4. Multi-Scope Isolation
- **Built-in Scopes**: `global`, `agent:<id>`, `custom:<name>`, `project:<id>`, `user:<id>`
- **Agent-Level Access Control**: Configure per-agent scope access via `scopes.agentAccess`
- **Default Behavior**: Each agent accesses `global` + its own `agent:<id>` scope
### 5. Adaptive Retrieval
- Skips queries that don't need memory (greetings, slash commands, simple confirmations, emoji)
- Forces retrieval for memory-related keywords ("remember", "previously", "last time", etc.)
- CJK-aware thresholds (Chinese: 6 chars vs English: 15 chars)
### 6. Noise Filtering
Filters out low-quality content at both auto-capture and tool-store stages:
- Agent refusal responses ("I don't have any information")
- Meta-questions ("do you remember")
- Greetings ("hi", "hello", "HEARTBEAT")
### 7. Session Memory
- Triggered on `/new` command โ saves previous session summary to LanceDB
- Disabled by default (OpenClaw already has native `.jsonl` session persistence)
- Configurable message count (default: 15)
### 8. Auto-Capture & Auto-Recall
- **Auto-Capture** (`agent_end` hook): Extracts preference/fact/decision/entity from conversations, deduplicates, stores up to 3 per turn
- **Auto-Recall** (`before_agent_start` hook): Injects `<relevant-memories>` context (up to 3 entries)
---
## Installation
1. Clone into your OpenClaw plugins directory:
```bash
cd /path/to/your/openclaw/workspace
git clone https://github.com/win4r/memory-lancedb-pro.git plugins/memory-lancedb-pro
cd plugins/memory-lancedb-pro
npm install
```
2. Add to your OpenClaw config (`openclaw.json`):
```json
{
"plugins": {
"load": {
"paths": ["plugins/memory-lancedb-pro"]
},
"entries": {
"memory-lancedb-pro": {
"enabled": true,
"config": {
"embedding": {
"apiKey": "${JINA_API_KEY}",
"model": "jina-embeddings-v5-text-small",
"baseURL": "https://api.jina.ai/v1",
"dimensions": 1024,
"taskQuery": "retrieval.query",
"taskPassage": "retrieval.passage",
"normalized": true
}
}
}
},
"slots": {
"memory": "memory-lancedb-pro"
}
}
}
```
3. Restart the gateway:
```bash
openclaw gateway restart
```
> **Note:** If you previously used the built-in `memory-lancedb`, disable it when enabling this plugin. Only one memory plugin can be active at a time.
---
## Configuration
<details>
<summary><strong>Full Configuration Example (click to expand)</strong></summary>
```json
{
"embedding": {
"apiKey": "${JINA_API_KEY}",
"model": "jina-embeddings-v5-text-small",
"baseURL": "https://api.jina.ai/v1",
"dimensions": 1024,
"taskQuery": "retrieval.query",
"taskPassage": "retrieval.passage",
"normalized": true
},
"dbPath": "~/.openclaw/memory/lancedb-pro",
"autoCapture": true,
"autoRecall": true,
"retrieval": {
"mode": "hybrid",
"vectorWeight": 0.7,
"bm25Weight": 0.3,
"minScore": 0.3,
"rerank": "cross-encoder",
"rerankApiKey": "jina_xxx",
"rerankModel": "jina-reranker-v2-base-multilingual",
"candidatePoolSize": 20,
"recencyHalfLifeDays": 14,
"recencyWeight": 0.1,
"filterNoise": true,
"lengthNormAnchor": 500,
"hardMinScore": 0.35,
"timeDecayHalfLifeDays": 60
},
"enableManagementTools": false,
"scopes": {
"default": "global",
"definitions": {
"global": { "description": "Shared knowledge" },
"agent:discord-bot": { "description": "Discord bot private" }
},
"agentAccess": {
"discord-bot": ["global", "agent:discord-bot"]
}
},
"sessionMemory": {
"enabled": false,
"messageCount": 15
}
}
```
</details>
### Embedding Providers
This plugin works with **any OpenAI-compati
... (truncated)
tools
Comments
Sign in to leave a comment