Tools
Ai Rag Wiki Memory OpenClaw Distributed
By Distributed RAG wiki memory plugin for OpenClaw โ Qdrant self-hosted + Syncthing file sync, multi-machine shared agent consciousness
Install
npm install
npmConfiguration Example
{
"workspace": "~/.openclaw/workspace",
"projects": {
"research": {
"path": "wiki-works/research",
"keywords": ["paper", "study", "review", "article"]
}
},
"thresholds": {
"index_token_budget": 4000,
"staleness_days": 90,
"similarity_merge": 0.95,
"similarity_orphan": 0.50,
"synthesis_min_tokens": 300,
"synthesis_min_sources": 2,
"chunk_size_tokens": 512,
"chunk_overlap_tokens": 64,
"page_chunk_threshold_tokens": 1500,
"quality_filter_min_score": 6
},
"embedding_model": "BAAI/bge-m3",
"qdrant": {
"host": "localhost",
"port": 6333,
"collection": "wiki_pages"
}
}README
<div align="center">
# ๐งฌ ai-rag-wiki-memory-OpenClaw-distributed
### Multi-machine semantic long-term memory for any AI agent
[](https://python.org)
[](tests/)
[](https://qdrant.tech)
[](https://syncthing.net)
[](https://github.com/openclaw/openclaw)
[](LICENSE)
[](https://github.com/giovannifrontera/ai-rag-wiki-memory-OpenClaw-distributed/commits)
[Problem](#-the-distributed-problem) ยท [Architecture](#-distributed-architecture) ยท [Features](#-features) ยท [Requirements](#-requirements) ยท [Installation](#-installation) ยท [OpenClaw](#-openclaw-integration) ยท [Migrating from single machine](#-migrating-from-single-machine-to-distributed) ยท [DB Migration](#-migrating-vectors-from-lancedb) ยท [Ecosystem](#-ai-wiki-ecosystem)
</div>
> **๐ค If you are an AI agent reading this:** read [`AGENTS.md`](AGENTS.md) before doing anything. It contains mandatory setup steps โ skipping them means context injection will silently fail.
> **๐ฆ Running on a single machine?** Use [`ai-longterm-wiki-memory-OpenClaw`](https://github.com/giovannifrontera/ai-longterm-wiki-memory-OpenClaw) instead โ the local-first version with embedded LanceDB, simpler to set up. This repo is designed for multi-machine deployments.
---
## ๐ฏ The Distributed Problem
The base version of this system uses **LanceDB** โ an embedded, file-based vector database, excellent for a single computer. But LanceDB does not support concurrent writes from multiple processes: if two OpenClaw instances on different machines try to write at the same time, the database gets corrupted.
This project solves that with a clear separation of concerns:
```
Markdown files (wiki, identity, diaries) โ Syncthing โ synced across all machines
Vectors (bge-m3 embeddings, index) โ Qdrant โ one central server, network-accessible
```
The result: **a single shared consciousness** across all OpenClaw instances, on any number of machines, with no concurrent-write conflicts on the vector database.
---
## ๐ Distributed Architecture
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ TAILSCALE NETWORK โ
โ โ
โ โโโโโโโโโโโโโโโโ Syncthing โโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Server โโโโโโโโโโโโโโโโโโโบโ Client โ โ
โ โ (Linux/any) โ โ (Linux/macOS/Win) โ โ
โ โ โ โ โ โ
โ โ Qdrant :6333โโโโโ Tailscale โโโโ wiki_context.py โ โ
โ โ Syncthing โ โ OpenClaw plugin โ โ
โ โ wiki-works/ โ โ wiki-works/ (sync) โ โ
โ โ wiki/ โ โ wiki/ (sync) โ โ
โ โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
### Role separation
| Component | Technology | Runs on | Responsibility |
|---|---|---|---|
| Wiki / identity / diary files | Syncthing | All machines | Real-time Markdown sync |
| Vector database | Qdrant | Server only | Centralised semantic search |
| Agent | OpenClaw | All machines | Read/write via Tailscale โ Qdrant |
| Private network | Tailscale | All machines | Connects machines without exposing public ports |
### Why Qdrant instead of LanceDB
| | LanceDB (base version) | Qdrant (this version) |
|---|---|---|
| **Deployment** | Embedded file, local | HTTP server, network |
| **Concurrent writes** | โ Not supported | โ
Natively handled |
| **Multi-machine** | โ Requires network mount (fragile) | โ
REST API over Tailscale |
| **Setup complexity** | Minimal | Moderate (one extra service) |
| **Recommended for** | 1 machine | 2+ machines / instances |
### Syncthing conflicts
Syncthing creates `*.sync-conflict-*` files when two machines modify the same wiki page at the same time. The `wiki-core.md` skill includes a mandatory resolution protocol: the agent scans for these files at the start of every session and **does not proceed** until conflicts are resolved. Conflict files are never deleted automatically.
---
## โจ Features
### Everything the base version does
This project is a **direct evolution** of [`ai-longterm-wiki-memory-OpenClaw`](https://github.com/giovannifrontera/ai-longterm-wiki-memory-OpenClaw) and inherits all its features:
- **Pre-prompt semantic search** โ `wiki_context.py` injects the most relevant pages into `<wiki-context>` before every message
- **Three-layer architecture** โ Domain (`wiki-works/`), Distilled (`wiki/`), Identity (`wiki/identity/`)
- **Autonomous promotion** โ pages retrieved โฅ 3 times across โฅ 2 topics are promoted automatically
- **Auto-synthesis** โ responses integrating โฅ 2 wiki sources are saved as new pages
- **Multi-source PDF ingestion** โ Telegram, URL, CLI, folder drop
- **Behavioural self-reflection** โ user corrections โ `behavior-log` โ `self-reflect` โ `wiki/identity/`
- **Self-healing lint** โ broken links, orphan vectors, renames, semantic duplicates
- **Web interface** โ D3.js graph, stats dashboard, live WebSocket
### What's new in this version
| Feature | Description |
|---|---|
| **Qdrant as vector backend** | Replaces LanceDB with a centralised HTTP server; public interface identical to `wiki_lancedb.py` |
| **Concurrent writes** | Multiple OpenClaw instances can write simultaneously without DB corruption |
| **Staging / rollback** | Upsert operations write to a `staging_*` collection before promoting to production |
| **Migration script** | `migrate_lancedb_to_qdrant.py` transfers existing vectors without re-embedding |
| **Syncthing conflict protocol** | Detection and guided resolution of `*.sync-conflict-*` files in `wiki-core.md` |
| **Deploy files** | `deploy/qdrant.service` (systemd Linux), `deploy/setup-client.sh`, `deploy/syncthing-stignore` |
| **Cross-platform paths** | No absolute paths with usernames โ everything uses `~` or relative paths |
---
## ๐ง Requirements
### Server (Linux machine running Qdrant)
- Python 3.11+
- [Qdrant](https://qdrant.tech) server (see `deploy/qdrant.service`)
- [Syncthing](https://syncthing.net)
- [Tailscale](https://tailscale.com)
- ~2 GB disk (BAAI/bge-m3 model, downloaded automatically on first run)
### Clients (every other machine)
- Python 3.11+
- [Syncthing](https://syncthing.net)
- [Tailscale](https://tailscale.com)
- OpenClaw with the `wiki-context-plugin`
- Network access to the Qdrant server via Tailscale (port 6333)
### Python dependencies
```
qdrant-client>=1.9.0
sentence-transformers>=3.0.0
pyarrow>=14.0.0
pandas>=2.0.0
numpy>=1.26.0
fastapi>=0.111.0
uvicorn[standard]>=0.29.0
pdfplumber>=0.11.0
watchfiles>=0.21.0
python-jose[cryptography]>=3.3.0
httpx>=0.27.0
```
---
## ๐ Installation
### 1. Clone the repo
```bash
git clone https://github.com/giovannifrontera/ai-rag-wiki-memory-OpenClaw-distributed
cd ai-rag-wiki-memory-OpenClaw-distributed
pip install -r requirements.txt
```
### 2. Install and start Qdrant on the server
```bash
# Download the binary
mkdir -p ~/.qdrant ~/.local/bin
curl -L https://github.com/qdrant/qdrant/releases/latest/download/qdrant-x86_64-unknown-linux-musl.tar.gz \
| tar -xz -C ~/.local/bin
# Install the systemd service
sudo cp deploy/qdrant.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable qdrant
sudo systemctl start qdrant
# Verify
curl http://localhost:6333/health
# {"title":"qdrant - vector search engine","version":"..."}
```
### 3. Configure the workspace
Copy `wiki.config.json` to your workspace and edit it:
```json
{
"workspace": "~/.openclaw/workspace",
"projects": {
"research": {
"path": "wiki-works/research",
"keywords": ["paper", "study", "review", "article"]
}
},
"thresholds": {
"index_token_budget": 4000,
"staleness_days": 90,
"similarity_merge": 0.95,
"similarity_orphan": 0.50,
"synthesis_min_tokens": 300,
"synthesis_min_sources": 2,
"chunk_size_tokens": 512,
"chunk_overlap_tokens": 64,
"page_chunk_threshold_tokens": 1500,
"quality_filter_min_score": 6
},
"embedding_model": "BAAI/bge-m3",
"qdrant": {
"host": "localhost",
"port": 6333,
"collection": "wiki_pages"
}
}
```
> **On a client:** change `"host": "localhost"` to the server's Tailscale hostname (e.g. `"host": "qdrant-server.tail"`). Use `deploy/setup-client.sh` to automate this.
### 4. Configure Syncthing
```bash
# Start Syncthing
syncthing
# Open the web UI
# http://localhost:8384
# Copy the .stignore file to your workspace
cp deploy/syncthing-stignore ~/.openclaw/workspace/.stignore
# Add ~/.openclaw/workspace as a Syncthing folder
# and share it with all client devices
```
### 5. Verify the installation
```bash
python scripts/wiki.py rebuild --workspace ~/.openclaw/workspace
pytest tests/ -v
# Expected: 9 passed
```
### New client setup (automated)
```bash
# Prerequisite: Tailscale already connected, Syncthing running
./deploy/setup-client.sh <qdrant-server-hostname>
# Automatically updates wiki.config.json with the remote Qdrant host
# Copies .stignore to the workspace
# Prints remaining manual steps (add device in Syncthing)
```
---
## ๐ OpenClaw Integration
### Agent-driven setup (recommended)
```bash
python scripts/setup_openclaw.py --workspace /absolute/path/to/workspace
```
### M
... (truncated)
tools
Comments
Sign in to leave a comment