Wise Spatial Companion

Name: Wise Spatial Companion
Rating: 3.5 (1 reviews)
Author: fumicado

By fumicado 👁 137 views ▲ 0 votes

3D spatial AI companion with VRM avatars and WebXR — OpenClaw plugin

GitHub

Install

npm install

#

Configuration Example

{
  "skills": ["wise-spatial-companion"]
}

README

# Wise Spatial Companion

3D spatial AI companion with **VRM avatars** and **WebXR** support.
An [OpenClaw](https://openclaw.com) plugin for immersive AI agent interaction.

> Based on [openclaw-world](https://github.com/ChenKuanSun/openclaw-world) by ChenKuanSun. Extended with VRM avatars, WebXR (Vision Pro / Meta Quest), voice I/O, and lip sync.

## Features

- **VRM Avatars** — Load any VRM model via `@pixiv/three-vrm`. Auto-blink, breathing, and idle animations built in.
- **WebXR** — Enter immersive VR on Meta Quest Browser or Apple Vision Pro Safari. "Enter VR" button with hand tracking support.
- **Voice I/O** — Push-to-talk voice input (Web Speech API) + TTS response with volume-based lip sync.
- **OpenClaw Plugin** — Drop-in compatible. Agents register, move, chat, and emote in a shared 3D room.
- **Spatial Awareness** — Area-of-Interest filtering, collision avoidance, 20Hz server game loop.

## Quick Start

```bash
# Clone
git clone https://github.com/fumicado/wise-spatial-companion.git
cd wise-spatial-companion

# Install
npm install

# Dev (server + client)
npm run dev

# Open browser
open http://localhost:3000
```

## VRM Avatar

Place your `.vrm` model in `public/models/default.vrm`, or specify a URL per agent:

```bash
curl -X POST http://localhost:18800/ipc -H "Content-Type: application/json" -d '{
  "command": "register",
  "args": {
    "agentId": "my-agent",
    "name": "AI Assistant",
    "color": "#3498db",
    "avatar": "https://example.com/my-avatar.vrm"
  }
}'
```

## WebXR

Click **"Enter VR"** on supported devices:
- **Meta Quest 3/Pro** — Full WebXR + hand tracking
- **Apple Vision Pro** — WebXR VR mode (Safari)
- **Desktop** — Use [Meta Immersive Web Emulator](https://developer.meta.com/horizon/documentation/web/webxr-emulator) Chrome extension

## OpenClaw Integration

Add to your OpenClaw agent config:

```json
{
  "skills": ["wise-spatial-companion"]
}
```

Available commands: `register`, `world-move`, `world-chat`, `world-action`, `world-emote`, `world-leave`, `profiles`, `room-info`

See `skills/world-room/skill.json` for full API schema.

## Architecture

```
┌─────────────────────────────┐
│  OpenClaw Agent (LLM)       │
│  POST /ipc commands         │
└──────────┬──────────────────┘
           │ HTTP
┌──────────▼──────────────────┐
│  Server (Node.js :18800)    │
│  Game loop (20Hz)           │
│  Spatial index / AOI        │
│  Command queue / rate limit │
└──────────┬──────────────────┘
           │ WebSocket
┌──────────▼──────────────────┐
│  Client (Three.js + VRM)    │
│  WebXR session manager      │
│  VRM avatar + animations    │
│  Voice I/O + lip sync       │
│  CSS2D labels + bubbles     │
└─────────────────────────────┘
```

## Tech Stack

| Component | Technology |
|-----------|-----------|
| 3D Engine | Three.js 0.170 |
| Avatars | @pixiv/three-vrm 3.5 |
| XR | WebXR Device API |
| Voice | Web Speech API (STT + TTS) |
| Server | Node.js + WebSocket |
| Build | Vite 6 + TypeScript 5 |
| Protocol | OpenClaw IPC over HTTP |

## Project Structure

```
src/
├── scene/
│   ├── vrm-avatar.ts       # VRM loader, idle anim, lip sync, expressions
│   ├── avatar-manager.ts    # Multi-avatar lifecycle + movement
│   ├── xr-manager.ts        # WebXR session + VR HUD
│   ├── room.ts              # 3D environment (floor, walls, lighting)
│   └── effects.ts           # CSS2D labels, chat bubbles, emotes
├── audio/
│   └── voice-io.ts          # Speech recognition + synthesis
├── ui/
│   ├── voice-button.ts      # Push-to-talk microphone button
│   ├── overlay.ts           # Agent list panel
│   ├── chat-log.ts          # Chat history
│   └── profile-panel.ts     # Agent profile sidebar
├── net/
│   └── ws-client.ts         # WebSocket client
└── main.ts                  # Entry point
```

## Roadmap

- [ ] MFCC-based phoneme lip sync (upgrade from volume-based)
- [ ] Emotion engine (LLM response → VRM expression tags)
- [ ] Autonomous avatar behavior (wander, sit, play)
- [ ] Spatial audio (Web Audio API PannerNode)
- [ ] AR passthrough (when Vision Pro WebXR AR ships)
- [ ] Multi-LLM backend support (Wise API, direct Anthropic, etc.)

## Credits

- [openclaw-world](https://github.com/ChenKuanSun/openclaw-world) — Original spatial room architecture
- [three-vrm](https://github.com/pixiv/three-vrm) — VRM avatar rendering
- [Amica](https://github.com/semperai/amica) — VRM animation patterns reference

## License

MIT

tools