Voice
Llm Verifier
By OpenClaw plugin: call-chain verification with logprob-based scoring, per-call tracking, and cached re-scoring
Install
npm install
pnpmREADME
# OpenClaw LLM Verifier
An OpenClaw plugin that provides call-chain verification with logprob-based scoring, per-call tracking, and cached re-scoring. Scores trajectory pairs on criteria via Gemini 2.5 Flash logprobs, runs round-robin tournament selection.
Originally a Python research framework achieving state-of-the-art on Terminal-Bench 2 (86.4%) and SWE-Bench Verified (77.8%), now ported to TypeScript as a native OpenClaw plugin.
## Prerequisites
- Node.js 22.19+ / Node 23.11+ / Node 24+
- pnpm
- A Gemini or Vertex AI API key (for logprob extraction)
## Quick Start
```bash
cd shim
pnpm install
pnpm run build # tsc -> dist/
pnpm test # vitest (14 tests, no live Gemini)
```
Set your API key:
```bash
export GEMINI_API_KEY=your_key_here
# or
export VERTEX_API_KEY=your_key_here
```
## Plugin Tools
| Tool | Description |
|---|---|
| `score` | Score one trajectory pair on one criterion; returns normalized scores in [0,1] from Gemini logprobs. |
| `select` | Run a round-robin tournament over N trials; returns the winner with the most wins. |
Completed scoring calls are cached by `(criterion, task, pair, rep)`; re-runs serve from cache with zero fresh calls.
## Install in OpenClaw
```bash
# Build and pack
cd shim
pnpm run build
npm pack --pack-destination /tmp
# Install locally for testing
openclaw plugins install npm-pack:/tmp/openclaw-llm-verifier-0.1.0.tgz --force
# Inspect runtime
openclaw plugins inspect openclaw-llm-verifier --runtime --json
# Publish to ClawHub
clawhub package publish ninetyhe-90/openclaw-llm-verifier --dry-run
clawhub package publish ninetyhe-90/openclaw-llm-verifier
```
## Project Structure
```
shim/ # OpenClaw TypeScript plugin
package.json # type:module, peerDependencies.openclaw, openclaw.extensions/compat/build
openclaw.plugin.json # manifest: id/contracts.tools[score,select]/activation/configSchema
index.ts # definePluginEntry + api.registerTool(score, select)
verifier.ts # TS port of verifier_core.py (Gemini + logprob + scoring + tournament)
*.test.ts # vitest test suite (14 tests)
tsconfig.json # ESM compilation config
vitest.config.ts # test runner config
data/ # Benchmark trajectory data (Terminal-Bench + SWE-bench)
specs/ # Feature specification documents
```
## How it works
Rather than reducing each distribution into a single discrete score (as in LLM-as-a-Judge), **LLM-as-a-Verifier** approximates the reward of a trajectory $\tau$ on task $t$ as:
$$
R(t, \tau)
= \frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K}
\sum_{g=1}^{G} p_{\theta}(v_g \mid t, c, \tau)\,\phi(v_g)
$$
**Where:**
- $C$ = number of evaluation criteria
- $K$ = number of repeated verifications
- $G$ = number of score tokens (granularity level, g=20: A-T)
- $p_{\theta}(v_g \mid t, c, \tau)$ = probability assigned by model $\theta$ to score token $v_g$
- $\phi(v_g)$ = maps each scoring token to a scalar value
To pick the best trajectory among $N$ candidates, a round-robin tournament is run. For every pair $(i, j)$ the verifier produces $R(t, \tau_i)$ and $R(t, \tau_j)$. The trajectory with the higher reward receives a win; the trajectory with the most wins across all $\binom{N}{2}$ pairs is selected.
## Constitution
This plugin follows constitution v3.0.0 (TypeScript Native + OpenClaw Plugin SDK). See `.specify/memory/constitution.md` for the full governance document.
voice
Comments
Sign in to leave a comment