Tools
Probe
By OpenClaw Probe plugin: run experiments and measure key metrics
Install
npm install
npmConfiguration Example
{
"probe": {
"name": "baseline",
"mode": "start-stop", // "start-stop" (bracketed by /probe start../stop) or "range"
"generated_at": "2026-08-01T12:03:41.221Z"
},
"window": {
"ts_start": "2026-08-01T12:00:00.000Z",
"ts_end": "2026-08-01T12:03:41.210Z",
"wall_clock_sec": 221.21 // real time between ts_start and ts_end
},
"sessions": {
"session_ids": ["agent:main:main"], // every session with an audit event in the window
"agents_used": { "main": 3 } // completed agent runs per agentId (shows sub-agent participation)
},
"time": {
"agent_active_sec": 41.7, // sum of (run finished - run started) over completed runs
"llm_latency_sec": 33.2, // agent_active_sec - tool_exec_sec, floored at 0 (model wait time, approximated)
"tool_exec_sec": 8.5 // sum of (tool call finished - started)
},
"iterations": {
"agent_runs": 3, // completed top-level agent turns (main + any sub-agent runs)
"llm_calls": 7, // individual model completions, including mid-run tool-calling steps
"tool_calling_rounds": 4, // of those, how many produced at least one tool call
"tool_calls_total": 6 // individual tool invocations
},
"models_used": { "anthropic/claude-x": 7 }, // LLM calls per "provider/model"
"tokens": {
"input": 18422,
"output": 1310,
"cacheRead": 15900,
"cacheWrite": 2100,
"reasoningTokens": 0,
"total": 19732
},
"context": {
"system_prompt_chars_samples": [48211, 48380, 48211], // one sample per completed run
"system_prompt_chars_avg": 48267
},
"tools_used": { "k8s_get_pods": 2, "postgres_query": 1, "read": 3 },
"plugins_used": ["core", "k8s-ops", "postgres-ops"], // owning plugin per tool above ("core" = built-in)
"skills_used": { "aiops-incident": { "name": "aiops-incident", "uses": 1 } }, // {} if none used
"errors": {
"tool_call_errors": {
"count": 1,
"by_tool": { "postgres_query": 1 },
"by_status": { "failed": 1 },
"by_code": { "connection_refused": 1 }
},
"agent_run_errors": { "count": 0, "by_status": {}, "by_code": {} }
},
"llm_api_log": {
"entries_captured": 7,
"file": "/root/.openclaw/state/plugins/probe/results/baseline.rawrequests.jsonl"
},
"events": [
{ "date": "2026-08-01T12:00:02.940Z", "event": "agent run: main (started, still running at window end)" },
{ "date": "2026-08-01T12:00:15.703Z", "event": "LLM call: anthropic/claude-x (with tool call)" },
{ "date": "2026-08-01T12:00:15.900Z", "event": "tool call: read (0.2s)" },
{ "date": "2026-08-01T12:00:22.150Z", "event": "skill used: aiops-incident" },
{ "date": "2026-08-01T12:03:41.100Z", "event": "tool call: postgres_query (failed: connection_refused, 0.3s)" },
{ "date": "2026-08-01T12:03:41.210Z", "event": "agent run: main (41.7s)" }
],
"warnings": []
}README
# Probe
An [OpenClaw](https://openclaw.ai) plugin that measures **agent run cost, speed, and
behavior over a time window** - wall-clock/LLM/tool time, token usage, tool and skill
usage, error counts, and (optionally) a full raw LLM request/response archive.
It exists to answer one question repeatedly and consistently: *did this change (a new
skill, a prompt tweak, a different model, a new tool plugin) make the agent cheaper,
faster, or more/less reliable?* Run the same task with `/probe start` / `/probe stop`
bracketing it, make your change, run it again, and diff the two JSON reports.
Probe is self-contained: it captures its own LLM request/response data and detects its own
skill usage via hooks, and does **not** depend on the `llm-api-logger` or `skill-usage`
plugins, or any other plugin, being installed. Its core numeric metrics (time, tokens,
tool/skill usage, errors) come from OpenClaw's own audit ledger, trajectory files, and
probe's own hook-based detection, and work regardless of what else is installed.
## Install
```bash
openclaw plugins install clawhub:@bocharovf/openclaw-probe
```
or, from a local checkout:
```bash
npm install
npm run build
openclaw plugins install . --force
```
## Requirements
Probe reads from several OpenClaw subsystems (chat commands, the audit ledger, trajectory
files, its own hooks). Each has its own config gate. This section lists what has to be true
on the host for `/probe` to work at all, and separately, what the one genuinely *optional*
piece of the report additionally needs - so a report that is missing a field is easy to
diagnose instead of looking like a bug.
### Required - without these, `/probe` does not run at all
| Setting | Needed because | Symptom if missing/wrong |
| --- | --- | --- |
| `plugins.allow` includes `"probe"` | Plugin must be allowlisted to load. | Plugin never loads; no `/probe` command exists. |
| `plugins.entries.probe.enabled: true` | Explicit enable (set automatically by `openclaw plugins install`). | Same as above. |
| `commands.text` is not `false` (default `true`) | `/...` chat commands are only parsed when text-command parsing is on. | `/probe ...` is treated as a normal chat message and goes to the model instead of the plugin (it will typically reply something like it doesn't recognize the command, or - if you've added the `agentPromptGuidance` this plugin ships - decline to answer). |
| The sender is authorized | The command sets `requireAuth: true` (default for chat commands). Authorization comes from `commands.allowFrom`, or otherwise from channel allowlists/pairing plus `commands.useAccessGroups`. | Unauthorized senders get no response; the command is silently ignored. |
| `openclaw` is reachable in `PATH` for the Gateway process | Probe shells out to `openclaw audit --json` and `openclaw plugins list --json` for every report (see [Data sources](#data-sources)). | `/probe stop` / `/probe <range>` fails with a "Probe command failed" error instead of a report. Point `plugins.entries.probe.config.openclawBin` at an explicit path if `openclaw` isn't on the Gateway's `PATH`. |
### Required for the core metrics (time, tokens, tool/skill usage, errors)
| Setting | Needed because | Symptom if missing/wrong |
| --- | --- | --- |
| `audit.enabled` is not `false` (default `true`) | `time`, `iterations`, `errors`, `sessions`, and `tools_used` all come from the audit ledger - there is no other source for them. | Every report has zeroed-out `time`/`iterations`/`errors`, empty `sessions`/`tools_used`, and `/probe <start> <end>` for that window fails with the "No data found" error (see [Error messages](#error-messages)). `/probe start`/`stop` still "succeeds" but the report is empty. |
| The probed window is inside the audit ledger's retention (**30 days / 100,000 rows**, not configurable) | Older records are pruned; there's nothing to read. | Same "No data found" error for `/probe <start> <end>` on an old range. Always use `/probe start`/`stop` for anything you want reliably measured, and treat `/probe <start> <end>` as best-effort for anything more than a few days old. |
| Trajectory sidecar files for the involved runs still exist on disk (written automatically, no on/off switch - but `session.maintenance` can prune old ones as part of its retention/disk-budget cleanup) | `tokens`, `models_used`, `context`, and `llm_calls`/`tool_calling_rounds` come from each run's `<agent>/sessions/<session>.trajectory.jsonl`. | Those fields undercount or stay at `0` for the affected runs, and the report's `warnings` array names the run id it couldn't find a trajectory for. `time`/`iterations`/`errors`/`tools_used` are unaffected (audit-ledger-only). |
`skills_used` needs no separate config at all - it's a core metric like the rest of this
table, not gated behind an operator opt-in or another plugin. Probe detects skill use itself
by watching `before_tool_call`/`after_tool_call` (which, unlike `llm_input`/`llm_output`
below, are not gated by `hooks.allowConversationAccess`) for a read of a `SKILL.md` file and
recovering the declared skill name from its frontmatter - see [Data sources](#data-sources).
The one inherent limitation: it only recognizes a fixed set of read-tool names (`read`,
`functions.read`, `read_file`, `filesystem.read`, `fs.read`); a skill loaded through some
other, unrecognized read-tool alias would not be counted. This has not been observed in
practice - OpenClaw's core file-read tool is named `read`.
### Optional - raw LLM request/response archive (`llm_api_log`)
| Setting | Needed because | Symptom if missing/wrong |
| --- | --- | --- |
| `plugins.entries.probe.hooks.allowConversationAccess: true` | OpenClaw gates the `llm_input`/`llm_output` hooks behind an explicit opt-in for any plugin that isn't bundled with OpenClaw itself - without it, the host simply never invokes them. | `llm_api_log.entries_captured` is always `0` and `.file` is always `null`. Nothing else in the report is affected. |
| `plugins.entries.probe.config.llmLog.enabled` is not `false` (default `true`) | Plugin-side switch for the same capture. | Same as above. |
Even fully enabled, `llm_api_log.entries_captured` is capped at `sessions.agents_used` (the total
count of completed agent runs), **not** `iterations.llm_calls`: the `llm_input`/`llm_output`
hooks fire once per completed agent run, not once per individual model completion within a
run's tool-calling loop. A run that makes 3 sequential LLM calls while working through a
multi-step tool sequence contributes 3 to `llm_calls` but only 1 possible raw-archive entry.
This is host behavior (confirmed by cross-checking against the independent `llm-api-logger`
plugin, which shows the same ceiling), not something a plugin can widen. The raw archive is
still useful for inspecting a representative prompt/response per run - just don't expect
`entries_captured` to equal `llm_calls`.
```json5
{
"plugins": {
"entries": {
"probe": {
"hooks": {
"allowConversationAccess": true
}
}
}
}
}
```
None of the optional items above affect whether `/probe` runs or whether the core metrics
(time, tokens, tool/skill usage, errors) are correct - they only control whether one
specific field is populated or falls back to `null`/`0`/an empty object, always with a
matching note in the report's `warnings` array.
## Commands
| Command | Description |
| --- | --- |
| `/probe start <name>` | Start a named measurement. Fails if one is already active. |
| `/probe stop` | Stop the active measurement, compute its report, and save it. Fails if none is active. |
| `/probe <start> <end> [name]` | Build (and save) a measurement for a past time range. Timestamps are ISO 8601, e.g. `2026-08-01T00:00:00Z`. `name` is optional and defaults to `"<start> .. <end>"` if omitted. |
| `/probe <name>` | Print a saved measurement's report as JSON. |
| `/probe verbose <name>` | Print a saved measurement's report as an annotated, human-readable text report explaining every field. |
| `/probe list` | List the last 50 saved measurements, newest first, with their name, generation timestamp, and mode. |
| `/probe delete <name>` | Delete a saved measurement (its report and raw-request archive, if any). Fails if the name doesn't exist. |
| `/probe` (no args) | Print this command summary. |
Only **one** `start`ed measurement can be active at a time - `/probe start` while one is
already running is rejected, and you must `/probe stop` (or let it finish) before starting
another.
A probe name cannot be exactly `start`, `stop`, `verbose`, `list`, or `delete` (reserved
words), and cannot be two ISO 8601 timestamps separated by whitespace (that parses as a range
request instead). Names may contain spaces, e.g. `/probe start baseline before cache change`.
The same rules apply to the optional trailing name on `/probe <start> <end> [name]`.
Naming a range measurement is recommended - the auto-generated `"<start> .. <end>"` name
works but is unwieldy to type back exactly (it must match character-for-character, aside
from case) when you later want `/probe <name>` or `/probe verbose <name>`.
### Examples
```text
/probe start baseline
... run the task you want to measure ...
/probe stop
/probe baseline
/probe verbose baseline
/probe 2026-08-01T00:00:00Z 2026-08-01T06:00:00Z incident-postmortem
/probe incident-postmortem
/probe verbose incident-postmortem
/probe list
/probe delete baseline
```
### Error messages
Probe rejects the following with a clear, non-technical message instead of a stack trace or
silent wrong answer:
- Starting a measurement while one is already active.
- Stopping when no measurement is active.
- Showing (JSON or verbose) or deleting a measurement name that does not exist.
- A time range where the start is not strictly before the end.
- A time range whose source data is unavailable - nothing in OpenClaw's audit ledger falls
inside the window (wrong range, a typo, or the window is older than the ledge
... (truncated)
tools
Comments
Sign in to leave a comment