Tools
Pickysteve
By Skill router and context picker for Claude Code, Cursor, and Aider. Auto-picks the right skill for every prompt, with prompt-injection filtering for MCP.
Install
pip install --pythonREADME
<p align="center">
<img src="assets/img/social-banner.jpg" alt="PickySteve — picks the right skill for your coding agent" width="820">
</p>
<p align="center">
<em>Picky about what he loads into context, including what he refuses to load.</em>
</p>
### ▶ Watch the trailer
https://github.com/user-attachments/assets/8750946b-36be-4c48-bf73-79513451d1f5
<p align="center">
<img src="https://img.shields.io/badge/license-MIT-blue" alt="license: MIT">
<img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="python 3.11+">
<img src="https://github.com/KernelLord/pickysteve/actions/workflows/ci.yml/badge.svg" alt="CI">
</p>
PickySteve is a lightweight orchestration layer. A cheap model figures out which skill a request
actually needs, retrieves that one skill, and hands a small, focused, untrusted-data-boundaried
context bundle to a capable model. It does not dump every tool and document you own into context
on every request.
This repo is Phase 1 (MVP), built to an architecture spec. Phase 2 work (tracing platform,
standing eval harness, credential vault, sandbox) is not built yet. Each piece gets added only
when a real Phase 1 failure justifies it.
## 30-second quickstart
```bash
# from the repo root (Windows; uv 0.10+)
uv venv --python 3.11 .venv
uv pip install --python .venv/Scripts/python.exe -r requirements.txt
# choose your model — local Ollama, OpenAI, Claude, OpenRouter, or any OpenAI-compatible endpoint
.venv/Scripts/python.exe -m pickysteve.setup
# calibrate the reranker floor on the labeled set
.venv/Scripts/python.exe eval/calibrate.py
# run one request
.venv/Scripts/python.exe -m pickysteve "review my Rust endpoint for security and REST design"
```
**Bring your own model.** `python -m pickysteve.setup` asks which model to use and saves it.
Runs on anything that speaks the OpenAI-compatible API: local **Ollama** (offline, no key),
**OpenAI**, **Claude**, **Gemini**, **Llama**, etc. via OpenRouter / LiteLLM / their native
compat endpoints. The published benchmarks were measured on local `qwen3:8b`; a different model
just needs a `eval/calibrate.py` re-run.
> **Note:** this is currently a `uv` / `git clone` install. There is no PyPI package yet, so
> `uvx pickysteve` and `pipx install pickysteve` do not exist. If that changes, this section
> gets a one-liner. For now, the fastest path to a real coding agent is the connector installer
> below.
### Wire it into your agent (one command)
```bash
python -m pickysteve.connectors.install --list # see which of 18 agents are detected
python -m pickysteve.connectors.install --all # wire every detected agent (backs up configs first)
```
Supports **Claude Code, Codex, Cursor, Windsurf, Cline, Roo Code, Gemini CLI, Qwen Code, Goose,
OpenHands, GitHub Copilot, Kimi Code, OpenCode, ZeroClaw** via MCP stdio, and **Aider, Hermes,
OpenClaw, NanoClaw** via an OpenAI-compatible proxy on `:8077/v1`. Full per-agent config snippets
and the connectivity matrix are in [`INTEGRATIONS.md`](INTEGRATIONS.md).
## How it works
```mermaid
flowchart TD
A[Request] --> B[Security Gate\nscan raw request]
B -->|clean| C[Router\ncheap model → search query]
B -->|injection| X1[Abort]
C --> D[Retrieval\nBM25 + embeddings, RRF fused]
D --> E[Security Gate\nscan every retrieved doc]
E -->|clean| F[Rerank\ncross-encoder vs original request]
E -->|poisoned| X2[Abort / drop candidate]
F --> G[Floor + Dedupe\nbelow floor → clarify, don't guess]
G --> H[Knowledge Graph\nconfused_with edges + distinguishers]
H --> I[Judge\nLLM reads full skill bodies + KG notes]
I --> J[Compat Check\nflag conflicts, don't merge]
J --> K[Assembly\nnonce-wrapped untrusted-data boundary]
K --> L[Execution\ncapable model does the work]
L --> M[Log\nfull trace to logs/runs.jsonl]
```
Ten stages: gate, route, retrieve, gate again on retrieved content, rerank, floor/dedupe,
knowledge-graph context, judge, compat-check, assemble, execute, log. The second gate pass scans
every retrieved candidate, not just the user's request. Most similar projects skip that pass, and
it is the highest-risk surface: a poisoned skill doc is attacker-controlled content sitting right
next to your execution model.
## The stack (and why)
| Role | Choice | Note |
|---|---|---|
| Runtime | **Python 3.11** via `uv` | The default Python here is 3.14, which still has shaky `torch` wheels. `uv` pins an isolated 3.11 venv where the ML stack is stable. |
| Security gate | **`stackone-defender[onnx]`** | The real StackOne defender (Python port, v0.7.2), not a regex placeholder. Bundled ~22MB ONNX classifier, no download. |
| Router / compat / clarify / execution | local Ollama `qwen3:8b` via the **native `/api/chat` (`think:false`)** | Runs with no cloud key. The OpenAI-compat endpoint does not honor thinking control for qwen3 (it dumps output into a `reasoning` channel and leaves `content` empty, roughly 20x slower), so the client uses the native endpoint by default. Set `PS_OLLAMA_NATIVE=0` / `PS_LLM_BASE_URL` for any OpenAI-compatible host. |
| Retrieval | **`rank_bm25`** + **`sentence-transformers`** embeddings, fused with RRF | Hybrid keyword + dense. |
| Reranker | **`BAAI/bge-reranker-base`** cross-encoder | Exactly the model the spec names. Its output is a logit, not a probability, so the floor is calibrated rather than guessed. |
| Logging | flat **JSONL** | Manual review is the Phase-1 eval process. |
Total Phase-1 dependencies: `stackone-defender`, `rank-bm25`, `sentence-transformers`,
`openai`, `numpy`. That is the minimal set the spec prescribes.
## Two decisions the spec left open (decided and documented)
- **Retrieval unit (§2.3):** each markdown file is one retrieval unit. A skill folder with
several files (see `registry/rag-architecture/`) yields multiple units sharing a `skill_id`.
After reranking, units from the same skill collapse to the best one in assembly, so the
execution model never receives three chunks of one skill.
- **Gate policy on a poisoned retrieval (§2.1):** default `RETRIEVED_INJECTION_POLICY=abort`. If
a retrieved candidate trips the gate (high-risk), the whole request aborts. The documented
alternative is `drop`, which discards just that candidate and continues. For allowed-but-
sanitized content, the pipeline uses the Tier-1-sanitized text downstream (defense in depth)
and logs that sanitization happened.
## Refinements after a 21-agent adversarial review
The first validation surfaced three failures. Fixing them, and adversarially reviewing the
fixes, added these mechanisms. See [`FINDINGS.md`](FINDINGS.md) for the full before/after.
- **Tier-3 escalation (gate, request path only):** a legitimate question about prompt injection
was being blocked. The request gate now enables the defender's Tier-3 LLM hook over the gray
band `[0.64, 0.85)`, just above the model's calibrated 0.64 block threshold. A cheap
adjudicator can rescue a would-be block but never flip a would-be allow, while near-certain
attacks (≥0.85) still hard-block without consulting it. Retrieved third-party content never
escalates (strict gate).
- **Multi-intent router with §2.4-safe rescue:** the router emits sub-queries and retrieval
unions across them for recall. Reranking stays governed by the original request (§2.4). Only a
genuinely compound request (two or more distinct sub-intents) also maxes over its sub-queries,
to surface a secondary intent the full-request score would bury.
- **Relative-dominance gate:** a secondary skill is kept only if it scores at least
`DOMINANCE_RATIO` (0.08) times the top skill. This keeps PickySteve picky instead of dumping
marginal tag-alongs.
- **Honest #13 fix:** a correct skill that the reranker under-scored was fixed by enriching the
skill doc with real symptom vocabulary, not by lowering the floor onto leaked data. The floor
is recalibrated on a leakage-free labeled set with hard-negatives.
## Benchmarks
All numbers below come from this repo's own eval docs and logs.
<p align="center">
<img src="assets/img/hero-100x10.webp" alt="100% × 10 consecutive runs — Base 26/26, Harder 42/42, Held-out 47/47" width="720">
</p>
<p align="center">
<img src="assets/img/bench-ablation.webp" alt="Reranker alone 71% vs full PickySteve pipeline 96% on 24 confusable skill pairs" width="49%">
<img src="assets/img/bench-suites.webp" alt="Routing accuracy across suites: Base, Harder, Held-out 100%; Adversarial 96%" width="49%">
</p>
<p align="center">
<img src="assets/img/bench-security.webp" alt="Security gate: 100% attack detection, 0 bypasses, 0% false positives, 180-payload red-team" width="49%">
<img src="assets/img/bench-twotier.webp" alt="Two-tier conformal gate: 96% recall at 38% of frontier cost" width="49%">
</p>
**Routing accuracy, the trifecta** ([DEEP_CONTEXT.md](DEEP_CONTEXT.md)):
| Suite | Tasks | Result |
|---|---|---|
| Base | 26 | **100% × 10 consecutive runs** (qwen3 judge) |
| Harder (base + 16 brutal adversarial) | 42 | **100% × 10** (qwen3 judge) |
| Held-out (unseen, fresh confusion mechanisms) | 47 | **100% × 10** (Claude blind judge) |
| Heldout2 (hardest, deliberately unsaturated adversarial set) | 24 | **23/24 (96%)**. One genuine miss on a compound canary/feature-flag task where the trap ranked above gold (`logs/heldout2_final_run.log`) |
The heldout2 set is kept deliberately hard and unsaturated. New confusable-pair tasks get added
faster than the router/rerank stack is re-tuned, so it acts as a running canary for regressions
rather than a suite that is expected to hit 100%.
On a 40-request held-out accuracy set with no calibration overlap ([TEST_REPORT.md](TEST_REPORT.md)):
**90% overall correct**, **100% top-1 accuracy (30/30 answerable)**, **96.7% full recall**,
**MRR 1.000**, **100% off-domain rejection** (haiku/recipe requests correctly get `no_confident_match`).
**Two-tier gate (recall-all + conformal abstention).** The cheap local judge routes singl
... (truncated)
tools
Comments
Sign in to leave a comment