Tools
SwarmLLM
By Decentralized peer-to-peer LLM inference network. Single Rust binary, BitTorrent-inspired incentives, OpenAI-compatible API.
Install
pip install swarmllm-client`),Configuration Example
{ "mcpServers": { "swarmllm": { "url": "http://localhost:8800/mcp" } } }README
# SwarmLLM
[](https://github.com/enapt/SwarmLLM/actions/workflows/ci.yml)
[](LICENSE-MIT)
[](https://www.rust-lang.org/)
[](https://github.com/enapt/SwarmLLM/pkgs/container/swarmllm)
[](https://github.com/enapt/SwarmLLM/releases)
[](https://discord.gg/nq9be3u828)
A peer-to-peer LLM inference network in a single Rust binary. Pool hardware with other nodes to run 70B+ parameter models on machines that couldn't host them alone — no API tokens, no cloud fees, and encrypted traffic between every peer.
**Join the swarm. Run AI together — for free.**
> **Status — alpha**, actively developed. Distributed inference is stable across multi-node deployments. 2220 lib tests + 79 integration tests run on every PR; continuous security sweeps. [Report issues](https://github.com/enapt/SwarmLLM/issues).
>
> **Recent work (September 2026) — a model that does not quite fit your graphics card no longer loses the card.** Placement used to be all or nothing: a model needing a little more graphics memory than was free ran *entirely* on the processor while the card sat idle beside it. Since v0.3.145 the node splits it — the first layers on the card, the rest on the processor, the count chosen automatically — measured on an RTX 3070 at **1.8× the processor-only speed** for a 7B that no longer fits, against 7–8× when it does (see [Benchmarks](#benchmarks)). The same month: **a model no single node can hold is served** by chaining peers (a 14B across three machines on two continents), every node advertises a *measured* speed instead of a constant that was 5× low, the diagnostics report is safe to paste in public, Apple Silicon nodes can update themselves again — and a machine that vanishes or refuses mid-request now costs the request seconds, not minutes (v0.3.147).
>
> **Earlier (July 2026) — inference across NAT.** Two machines behind ordinary home routers can run a model together: a sealed application-level relay carries the tensor traffic when no direct path exists, and direct connections are established opportunistically on top. Verified end-to-end by an external tester. Local models also gained working **tool calling** on both API surfaces, streaming included.
>
> **Benchmarks:** prompt processing is up to **3× faster** and replying inside a long conversation up to **5.5× faster** as of v0.3.81 (measured 2026-08-07, see [Benchmarks](#benchmarks)). Cross-node prefix-KV sharing delivers a **12.9× iter-1 TTFT speedup** on 7B prompts when a peer has the same prefix cached (measured 2026-04-20). Windows release binaries reach Linux parity on single-node and split inference (validated 2026-04-23).
For long-form documentation see the [SwarmLLM book](https://enapt.github.io/SwarmLLM/).
---
<details>
<summary><strong>Table of Contents</strong></summary>
- [Quick Start](#quick-start)
- [Use it as an API](#use-it-as-an-api)
- [Use it with OpenClaw](#use-it-with-openclaw)
- [What it does](#what-it-does)
- [Networking & Privacy](#networking--privacy)
- [Capabilities](#capabilities)
- [Supported Models](#supported-models)
- [Benchmarks](#benchmarks)
- [Architecture](#architecture)
- [Installation](#installation)
- [CLI](#cli)
- [Configuration](#configuration)
- [API Endpoints](#api-endpoints)
- [Platform Support](#platform-support)
- [How SwarmLLM Compares](#how-swarmllm-compares)
- [Documentation](#documentation)
- [Contributing & Support](#contributing--support)
- [Development Transparency](#development-transparency)
- [License](#license)
</details>
## Quick Start
Download a binary from [GitHub Releases](https://github.com/enapt/SwarmLLM/releases), extract, and run:
```bash
./swarmllm run
```
Your browser opens to `localhost:8800`. The setup wizard auto-detects your hardware. Pick a model, download it, start chatting.
**It connects to the live network on its own.** On first run your node auto-joins the public swarm — nothing to configure, no ports to forward. A built-in bootstrap anchor gets you onto the network, UPnP opens your port when your router supports it, AutoNAT v2 tells you your reachability, and a relay fallback keeps you connected even behind CGNAT. Peers and shared models appear on the dashboard within seconds.
> 💬 **New here? [Join the Discord](https://discord.gg/nq9be3u828).** It's the fastest way to find peers to pool with, share node addresses, and get help — the network grows one member at a time, so come say hi.
| Platform | File | Notes |
|----------|------|-------|
| **Windows x86_64** | **`SwarmLLM-Setup.exe`** | **Recommended** — installer auto-detects GPU (NVIDIA / AMD / Intel) |
| Linux x86_64 + CUDA | `swarmllm-linux-x86_64-cuda.tar.gz` | NVIDIA GPU acceleration — **RTX 30-series or newer** |
| Linux x86_64 | `swarmllm-linux-x86_64.tar.gz` | CPU inference |
| Windows x86_64 (GPU) | `swarmllm-windows-x86_64-gpu.zip` | Raw binary: Vulkan + CUDA static |
| Windows x86_64 (CPU) | `swarmllm-windows-x86_64-cpu.zip` | Raw binary: CPU-only fallback |
| macOS Apple Silicon | `swarmllm-macos-aarch64.tar.gz` | CPU inference (Metal planned) |
> **NVIDIA GPU acceleration needs an RTX 30-series or newer** (compute
> capability 8.0+ — Ampere, Ada, Blackwell; the RTX 20-series and GTX 16-series
> are below it). This is FlashAttention's own requirement, and it is what makes
> attention fast enough to be worth shipping. **Older cards are not left
> broken**: SwarmLLM detects them at startup, says so in plain language, and
> runs on the processor instead. On Windows, local inference goes through Vulkan
> and is unaffected on any GPU — it is the distributed path that needs CUDA.
See the [Getting Started Guide](https://enapt.github.io/SwarmLLM/getting-started.html) for platform-specific instructions, or [Installation](#installation) below for package managers, Docker, and source builds.
## Use it as an API
**Already running an AI agent, coding assistant or chat UI?** Point it at
SwarmLLM the way you would point it at Ollama or vLLM. Anything that speaks
the OpenAI or Anthropic API works — [OpenClaw](https://github.com/openclaw/openclaw),
Claude Code, Open WebUI, Continue, LibreChat, the `openai` and `anthropic`
SDKs — and what it gets is:
- **No per-token bill.** Every model in the swarm is free to use; the only
cost is the hardware you already own.
- **Models bigger than your machine.** A 14B or 70B that will not fit your
card runs anyway, split across peers, behind the same endpoint.
- **Your cloud keys in one place.** Add OpenAI, Anthropic, DeepSeek or any of
12 providers once, and route to them by model name when you want to.
- **Traffic sealed between peers**, and an optional mode where no remote
machine ever sees your prompt or the reply.
> **Agents send big prompts.** An agent framework's system prompt alone can
> exceed SwarmLLM's shipped 8192-token context, so raise it before pointing an
> agent at a node: `max_seq_len_override = 32768` under `[inference]` in
> `config.toml`. The [OpenClaw section](#use-it-with-openclaw) below has the
> full setup and what to expect from a small graphics card. And for a tool-heavy agent loop, pick the largest model the
> swarm offers you: small models call tools less reliably, and serving models
> your machine cannot hold alone is what the swarm is for.
Your access key is written to `api_key` in SwarmLLM's data directory, and is
shown under Settings → Access Token in the dashboard.
```bash
# Linux; macOS uses ~/Library/Application Support/swarmllm/api_key
KEY=$(cat ~/.local/share/swarmllm/api_key)
curl http://localhost:8800/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $KEY" \
-d '{
"model": "llama3-70b-q4km",
"messages": [{"role": "user", "content": "Hello!"}],
"stream": true
}'
```
**As a Claude Code backend** — full Anthropic Messages API with tools, thinking, and streaming. Claude Code reaches every model in the swarm: local GGUF, distributed across peers, or any of 12 cloud providers (`claude --model gpt-5.4`, `claude --model claude-sonnet-5`, etc.).
```bash
ANTHROPIC_BASE_URL="http://localhost:8800" \
ANTHROPIC_AUTH_TOKEN="$KEY" \
claude --model "qwen2.5-coder-7b"
```
**As an MCP server** — add to `~/.claude/settings.json`:
```json
{ "mcpServers": { "swarmllm": { "url": "http://localhost:8800/mcp" } } }
```
Tools: `chat`, `models`, `compare` (multi-model side-by-side), `research` (fan-out), `batch_prompts`, `delegate`, `node_info`.
## Use it with OpenClaw
[OpenClaw](https://github.com/openclaw/openclaw) is a personal AI agent that
runs on your own machine and talks to you over the messaging apps you already
use. It needs a model behind it, and a SwarmLLM node gives it one for free —
local, or split across a swarm for the models your machine cannot hold alone.
**1. Give the node room for an agent's prompt.** OpenClaw's first turn is
about 14,600 tokens before you have said a word, plus room reserved for the
reply; the shipped 8192-token context refuses it. In SwarmLLM's `config.toml`:
```toml
[inference]
max_seq_len_override = 32768
```
Restart the node. This is a config-file setting, not a dashboard control.
**2. Export the access key.** OpenClaw's "local marker" keys send no
`Authorization` header at all, and every `/v1` route on a node requires one.
The daemon reads the same variable as its own key override, so one export
serves both sides:
```bash
export SWARMLLM_API_KEY="$(cat ~/.local/share/swarmllm/api_key)" # Linux
# macOS: ~/Library/Application Support/swarmllm/api_key
```
**3. Install th
... (truncated)
tools
Comments
Sign in to leave a comment