Tools
Nvidia Speech
By Nvidia Speech Plugin for OpenClaw
Install
npm install &&README
# `@dhiraj-salian/openclaw-nvidia-speech`
[](https://github.com/dhiraj-salian/openclaw-nvidia-speech/actions/workflows/ci.yml)
[](https://www.npmjs.com/package/@dhiraj-salian/openclaw-nvidia-speech)
[](./LICENSE)
[](https://docs.openclaw.ai)
OpenClaw plugin that adds **NVIDIA TTS** (Magpie Multilingual) and **STT** (Parakeet CTC) capabilities using **only your existing `NVIDIA_API_KEY`**.
- ๐๏ธ **TTS** โ natural multilingual speech from the Magpie Multilingual family.
- ๐ค **STT** โ fast, accurate transcription from Parakeet CTC (multilingual variant).
- ๐ **No Google. No OpenAI. No Deepgram. No extra accounts.**
- โก **Zero runtime npm deps** โ uses Node 22's built-in `fetch`, `FormData`, `Blob`.
- ๐งช **116 unit + integration tests** โ every layer verified offline before runtime.
If you already pay for NVIDIA NIM access (it's free for the bundled models), this is the only plugin you need for voice in + voice out.
---
## Table of contents
- [Install](#install)
- [Configure](#configure)
- [Use](#use)
- [API reference](#api-reference)
- [Troubleshoot](#troubleshoot)
- [Architecture](#architecture)
- [Contributing](#contributing)
- [License](#license)
---
## Install
> **Requirements:** OpenClaw `>=2026.3.24-beta.2`, Node `>=22.19.0`.
Pick **one** of the install paths below. They all install the same plugin.
### A. From GitHub (recommended)
```bash
openclaw plugins install github:dhiraj-salian/openclaw-nvidia-speech
```
### B. From npm
```bash
openclaw plugins install npm:@dhiraj-salian/openclaw-nvidia-speech
```
### C. From a local checkout (dev / hot-reload)
```bash
git clone https://github.com/dhiraj-salian/openclaw-nvidia-speech.git
cd openclaw-nvidia-speech
npm install && npm run build
openclaw plugins install --link ./
```
The `--link` flag symlinks the source so edits + a `npm run build` are picked up on the next plugin reload.
### D. From ClawHub (when published)
```bash
openclaw plugins install clawhub:openclaw-nvidia-speech
```
---
## Configure
You need one thing: **a NVIDIA API key with free credits** on
[build.nvidia.com](https://build.nvidia.com/settings/api-keys). The same key works for both TTS and STT.
### 1. Set the env var
```bash
# ~/.bashrc, ~/.zshrc, or your deployment secret store
export NVIDIA_API_KEY="nvapi-โฆ"
```
### 2. Wire it into OpenClaw
OpenClaw interpolates `${NVIDIA_API_KEY}` at config-load time, so the secret
ref stays out of your committed JSON. Add this to `~/.openclaw/openclaw.json`:
```json5
{
// โโ TTS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
"messages": {
"tts": {
"auto": "always", // or "tagged" for opt-in
"provider": "nvidia", // pick the bundled Magpie provider
"providers": {
"nvidia": {
"apiKey": "${NVIDIA_API_KEY}", // env-var secret ref
"model": "magpie-tts-multilingual", // (default; can be omitted)
"voice": "Magpie-Multilingual.EN-US.Aria",
"language": "en-US",
"sampleRate": 22050,
"format": "wav" // wav | mp3 | flac | ogg | opus
}
}
}
},
// โโ STT โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
"tools": {
"media": {
"audio": {
"enabled": true,
"models": [
{ "provider": "nvidia" } // picks the default Parakeet model
]
}
}
}
}
```
Restart the gateway once (`openclaw gateway restart` or the equivalent for your platform).
### 3. Verify
```bash
openclaw plugins inspect nvidia-speech --runtime --json
```
You should see both `speechProviders: ["nvidia"]` and `mediaUnderstandingProviders: ["nvidia"]` registered.
---
## Use
### TTS
- **Auto-TTS** โ every outbound reply becomes audio (after the `messages.tts.auto: "always"` config above).
- **One-off** โ `/tts audio "Hello from NVIDIA"`
- **Status** โ `/tts status`
- **Per-reply overrides** โ use model-emitted directives:
```text
Say it in a different voice.
[[tts:voice=Magpie-Multilingual.HI-IN.Aditi lang=hi-IN format=mp3]]
```
- **Voices** โ programmatically list via the bundled `tts voices` machinery; Magpie's available voices will show alongside the other providers.
### STT
Just send a voice note from any supported channel (WhatsApp, Telegram, Feishu, Matrix, Discord, โฆ). OpenClaw will:
1. Detect the audio attachment.
2. Pick the first eligible entry in `tools.media.audio.models` โ your NVIDIA provider.
3. POST the file as `multipart/form-data` to `https://integrate.api.nvidia.com/v1/audio/transcriptions`.
4. Inject `[Audio] <transcript>` into the conversation.
> **Tip:** Add a local fallback for OOO cases:
> ```json5
> "tools": { "media": { "audio": { "models": [
> { "provider": "nvidia" },
> { "type": "cli", "command": "whisper", "args": ["--model", "base", "{{MediaPath}}"] }
> ] } } }
> ```
---
## API reference
### Default IDs
| Provider id | Purpose | Default model | Auto-priority |
|---|---|---|---|
| `nvidia` (TTS) | SpeechProviderPlugin โ Magpie Multilingual | `magpie-tts-multilingual` | n/a |
| `nvidia` (STT) | MediaUnderstandingProviderPlugin โ Parakeet CTC | `parakeet-ctc-1.1b-en-multilingual` | `50` (higher than ElevenLabs at 45) |
### Per-provider config
| Path | Type | Default | Notes |
|---|---|---|---|
| `messages.tts.providers.nvidia.apiKey` | `string \| {value: string}` | โ | Required. Use `"${NVIDIA_API_KEY}"` for env-var ref. |
| `messages.tts.providers.nvidia.model` | `string` | `magpie-tts-multilingual` | |
| `messages.tts.providers.nvidia.voice` | `string` | `Magpie-Multilingual.EN-US.Aria` | See `/audio/voices` for full list. |
| `messages.tts.providers.nvidia.language` | `string` | `en-US` | e.g. `en-US`, `hi-IN`, `de-DE`. |
| `messages.tts.providers.nvidia.sampleRate` | `number` | `22050` | 8000 / 16000 / 22050 / 24000 / 44100 / 48000. |
| `messages.tts.providers.nvidia.format` | `string` | `wav` | `wav` \| `mp3` \| `flac` \| `ogg` \| `opus`. |
| `messages.tts.providers.nvidia.baseUrl` | `string` | `https://integrate.api.nvidia.com/v1` | Override for a proxy / private NIM. |
| `tools.media.audio.models[].provider` | `"nvidia"` | โ | Required. |
| `tools.media.audio.models[].model` | `string` | default above | Optional override. |
### Inline TTS directives
| Key | Example | Policy default |
|---|---|---|
| `voice` | `voice=Magpie-HI-IN.Aditi` | allowed |
| `model` | `model=magpie-tts-multilingual` | allowed |
| `lang` / `language` | `lang=hi-IN` | allowed |
| `format` | `format=mp3` | allowed |
| `sampleRate` | `sampleRate=24000` | allowed |
| `provider` | `provider=nvidia` | **disallowed** by default |
Enable `modelOverrides.allowProvider: true` if you want the assistant to switch providers mid-message.
---
## Troubleshoot
### `MissingApiKeyError: NVIDIA API key not found`
Either `NVIDIA_API_KEY` isn't set in the shell that runs the gateway, or the
`apiKey` config interpolation failed. Run:
```bash
echo "$NVIDIA_API_KEY" # should print `nvapi-โฆ`
openclaw plugins inspect nvidia-speech --runtime --json | jq '.config'
```
If the key is set but OpenClaw still complains, add `apiKey: "${NVIDIA_API_KEY}"` directly under `messages.tts.providers.nvidia` in `~/.openclaw/openclaw.json` (the path the runtime substitution uses).
### `HTTP 401: invalid api key`
The key is set but invalid. Generate a fresh one at
[build.nvidia.com/settings/api-keys](https://build.nvidia.com/settings/api-keys).
### `HTTP 429: rate limited`
NVIDIA's free tier rate-limits aggressively. The plugin auto-retries up to 3
times with exponential backoff (500 ms โ 1 s โ 2 s, capped at 8 s, with 0โ250 ms jitter). If retries fail you'll see the underlying error surfaced โ back off manually or upgrade your NVIDIA tier.
### `NvSpeechError kind=not_found`
The model id is wrong, or your account doesn't have access. The current
defaults (`magpie-tts-multilingual`, `parakeet-ctc-1.1b-en-multilingual`) are
on the free tier as of 2026-06.
### Voice note plays as a regular audio file in WhatsApp / Feishu
Make sure `format` is `mp3`, `ogg`, or `opus` (all native-supported codecs for
voice notes in those channels). `wav` works but is larger.
### Voice comes back as silent bytes
`sampleRate: 8000` is sometimes rejected by Magpie for female voices. Use `22050` (default) or `24000`.
### Plugin doesn't register after install
```bash
openclaw plugins list # confirm install
openclaw plugins inspect nvidia-speech # check manifest + contracts
openclaw doctor # catch config-shape errors
```
---
## Architecture
| Layer | Path | Purpose |
|---|---|---|
| HTTP | `src/http/` | `HttpClient` interface + `FetchHttpClient` + `RetryHttpClient` decorator + `FakeHttpClient` (tests). |
| Errors | `src/http/errors.ts` | `NvSpeechError` hierarchy with `kind`, `status`, `requestId`. |
| Config | `src/config/` | Static defaults, validators, `normalizeRawConfig`. |
| Secrets | `src/utils/secret-resolver.ts` | API-key precedence chain; never logs. |
| TTS | `src/tts/nvidia-tts-client.ts` | Raw `POST /audio/synthesize`. |
| TTS | `src/tts/voices.ts` | `GET /audio/voices` with 1 h cache. |
| TTS | `src/tts/speech-provider.ts` | The OpenClaw `SpeechProviderPlugin` boundary. |
| STT | `src/stt/nvidia-stt-client.ts` | Raw `POST /audio/transcriptions` (multipart). |
| STT | `src/stt/media-provider.ts` | The OpenClaw `MediaUnderstandingProviderPlugin` boundary. |
| Entry | `src/index.ts` | `definePluginEntry` + factory wiring. |
**SOLID highlights:**
- **D (Dependency Inversion)** โ every layer depends on `HttpClient`, not `fetch`.
- **S (Single
... (truncated)
tools
Comments
Sign in to leave a comment