Uscis Knowledge Base

README

# USCIS Knowledge Base

This OpenClaw plugin crawls `uscis.gov`, turns the site into a local RAG dataset, and refreshes the dataset daily.

What it captures:

- Newsroom pages, including `all-news`, `news-releases`, and `alerts`
- Process and instruction pages under forms, policy, tools, family, citizenship, humanitarian, green card, and work sections
- Sitemap-discovered pages across `uscis.gov`
- Optional PDF URLs for tracking, even though PDF text extraction is not available in the current dependency-free build

Local artifacts:

- `uscis_dataset.schema.json`
- `documents.jsonl`
- `document_versions.jsonl`
- `pages.jsonl`
- `blocks.jsonl`
- `chunks.jsonl`
- `change_log.jsonl`
- `qa_pairs.jsonl`
- `state.json`
- `manifest.json`
- `runs/<timestamp>/`

The plugin uses local Ollama Gemma models for enrichment and can use an OpenAI-compatible LM Studio endpoint for QA checks. The default enrichment model tag is `gemma4:27b`, with fallbacks configured in the plugin settings. The default QA backend is LM Studio with `google/gemma-4-e4b`, and the default second-layer usefulness reviewer is `nvidia/nemotron-3-super`.

For LM Studio QA, set the QA provider to `lmstudio`, the base URL to `http://127.0.0.1:1234`, the QA model to `google/gemma-4-e4b`, and the QA review model to `nvidia/nemotron-3-super`.

Use the OpenClaw tool `uscis_rag_sync` or the scheduled task scripts:

```powershell
powershell -ExecutionPolicy Bypass -File C:\Users\roger\Documents\GitHub\uscis-knowledge-base\scripts\run-uscis-refresh.ps1
powershell -ExecutionPolicy Bypass -File C:\Users\roger\Documents\GitHub\uscis-knowledge-base\scripts\register-uscis-daily-task.ps1 -TaskName "OpenClaw USCIS Knowledge Base Refresh" -Hour 2 -Minute 15
```

Notes:

- The crawler prefers the official sitemap and newsroom pages rather than brute-force link walking.
- The crawler is incremental: changed pages are appended to the local dataset and tracked in `state.json`.
- PDF handling is layered: `PyMuPDF` first, `pypdf` second, then image OCR with local `gemma4:26b` for scanned or image-only pages.
- QA checks use `google/gemma-4-e4b` from LM Studio through the OpenAI-compatible `/v1/chat/completions` API by default.
- PDF/HTML usefulness review uses `nvidia/nemotron-3-super` by default.
- QA rows now preserve `purpose_display` as the human-facing purpose field.
- PDF QA rows also preserve `purpose_answer`, `support_quote`, and `mentioned_forms_or_steps` in addition to the compatibility `answer` and `evidence_quote` fields.
- The HTML extractor filters site chrome when a `<main>` or `<article>` region is present, which reduces duplicate nav/footer text in chunks.
- `documents -> document_versions -> pages -> blocks -> chunks -> change_log` is the canonical source order for refreshes and downstream training.
- If `pymupdf` and `pypdf` are not installed yet, install them locally and rerun the refresh.


---

## Repository layout

This repo is the canonical location. `~/openclaw-plugins/uscis-knowledge-base` is a directory junction pointing here, so OpenClaw discovers the plugin at its usual path while all files live in this repo.

```
uscis-knowledge-base/
├── index.js                 # OpenClaw plugin entry (uscis_rag_sync tool)
├── openclaw.plugin.json     # Plugin manifest + config schema
├── package.json
├── scripts/
│   ├── uscis_pipeline.py    # Crawler + RAG dataset pipeline
│   ├── run-uscis-refresh.ps1
│   ├── register-uscis-daily-task.ps1
│   ├── run-dots-ocr.cmd     # Dots OCR runner (needs third_party setup)
│   └── install-pdf-deps.ps1
├── skills/                  # OpenClaw skill docs
└── uscis_dataset.schema.json
```

### Third-party dependencies

`third_party/` (dots.ocr model weights + venvs, ~16.7 GB) is **not** committed.
It lives locally at `~/Documents/GitHub/uscis-knowledge-base/third_party/` (inside this repo, gitignored).
Restore after a fresh clone:

```powershell
git clone https://github.com/toruki13/uscis-knowledge-base.git
# re-create third_party/dots.ocr from https://github.com/rednote-hilab/dots.ocr
# and download DotsMOCR weights (huggingface.co/rednote-hilab/dots.mocr)
```

### Runtime data

Crawl artifacts (`documents.jsonl`, `chunks.jsonl`, `state.json`, `runs/`) are
generated by the pipeline and gitignored. Default data dir is configurable via
the plugin's `dataDir` setting (currently points at the local plugin folder).