emergency-rescue

---
name: emergency-rescue
description: Recover from developer disasters. Use when someone force-pushed to main, leaked credentials in git, ran out of disk space, killed the wrong process, corrupted a database, broke a deploy, locked themselves out of SSH, lost commits after a bad rebase, or hit any other "oh no" moment that needs immediate, calm, step-by-step recovery.
metadata: {"clawdbot":{"emoji":"🚨","requires":{"anyBins":["git","bash"]},"os":["linux","darwin","win32"]}}
---

# Emergency Rescue Kit

Step-by-step recovery procedures for the worst moments in a developer's day. Every section follows the same pattern: **diagnose → fix → verify**. Commands are non-destructive by default. Destructive steps are flagged.

When something has gone wrong, find your situation below and follow the steps in order.

## When to Use

- Someone force-pushed to main and overwrote history
- Credentials were committed to a public repository
- A rebase or reset destroyed commits you need
- Disk is full and nothing works
- A process is consuming all memory or won't die
- A database migration failed halfway through
- A deploy needs to be rolled back immediately
- SSH access is locked out
- SSL certificates expired in production
- You don't know what went wrong, but it's broken

---

## Git Disasters

### Force-pushed to main (or any shared branch)

Someone ran `git push --force` and overwrote remote history.

```bash
# DIAGNOSE: Check the reflog on any machine that had the old state
git reflog show origin/main
# Look for the last known-good commit hash

# FIX (if you have the old state locally):
git push origin <good-commit-hash>:main --force-with-lease
# --force-with-lease is safer than --force: it fails if remote changed again

# FIX (if you DON'T have the old state locally):
# GitHub/GitLab retain force-pushed refs temporarily

# GitHub: check the "push" event in the audit log or use the API
gh api repos/{owner}/{repo}/events --jq '.[] | select(.type=="PushEvent") | .payload.before'

# GitLab: check the reflog on the server (admin access needed)
# Or restore from any CI runner or team member's local clone

# VERIFY:
git log --oneline -10 origin/main
# Confirm the history looks correct
```

### Lost commits after rebase or reset --hard

You ran `git rebase` or `git reset --hard` and commits disappeared.

```bash
# DIAGNOSE: Your commits are NOT gone. Git keeps everything for 30+ days.
git reflog
# Find the commit hash from BEFORE the rebase/reset
# Look for entries like "rebase (start)" or "reset: moving to"

# FIX: Reset back to the pre-disaster state
git reset --hard <commit-hash-before-disaster>

# FIX (alternative): Cherry-pick specific lost commits
git cherry-pick <lost-commit-hash>

# FIX (if reflog is empty — rare, usually means different repo):
git fsck --lost-found
# Look in .git/lost-found/commit/ for dangling commits
ls .git/lost-found/commit/
git show <hash>  # Inspect each one

# VERIFY:
git log --oneline -10
# Your commits should be back
```

### Committed to the wrong branch

You made commits on `main` that should be on a feature branch.

```bash
# DIAGNOSE: Check where you are and what you committed
git log --oneline -5
git branch

# FIX: Create the feature branch at current position, then reset main
git branch feature-branch          # Create branch pointing at current commit
git reset --hard HEAD~<N>          # Move main back N commits (⚠️ destructive)
git checkout feature-branch        # Switch to the new branch

# FIX (safer alternative using cherry-pick):
git checkout -b feature-branch     # Create and switch to new branch
git checkout main
git reset --hard origin/main       # Reset main to remote state
# Your commits are safely on feature-branch

# VERIFY:
git log --oneline main -5
git log --oneline feature-branch -5
```

### Merge gone wrong (conflicts everywhere, wrong result)

A merge produced a bad result and you want to start over.

```bash
# FIX (merge not yet committed — still in conflict state):
git merge --abort

# FIX (merge was committed but not pushed):
git reset --hard HEAD~1

# FIX (merge was already pushed): Create a revert commit
git revert -m 1 <merge-commit-hash>
# -m 1 means "keep the first parent" (your branch before merge)
git push

# VERIFY:
git log --oneline --graph -10
git diff HEAD~1  # Review what changed
```

### Corrupted git repository

Git commands fail with "bad object", "corrupt", or "broken link" errors.

```bash
# DIAGNOSE: Check repository integrity
git fsck --full

# FIX (if remote is intact — most common):
# Save any uncommitted work first
cp -r . ../repo-backup

# Re-clone and restore local work
cd ..
git clone <remote-url> repo-fresh
cp -r repo-backup/path/to/uncommitted/files repo-fresh/

# FIX (repair without re-cloning):
# Remove corrupt objects and fetch them again
git fsck --full 2>&1 | grep "corrupt\|missing" | awk '{print $NF}'
# For each corrupt object:
rm .git/objects/<first-2-chars>/<remaining-hash>
git fetch origin  # Re-download from remote

# VERIFY:
git fsck --full  # Should report no errors
git log --oneline -5
```

---

## Credential Leaks

### Secret committed to git (API key, password, token)

A credential is in the git history. Every second counts — automated scrapers monitor public GitHub repos for leaked keys.

```bash
# STEP 1: REVOKE THE CREDENTIAL IMMEDIATELY
# Do this FIRST, before cleaning git history.
# The credential is already compromised the moment it was pushed publicly.

# AWS keys:
aws iam delete-access-key --access-key-id AKIAXXXXXXXXXXXXXXXX --user-name <user>
# Then create a new key pair

# GitHub tokens:
# Go to github.com → Settings → Developer settings → Tokens → Revoke

# Database passwords:
# Change the password in the database immediately
# ALTER USER myuser WITH PASSWORD 'new-secure-password';

# Generic API tokens:
# Revoke in the provider's dashboard, generate new ones

# STEP 2: Remove from current branch
git rm --cached <file-with-secret>    # If the whole file is secret
# OR edit the file to remove the secret, then:
git add <file>

# STEP 3: Add to .gitignore
echo ".env" >> .gitignore
echo "credentials.json" >> .gitignore
git add .gitignore

# STEP 4: Remove from git history (⚠️ rewrites history)
# Option A: git-filter-repo (recommended, install with pip install git-filter-repo)
git filter-repo --path <file-with-secret> --invert-paths

# Option B: BFG Repo Cleaner (faster for large repos)
# Download from https://rtyley.github.io/bfg-repo-cleaner/
java -jar bfg.jar --delete-files <filename> .
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# STEP 5: Force push the cleaned history
git push origin --force --all
git push origin --force --tags

# STEP 6: Notify all collaborators to re-clone
# Their local copies still have the secret in reflog

# VERIFY:
git log --all -p -S '<the-secret-string>' --diff-filter=A
# Should return nothing
```

### .env file pushed to public repo

```bash
# STEP 1: Revoke ALL credentials in that .env file. All of them. Now.

# STEP 2: Remove and ignore
git rm --cached .env
echo ".env" >> .gitignore
git add .gitignore
git commit -m "Remove .env from tracking"

# STEP 3: Remove from history (see credential removal above)
git filter-repo --path .env --invert-paths

# STEP 4: Check what was exposed
# List every variable that was in the .env:
git show HEAD~1:.env 2>/dev/null || git log --all -p -- .env | head -50
# Rotate every single value.

# PREVENTION: Add a pre-commit hook
cat > .git/hooks/pre-commit << 'HOOK'
#!/bin/bash
if git diff --cached --name-only | grep -qE '\.env$|\.env\.local$|credentials'; then
    echo "ERROR: Attempting to commit potential secrets file"
    echo "Files: $(git diff --cached --name-only | grep -E '\.env|credentials')"
    exit 1
fi
HOOK
chmod +x .git/hooks/pre-commit
```

### Secret visible in CI/CD logs

```bash
# STEP 1: Revoke the credential immediately

# STEP 2: Delete the CI run/logs if possible
# GitHub Actions:
gh run delete <run-id>
# Or: Settings → Actions → delete specific run

# STEP 3: Fix the pipeline
# Never echo secrets. Mask them:
# GitHub Actions: echo "::add-mask::$MY_SECRET"
# GitLab CI: variables are masked if marked as "Masked" in settings

# STEP 4: Audit what was exposed
# Check the log output for patterns like:
# AKIAXXXXXXXXX (AWS)
# ghp_XXXXXXXXX (GitHub)
# sk-XXXXXXXXXXX (OpenAI/Stripe)
# Any connection strings with passwords
```

---

## Disk Full Emergencies

### System or container disk is full

Nothing works — builds fail, logs can't write, services crash.

```bash
# DIAGNOSE: What's using space?
df -h                          # Which filesystem is full?
du -sh /* 2>/dev/null | sort -rh | head -20    # Biggest top-level dirs
du -sh /var/log/* | sort -rh | head -10        # Log bloat?

# QUICK WINS (safe to run immediately):

# 1. Docker cleanup (often the #1 cause)
docker system df               # See Docker disk usage
docker system prune -a -f      # Remove all unused images, containers, networks
docker volume prune -f          # Remove unused volumes
docker builder prune -a -f      # Remove build cache
# ⚠️ This removes ALL unused Docker data. Safe if you can re-pull/rebuild.

# 2. Package manager caches
# npm
npm cache clean --force
rm -rf ~/.npm/_cacache

# pip
pip cache purge

# apt
sudo apt-get clean
sudo apt-get autoremove -y

# brew
brew cleanup --prune=all

# 3. Log rotation (immediate)
# Truncate (not delete) large log files to free space instantly
sudo truncate -s 0 /var/log/syslog
sudo truncate -s 0 /var/log/journal/*/*.journal  # systemd journals
find /var/log -name "*.log" -size +100M -exec truncate -s 0 {} \;
# Truncate preserves the file handle so services don't break

# 4. Old build artifacts
find . -name "node_modules" -type d -prune -exec rm -rf {} + 2>/dev/null
find . -name ".next" -type d -exec rm -rf {} + 2>/dev/null
find . -name "dist" -type d -exec rm -rf {} + 2>/dev/null
find /tmp -type f -mtime +7 -delete 2>/dev/null

# 5. Find the actual culprit
find / -xdev -type f -size +100M -exec ls -l

... (truncated)