Troubleshooting

Diagnostic procedures for common issues.

Quick Diagnostics

# Server health at a glance
ssh camlab 'docker compose -f /opt/camlab/docker-compose.yml ps'
ssh camlab 'free -h && df -h && uptime'

# Container resource usage
ssh camlab 'docker stats --no-stream'

# Recent errors across all containers
ssh camlab 'docker compose -f /opt/camlab/docker-compose.yml logs --tail 20 --since 1h'

Problem: Ghost Returns 502

Symptoms: Blog URL returns 502 Bad Gateway

Diagnose:

# Check if Ghost container is running
docker compose ps ghost-cam4

# Check Ghost logs
docker compose logs --tail 50 ghost-cam4

# Check if MySQL is accessible from Ghost
docker exec ghost-cam4 sh -c 'nc -zv mysql 3306'

Common causes: - Ghost container crashed → docker compose restart ghost-cam4 - MySQL not ready → docker compose restart mysql, wait 30s, restart Ghost - Out of memory → check free -h, consider upgrading or stopping unused services

---

Problem: n8n Workflow Fails

Symptoms: Workflow execution shows error in n8n UI

Diagnose:

# Check n8n logs
docker compose logs --tail 100 n8n

# Check runner logs (for the blog that failed)
docker compose logs --tail 50 runner-cam4

# Check if runner container is responsive
docker exec runner-cam4 echo "alive"

# Check if Claude Code works in runner
docker exec runner-cam4 claude --version

Common causes: - Runner container crashed → docker compose restart runner-cam4 - Anthropic API key expired → check .env, verify at console.anthropic.com - MCP server down → check relevant MCP container logs - Workspace full → docker exec runner-cam4 df -h /workspace

---

Problem: Ghost MCP Returns 403

Symptoms: MCP tool calls fail with "Invalid token"

Diagnose:

# Check MCP server logs
docker compose logs --tail 50 ghost-mcp

# Test with curl (use the blog's MCP token from .env)
curl -X POST http://localhost:3002/<token> \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

Common causes: - Token mismatch between .env and runner .mcp.json - Ghost MCP container needs restart after .env change - Ghost API key expired/regenerated but MCP not updated

---

Problem: Reddit MCP Not Responding

Symptoms: Researcher agent fails, can't reach Reddit

Diagnose:

docker compose ps reddit-mcp
docker compose logs --tail 50 reddit-mcp
curl http://localhost:8080/sse

Common causes: - Reddit API credentials expired → check Reddit app at reddit.com/prefs/apps - Container crashed → docker compose restart reddit-mcp - Rate limited by Reddit → wait, check logs for 429 errors

---

Problem: Runware Images Not Generating

Symptoms: Photographer agent fails or produces no images

Diagnose:

docker compose ps runware-mcp
docker compose logs --tail 50 runware-mcp
curl http://localhost:8081/mcp

Common causes: - Runware API key expired or out of credits - Container unhealthy → docker compose restart runware-mcp - Network issue → check from runner: docker exec runner-cam4 curl http://runware-mcp:8081/mcp

---

Problem: High Memory Usage

Symptoms: Server slow, swap usage high, OOM kills in logs

Diagnose:

free -h
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"
dmesg | grep -i "oom\|killed" | tail -20

Fix: - If a specific container is bloated → restart it - If all containers are normal but total is high → consider upgrading to CPX41 - Emergency: stop non-essential containers (dozzle, glances)

---

Problem: Disk Running Low

Symptoms: df -h shows > 80% usage

Diagnose:

df -h
du -sh /opt/camlab/*/  # Which directories are biggest?
docker system df        # Docker disk usage

Fix:

# Clean Docker build cache and unused images
docker system prune -f
docker image prune -a -f  # WARNING: removes all unused images

# Clean old Ghost content (if applicable)
# Check workspace cleanup
du -sh /opt/camlab/runners/*/workspace/

# Check MySQL binary logs
docker exec mysql ls -lh /var/lib/mysql/

---

Problem: Caddy TLS Certificate Failure

Symptoms: Browser shows certificate error, Caddy logs show ACME failures

Diagnose:

sudo systemctl status caddy
sudo journalctl -u caddy --since "1 hour ago"
# Check if port 80 is accessible (needed for ACME challenges)
curl -I http://blog.cam4.camlab.dev

Common causes: - DNS not pointing to server → verify with dig blog.cam4.camlab.dev - Port 80 blocked by firewall → sudo ufw status - Rate limited by Let's Encrypt → wait 1 hour, retry

---

Diagnostic Command Reference

What	Command
All container status	docker compose ps
Container resource usage	docker stats --no-stream
Specific container logs	docker compose logs --tail 100 <name>
Follow logs live	docker compose logs -f <name>
Disk usage by directory	du -sh /opt/camlab/*/
Docker disk usage	docker system df
Check open ports	sudo ss -tlnp
DNS resolution	dig +short <domain>
TLS certificate info	echo | openssl s_client -connect <domain>:443 2>/dev/null | openssl x509 -noout -dates
MySQL connection test	docker exec mysql mysqladmin -u root -p ping
System resource overview	htop or visit glances.camlab.dev