System Test

The system test at /admin/system-test is a deep validation tool that goes beyond HTTP status code checks. It probes every backend service, parses response bodies, verifies actual data structures, checks all 31 module manifests, and audits every environment variable.

Unlike /admin/site-test (client-side, status codes only) and /admin/health (lightweight service pings), System Test proves services are returning real data — model lists, subscription details, account status, env var presence — with expandable JSON evidence on every row.

Three Test Suites

Suite 1: Service Deep Checks (16 services)

Server-side probes against all backend services. Each probe fetches the service’s health or status endpoint AND parses the response body to validate the data structure.

Service	What’s Validated	Example Output
OpenClaw Gateway	GET /v1/models — data array has models	”4 models: gpt-4o, claude-3.5…”
Ollama	GET /api/tags — models array exists	”3 models: llama3.2:3b, mistral:7b…”
Argonaut Daemon	GET /health — status and capabilities	”Status: ok, capabilities: chat, rag”
Build Swarm Gateway	GET /status — fleet/drone data	”Gateway up, 4 drones”
Forge Relay	GET /health — response exists	”Relay healthy”
Pentest Sentinel	GET /health — tools list	”Online, tools: nmap, nikto…”
Pentest Tarn-Host	GET /health — daemon status	”Tarn-Host node online”
Pentest Izar-Host	GET /health — daemon status	”Izar-Host node online”
Meridian-Host AdminBox	GET /health — response exists	”AdminBox healthy”
Tarn-Host AdminBox	GET /health — response exists	”AdminBox healthy”
Uptime Kuma	GET /api/status-page/heartbeat — monitor data	”Monitoring active, 12 monitors”
Twilio	GET /Accounts/{sid}.json — status field	”Account active: My Account”
ElevenLabs	GET /v1/user/subscription — char usage	”2341/10000 chars used”
Cloudflare	GET /tokens/verify — success field	”Token valid, status: active”
Applyr (Jobs)	GET /health — response OK	”Service running”
Network Scanner	GET /health — response OK	”Scanner running”

Services without configured env vars are reported as Skip (gray), not failures.

Suite 2: Module Integrity (31 modules)

For every module in the module registry:

• Enabled state — reads from Cloudflare KV via getModuleStates()
• Env var check — verifies each requiredEnvVars[] entry is set
• API route sample — fetches the first non-parameterized API route and checks for non-500 response

Result logic:

• All checks pass → Pass (green)
• Missing env vars on an enabled module → Warn (amber)
• API route returns 500+ → Fail (red)
• Module disabled → Skip (gray)

Suite 3: Environment Audit

Scans all requiredEnvVars across all 31 modules plus system-level vars (API_KEYS_ENCRYPTION_KEY, CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, ADMIN_EMAILS, ADMIN_DB). Reports which modules need each variable.

• Set → Pass, “Set (used by: openclaw, argonaut)”
• Not set → Warn, “NOT SET (needed by: twilio)”

Never exposes actual values — only presence/absence.

How It Works

1. Click Run Tests. The page calls GET /api/admin/system-test.
2. The API runs all 3 suites in parallel (Promise.all).
3. Within each suite, individual checks run concurrently (Promise.allSettled, 5s timeout per probe).
4. Results return as structured JSON with evidence objects for every test.
5. The page renders suites with collapsible sections. All-passing suites auto-collapse.

Total time: ~5-10 seconds (bounded by the slowest parallel probe, not the sum).

Runtime Hardening (2026-02-28)

Symptom

After repeated ViewTransitions navigations, the browser console showed repeated client errors from system-test scripts:

• Cannot read properties of undefined (reading 'style')
• Cannot read properties of null (reading 'appendChild')

This could happen when initialization code ran outside the expected route context or before required DOM nodes were available.

Root Cause

The page script assumed all required elements always exist and was not route-scoped. During client-side navigation, a page-load handler can run while DOM timing/context differs, causing null/undefined element usage.

Fixes Applied

In src/pages/admin/system-test.astro:

1. Added route-aware init guard (/admin/system-test and /system-test only).
2. Added required-element checks before any UI mutation.
3. Hardened rendering against missing/partial response shapes.
4. Bound astro:page-load initializer exactly once to avoid duplicate handler buildup.
5. Removed a stray extra closing </CosmicLayout> tag at the end of file.

Commits

• 3dde152 — lifecycle + DOM guard hardening for system test page
• 4a1cc9a — companion admin parser fix + troubleshooting documentation

Expandable Evidence

Click any row to expand its evidence panel — a syntax-highlighted JSON view of the actual data returned by the service. This is the key differentiator: you can see the model names, subscription details, module metadata, and env var groupings. It proves the service is returning real data, not empty 200s.

Comparison with Other Health Tools

Check	/admin/health	/admin/site-test	/admin/system-test
Service reachable	Yes	—	Yes
Response has valid data	—	—	Yes
Module enabled state	—	—	Yes
Env vars configured	—	—	Yes
API routes return real data	—	Status code only	Yes + evidence
All 31 modules verified	—	—	Yes
Expandable proof	—	—	Yes

Related

Item	Value
Page source	src/pages/admin/system-test.astro
API source	src/pages/api/admin/system-test.ts
Module	health-monitor
Auth	CF Access (admin)
SSR	Yes (export const prerender = false)
Key commits	2267181, 3dde152, 82042c3