pnpm agent-eval
The scripted rubric behind the AI-first claim: can this environment go from zero to a validated, deploy-ready app using only the public CLI? Any agent or CI runs it and gets a score.
pnpm agent-eval # ~1 min, 9 steps
pnpm agent-eval --full # adds the convex + browser smoke (~4 min)
pnpm agent-eval --json # machine-readable result line
The steps
- doctor: environment ready.
- scaffold a reshaped crm demo (supabase, mcp, --fields, --table/--labels).
- install + typecheck the generated app (tsc-clean).
- deploy dry-run (plans + builds).
- gallery spec: bare-name scaffold + typecheck.
- generate: extractJson parses fenced/nested/bare + buildPrompt shape.
- mcp-server: stdio handshake + tools/list exposes scaffold/generate/deploy.
- upgrade-app: dry-run round-trip on a fresh scaffold.
Self-cleaning; exits 0 only on a perfect score. Current: 9/9 in under a minute.