Supabase has launched "Supabase Evals," an innovative open-source benchmark and framework designed to rigorously evaluate AI coding agents on practical Supabase development tasks. Released under the Apache-2.0 license, Evals tests agents like Claude Code, Codex, and OpenCode against real-world scenarios such as building schemas, debugging Edge Functions, or fixing RLS policies. The framework operates by deploying agents within a containerized environment that mimics a full Supabase stack, enabling interaction with actual MCP servers and CLI tools. Scenarios, derived from real support tickets and GitHub issues, cover diverse Supabase products, topics, and development stages (build, deploy, investigate, resolve). Each evaluation includes a PROMPT.md defining the task and an EVAL.ts script for scoring, which blends deterministic checks with an LLM-as-a-judge approach. This robust setup reveals agent weaknesses, such as a tendency to hand-write migrations instead of using declarative schemas or inconsistent documentation usage, even with loaded skills.
For developers and researchers, Supabase Evals is a crucial tool for advancing AI-assisted development. By offering an open, reproducible, and practical evaluation method, it provides invaluable insights into the real-world capabilities and limitations of current AI models. The benchmark's findings directly inform skill guidance updates and package selection recommendations, fostering the creation of more capable and context-aware AI agents. This framework empowers the community to test, compare, and ultimately accelerate the development of AI tools that genuinely streamline the developer experience on platforms like Supabase, driving innovation in how AI supports software engineering.