How we test vibe-coding tools
Every vibe-coding tool looks good in its own demo. To compare them fairly, we give each one the same real app to build with the same prompt and rules. Then we check it against the same list and score it on the same rubric. This page explains exactly how.
Benchmark #1: SlotKeeper
A solo service business (a yoga teacher, tutor or barber) needs a public booking page. SlotKeeper covers auth, a database, forms, validation, role-based access and a deploy: the things a non-technical founder actually needs.
Public side (no login): - A landing page with the business name "Maple Yoga Studio" and upcoming slots (date, start time, duration, spots left). - A booking form: full name, email, phone (optional), notes (optional). Name and a valid email are required. - Booking a slot with spots left reduces spots by 1 and shows a confirmation with the booking details. - A full slot shows "Join waitlist", and the same form adds the person to the waitlist with their position number. - The same email can't book the same slot twice.
Owner side (login required): - Email and password sign-up and login. Logged-out visitors to /admin are redirected to login. - Create, edit and delete slots (date, start time, duration in minutes, capacity). - See bookings and the waitlist for each slot. Cancelling a booking automatically moves the first waitlisted person into it. - A stats line: total upcoming slots, total bookings, total waitlisted.
Requirements: a real persistent database (data survives a refresh and a redeploy), a clean, mobile-friendly design, 3 seeded slots in the next 7 days (one with capacity 1), and deployment to a public URL.
The rules
- One prompt, pasted verbatim into every tool.
- Up to 5 fix prompts. A fix prompt may only describe observed behaviour. It can't give code, name a library or describe the solution. Each click of a built-in "auto-fix" button counts as one fix prompt.
- Clarifying questions get "Use your best judgment", which doesn't count as a fix.
- Setup the tool asks for (connecting its own database, clicking Deploy) is allowed and logged.
- No hand-editing of code. For pro baselines (Cursor, Claude Code), running terminal commands the agent asks for is allowed, but hand-editing code is not.
- Hard stop: 90 minutes wall-clock or 5 fixes, whichever comes first.
- Defaults: we use the model and settings the tool picks, and record which model.
- Clean setup: a fresh account on a dedicated BenchCoded email, a clean browser profile, screen recording and a stopwatch.
- Plan tier: Free tier first; we pay for the lowest paid tier only if the free tier can't finish the build, and we say which tier each review used. We record the plan tier and starting credits.
- Final check: after deploying, we rerun the checklist on the public URL from a phone (or 390px viewport) and an incognito window.
The 10-item checklist (3 points each)
- Landing page lists the 3 seeded slots with spots left.
- Booking with a valid name and email succeeds, a confirmation shows, and spots go down by 1.
- A missing name or invalid email is rejected with a clear message.
- Data persists after a hard refresh (and after a redeploy).
- A full slot (capacity 1, after one booking) shows "Join waitlist" and assigns a waitlist position.
- The same email can't double-book the same slot.
- The owner can sign up and log in, and logged-out /admin redirects to login.
- The owner can create, edit and delete a slot.
- Cancelling a booking promotes the first waitlisted person automatically.
- Stats line counts are correct, and it all works on the public deployed URL.
Extra security probe (a failure counts as a bug): a second owner account or an anonymous user can't read or modify another owner's data through the UI or by URL tampering, and no database keys are exposed in the client.
Scoring rubric (100 points)
| Category | Points | How it's scored |
|---|---|---|
| Works end to end | 30 | 3 points per checklist item passed |
| Time to first working version | 15 | From submit until a public booking is saved and owner login works. ≤10 min = 15, ≤20 = 12, ≤35 = 8, ≤60 = 4, >60 or never = 0 |
| Bugs | 15 | Start at 15 and subtract by severity (see below). Floor 0 |
| Design / UX | 15 | Scored by one reviewer for now (see below) |
| Deploy ease | 15 | 15 = one-click in-tool deploy with a working public URL. 10 = in-tool, but 1–2 extra steps (connect DB, env vars). 5 = needs an external service or CLI set up by the tester. 0 = no public URL |
| Cost / credits used | 10 | Credits, tokens or messages converted to $ at the plan's rate. ≤$2 = 10, ≤$5 = 8, ≤$10 = 6, ≤$25 = 3, >$25 = 0. A full pass on the free tier earns a "Free-tier pass" badge (no points) |
| Total | 100 |
Pro baselines (Cursor, Claude Code) are scored on the same rubric but reported separately as reference lines, not ranked against the no-code tools.
Design sub-rubric
For now, one reviewer scores three things from 1 to 5: visual polish, mobile responsiveness, and clarity of flows (form errors, confirmations). 5 = could ship to a client today, 3 = generic but usable, 1 = broken or unstyled.
The raw total (3–15) is mapped linearly: (raw − 3) / 12 × 15, so a raw 3–15 becomes 0–15. Design is currently scored by one reviewer, and we say so in every review. Having two reviewers and averaging their scores is a future goal.
Bug severity
| Severity | Examples | Deduction |
|---|---|---|
| Critical | Data loss, auth bypass, crash on a core flow | −5 |
| Major | Wrong spot count, duplicate booking allowed, broken mobile layout | −2 |
| Minor | Typo, cosmetic glitch | −1 |
We also record how many fix prompts each tool used.
Affiliate links and independence
Some links may be affiliate links; we may earn a commission at no cost to you. Scores are never affected. Every tool gets the same prompt, rules, checklist and rubric.
FAQ
How does BenchCoded test vibe-coding tools? Every tool gets the same SlotKeeper prompt, pasted word for word. After that we allow up to 5 fix prompts that only describe observed behaviour, no hand-editing of code, and a 90-minute hard stop. Then we run the same 10-item checklist and score the result on a 100-point rubric.
How is the design score calculated? For now, one reviewer scores visual polish, mobile responsiveness and clarity of flows from 1 to 5. The 3–15 total is mapped linearly to 0–15 with (raw − 3) / 12 × 15. We disclose this in every review; two reviewers is a future goal.
What counts as a critical, major or minor bug? Critical bugs (data loss, auth bypass, crash on a core flow) cost 5 points. Major bugs (wrong spot count, duplicate booking allowed, broken mobile layout) cost 2 points. Minor bugs (typo, cosmetic glitch) cost 1 point. The bug score starts at 15 and can't go below 0.
Do affiliate links affect BenchCoded scores? Some links may be affiliate links; we may earn a commission at no cost to you. Scores are never affected.
Score changes
- Oct 22, 2026 — Base44 (SlotKeeper): design changed from 11.25/15 to 12.5/15, and the total from 92.25 to 93.5/100. Why: we moved every tool to the same single-reviewer design method. Before, Base44's design score was an average of two reviewers, while Bolt's came from one. We made this change after Bolt's result was in, and we're telling you so you can judge it yourself.