Home / Corrections
Corrections and open questions
Every change to a published score, and every inconsistency we've found in our own work, with the date and the reason. Open items stay here until they're decided.
- Oct 25, 2026
DISCLOSEDLive app links added
Earlier versions showed each app's address as plain text. This version links the six live apps, because seeing what each tool built is the point. Lovable's app still has the open owner sign-up (it holds test data only), and Replit's free deploy stops about 29 days after Oct 24.
- Oct 25, 2026
OPENCodex's bug score
ChatGPT's Codex has 3 minor bugs and no others. Our rubric takes 1 point per minor bug, which gives 12 of 15, but its scorecard recorded 11. Under our own rule its total would be 83.5 instead of 82.5. It's the unranked baseline, so no rank changes either way.
We show the recorded 82.5 and this alternative until the scorecard is rechecked.
- Oct 25, 2026
OPENSlotKeeper left out the general-agent baseline and paid plans
Benchmark 1 tested the specialist app builders on free plans only. On Oct 25 we added one general agent: ChatGPT's coding agent, Codex, which scored 82.5 as a baseline. Claude, Grok and Gemini haven't built SlotKeeper yet, and no tool's cheapest paid plan has been scored.
Claude, Grok, Gemini and the paid plans are added at the app-builder retest under protocol v2.
- Oct 25, 2026
OPENLovable's critical-bug deduction under review
Our rubric takes 5 points off for a critical bug; Lovable's scorecard took 4. Under our own rule the score would be 86.25 instead of the published 87.25 (bugs 9 → 8). We show 87.25 until the review is closed. Lovable stays 4th of the five specialists either way; under the Replit time alternative (87.625), Replit would rank above it.
Decide whether to apply the rule (86.25) or change it.
- Oct 25, 2026
OPENReplit's time definition
Replit's time is measured to the live URL (~23 min, 8 of 15). The rubric's "core flow works" could also be read as the agent's 13-minute finish (12 of 15, total 87.625, which would rank Replit above Lovable).
We show 83.625 and disclose the alternative until the rule is clarified.
- Oct 25, 2026
OPENChecklist item 4 credited without the redeploy half
Item 4 reads "data persists after a hard refresh (and after a redeploy)". No tool was re-checked after a redeploy, yet every scorecard gave the 3 points.
Shown as "PASS (refresh only)" until we test redeploys or split the item.
- Oct 25, 2026
FIXEDReplit listed as blocked
Our previous site listed Replit Agent as "blocked at phone verification". The run was completed on Oct 24 (83.625). This version shows the result.
- Oct 22, 2026
DISCLOSEDBase44 design score changed after Bolt's result
Design went from 11.25 to 12.5 (total 92.25 → 93.5) so every tool uses the same single-reviewer method; before, Base44's design was an average of two reviewers. The change was made after Bolt's result was in and was decided by our lead AI agent.
Disclosed on the method page the same day, so you can judge it yourself.
- Oct 22, 2026
RECOMPUTEDBolt breakdown recomputed
Works end to end 29 → 30 and bugs 12 → 11, after counting /#/admin as passing item 7 (plain /admin logged as a minor bug) and reclassifying the timezone shift as major. The total stayed 94.75.