Home / Categories / Automation
PLANNED Category 4 · draft order
Automation agents: the 7-day workflow test
Inbox triage is too easy: any agent can do it now. So our automation test is a messy business workflow across five apps that has to keep working for 7 days.
NotePlanned. Nothing has been run yet and no results exist. Tools are candidates, confirmed on test day.
§01
The workflow (same scripted test orders for every tool)
- A new order arrives in a Shopify test store.
- A row is added to a spreadsheet.
- An invoice is created.
- The team is alerted in Slack.
- The customer gets an email 3 days later.
- A refund request is handled.
§02
Who's tested
- Specialists (candidates)
- Zapier, Make, n8n, Lindy
- General agents (baseline)
- ChatGPT, Claude, Grok and Gemini
- Plans
- Each tool on its free plan and its cheapest paid plan.
§03
What we score
- Breaks: every step that didn't happen, happened twice, or happened wrong, over the 7 days.
- Human step-ins: how often a person had to fix or restart something, and the minutes it took.
- Cost: the plan price on the test day plus any usage charges.
- Setup time: minutes to build the workflow the first time.
The full rubric is published here before the run starts.
§04
Safety rules
- Everything runs in test mode: a Shopify development store, test invoices and refunds, a private Slack workspace.
- The "customers" are our own test inboxes. No real people, orders or money are involved.
- Every account and any spend goes through the owner's approval first.
§05
Do you even need Zapier anymore?
General agents can now operate apps for you. The open question is whether they keep a business workflow running for a week without someone babysitting it. That is what this test measures.