← Ten breakthroughs that brought AI hereA2 · LESSON 7 OF 7

A2, Lesson 7: Agents: A Scorecard, and the Pattern Behind the Ten Stories

Tags: Sourced / General / Our view / Illustrative. About 14 minutes. No math or code. Fast-moving: every status claim carries an "as of" date and needs a scheduled re-check.

1. The story: "experimental, cumbersome and error-prone" (October 22, 2024)

On October 22, 2024, Anthropic announced "computer use" in public beta: developers could direct its model "to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text." Anthropic called it "the first frontier AI model to offer computer use in public beta," and said that "at this stage, it is still experimental—at times cumbersome and error-prone." On OSWorld, a test of using computers like people do, the company reported its model scored 14.9% in the screenshot-only category (per the verifier, the next best system scored 7.8%, and the company reported 22.0% when given more steps). [Sourced: Anthropic, "Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku," Oct 22, 2024, https://www.anthropic.com/news/3-5-models-and-computer-use ; company's own claims and benchmark result] That is the right tone for this lesson. An "agent" is software that does not just answer but acts: it takes steps, uses tools and keeps going toward a goal. This is where the ten stories (Transformer, pretraining, scale, assistants, images, open models, reasoning, tools, and now agents) meet. And it is where claims run furthest ahead of evidence. [Our view]

2. The one idea

An agent is an assistant (Lesson 4) that can think in steps (Lesson 6), use tools (Lesson 6), and repeat until it decides it is done. Everything that can go wrong in the earlier lessons can go wrong at each step, and the steps multiply the chances. Judge agents by measured, independent results on tasks like yours, and by how reliably they repeat. [General; Our view]

3. The kitchen-table version

You hand a new assistant a to-do list and a company card. A good one finishes the list. A mediocre one finishes some items, forgets others and occasionally buys the wrong thing. The question is not whether she can do it once, but whether she does it right every time without you watching. [General]

4. A scorecard of dated research results (as of October 7, 2026; sources rechecked)

These are historical paper results, not a ranking of the best agents available in October 2026. These are research benchmarks from university and other groups; they are one way to measure, not the final word. Results change quickly and each benchmark has its own definitions. Treat every number as a dated snapshot.

  • Office-style tasks: Carnegie Mellon's TheAgentCompany tests agents on tasks in a simulated small software company (browsing, coding, running programs, messaging coworkers). The cited paper reports its best tested agent, Gemini 2.5 Pro, completing 30.3% of tasks autonomously (39.3% when partial credit is included), and the authors say simpler tasks can be solved while "more difficult long-horizon tasks are still beyond the reach of current systems." [Sourced: https://arxiv.org/abs/2412.14161 ; historical paper result, source rechecked October 7, 2026]
  • Rules and customers: Sierra's τ-bench simulates conversations between a user and an agent that must follow company rules with tools. The authors report that leading function-calling agents of mid-2024 (such as gpt-4o) succeeded on fewer than 50% of tasks and were "quite inconsistent" (pass^8 below 25% in retail), and propose "pass^k," a measure of reliability across repeated tries. These are results for the models tested then, not today's. [Sourced: https://arxiv.org/abs/2406.12045 ; Sierra authorship confirmed by the author affiliation on the paper]
  • How long a task, how often: METR, a nonprofit that evaluates AI, proposed measuring the length of tasks, in human time, that models complete 50% of the time. Its March 2025 paper reported about 110 minutes for o3, a horizon that "has doubled approximately every seven months since 2019" and a trend that "may have accelerated since 2024." That was an early-2025 figure. Per the verifier, METR's January 29, 2026 update (TH1.1) keeps the roughly seven-month doubling over the long run but post-2023 fits are about four to five months, and METR's live page (https://metr.org/time-horizons/) now warns that "Measurements above 16 hrs are unreliable with our current task suite" (I read that warning on the page myself). The paper also says that "naively extrapolating" the trend gives a one-month (167 work hour) horizon between mid-2028 and mid-2031 (Figure 6), cautions about external validity, and says the trend may change. [Sourced: https://arxiv.org/abs/2503.14499 ; forecast with stated caveats, not a result about your business; the TH1.1 details are per the verifier] Read these together: agents are real and improving fast; at 50% success they are not yet dependable; and "can do it once" is not "can do it every time." Any specific number here will be out of date soon. [Our view]

5. Worked example: why reliability matters more than the headline

All numbers [Illustrative], computed by script. Same task, repeated. Suppose an agent succeeds 60% of the time on a task, independently each try. The chance it succeeds every time in 2 tries is 36%; in 4, 13%; in 8, 1.7%. (This is the idea behind pass^k in τ-bench; the real benchmark measures it directly.) For a task you repeat daily, that is the number that matters. Many steps. At 95% per step, a 5-step job completes correctly 77.4% of the time; 10 steps, 59.9%; 20 steps, 35.8%. How long until the horizon reaches a work day? If a 110-minute horizon doubled every 7 months (METR's stated trend, which may not continue), 8 hours would take about 14.9 months, and a month of work (about 10,000 minutes) about 45.5 months. This is arithmetic on a trend, not a prediction we endorse. [Our view]

6. What it costs and what it breaks

  • Compounding errors: see the example.
  • Actions with consequences: a wrong answer wastes a minute; a wrong action sends money or email.
  • Security: agents that read untrusted pages or messages can be tricked into doing things the owner did not ask. The US government's NIST standards center wrote in January 2025 that "many AI agents are vulnerable" to this, called agent hijacking: an attacker hides instructions in an email, file or web page the agent reads. Treat what an agent reads as untrusted. [Sourced: NIST Center for AI Standards and Innovation technical blog, Jan 17, 2025, https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations ; the advice to treat content as untrusted is ours]
  • Accountability: someone remains responsible for what the agent did.
  • Oversight cost: checking an agent's work takes time; if checking costs as much as doing, you gain nothing.
  • Hype: claims about "digital employees" run far ahead of measured results; ask for independent tests on your type of task.

7. The pattern behind the ten stories

Looking back across Lessons 1 to 7:

  1. A narrow tool (translation) turned out to be a general design. (Lesson 1)
  2. Learn from text first, specialize second. (Lesson 2)
  3. Scale helped predictably, with diminishing returns, and the recipe was revised. (Lesson 3)
  4. Human judgment turned a text engine into an assistant. (Lesson 4)
  5. The same ideas spread to images, and the weights spread to more hands. (Lesson 5)
  6. Thinking in steps and using tools made multi-step work possible. (Lesson 6)
  7. Agents combine them, and reliability is the open problem. (this lesson) In every story, a builder's announcement came first and independent measurement came later, often with a more modest picture. That is the habit worth keeping: read the announcement, then look for the test someone else ran. [Our view]

8. A short note on what people say comes next

Some researchers and company leaders expect far more capable systems, sometimes called superintelligence, in the coming years. Others expect slower progress or hard limits. These are forecasts, not results, and reasonable people disagree. This course does not take a side; it asks you to judge claims by evidence at the time you read them. [Our view]

9. Monday checklist

  1. List one repeating task you would trust an agent with. What is the cost of one wrong action?
  2. Ask any vendor for independent test results on tasks like yours, and for repeat-run reliability.
  3. Start with agents that suggest, not agents that act.
  4. Limit the money, messages and deletions an agent can do without approval.
  5. Re-read this lesson in three months; the scorecard will have changed.

10. Try it yourself (no code, 30 minutes)

Write the steps of a task you do weekly (for example, "answer a supplier question"). Mark each step "safe to automate," "needs a check," or "never automate." Estimate the chance each step goes right and multiply them. Is that good enough?

11. Self-check

  1. What did Anthropic call computer use at launch? "Experimental, at times cumbersome and error-prone."
  2. What does pass^k measure? Whether an agent succeeds on every one of k repeated tries.
  3. What did the TheAgentCompany abstract say the best agent completed? 30% of tasks autonomously, as read.
  4. If each of 10 steps is 95% right, how often is the whole job right? About 60%.
  5. What habit does the course suggest for any announcement? Look for an independent test.

Sources

Review flags

  • The cited TheAgentCompany, τ-bench and METR paper results were rechecked against the live source texts on October 7, 2026. They describe the models and versions in those papers, not the best current agents. TheAgentCompany distinguishes full completion (30.3%) from its partial-credit score (39.3%). The live METR page was also checked and still warns that measurements above 16 hours are unreliable; its 2026 model updates do not turn the early-2025 o3 figure into a current-model result. No claim about current third-party leaderboard rankings is made.
  • No test result newer than the models in those papers is covered, so the scorecard understates current systems if they have improved. A scheduled re-check with new independent tests is required.
  • The Anthropic opener is the company's own program; I did not check any later computer-use results or the OSWorld human baseline.
  • The METR trend and extrapolation are METR's stated, caveated forecast; I do not endorse them. The extrapolation phrasing was checked against the paper's text for Figure 6.
  • RESOLVED (Oct 7, 2026): the agent-security statement now cites the NIST Center for AI Standards and Innovation technical blog of Jan 17, 2025 (page read; "many AI agents are vulnerable to agent hijacking" is verbatim). The "treat what an agent reads as untrusted" advice is tagged as ours.
  • The superintelligence note is short and neutral by design; it cites no source.
  • All reliability arithmetic is illustrative and assumes independent tries.
  • Sierra authorship of τ-bench is confirmed by the paper's affiliation line.

Claim ledger (A2 Lesson 7)

#ClaimSourceVerifiedStatusRe-check
1Oct 22, 2024: computer use in public beta; "experimental... error-prone"; OSWorld 14.9% screenshot-only; next best 7.8%, 22.0% with more steps (per verifier)Anthropic post2026-10-06Read; company claimLater results
2TheAgentCompany: paper's best tested agent Gemini 2.5 Pro 30.3% full completion, 39.3% partial-credit scorearXiv 2412.141612026-10-07Paper results rechecked October 7, 2026; not a current-model rankingScheduled
3tau-bench: under 50% success; pass^8 below 25% in retail; pass^k definedarXiv 2406.120452026-10-07Read abstractScheduled
4METR: ~110 min (o3, early 2025); doubling ~7 months since 2019; naive extrapolation to one month mid-2028 to mid-2031 (Fig. 6); TH1.1 Jan 29, 2026 and 4-5 month recent fits (per verifier); live page warns above 16 hrs unreliablearXiv 2503.14499; metr.org/time-horizons2026-10-07Paper read; warning read on page; TH1.1 per verifierScheduled
6Agents can be hijacked by instructions hidden in content they readNIST CAISI blog, Jan 17, 20252026-10-07Readnone
5Reliability, compounding and horizon arithmeticOur computation2026-10-06Illustrative, script-checkednone