Part 1: What an AI Agent Actually Is: From Answering to Acting

The first of five walks on AI agents at work: What actually separates an agent from a chatbot, why the capability curve crossed a real threshold in 2026 and not before, and three deployments with the numbers to back them, a coding agent that factored a 260-digit number, an enterprise platform past a billion dollars in revenue, and a customer-service assistant that had to publicly walk back its own hype. Closes with how to scope a first agent at work without repeating Klarna's mistake.

Tue, 06 Oct 2026 · 15 min

Made for you, by AI, from the sources below.

Updated in October 2026: METR's latest measurement is now Claude Mythos, at least 16 hours, published in May. Salesforce now reports 7 billion units of agent work to date.

Updated in October 2026

Listen to it in AI Agents at Work

Transcript

The script the two voices were given, with the sound cues taken out. A voice may say a word a little differently, so the audio is the episode.

Mia: Welcome to Walking Points. In September, a coding program did something worth pausing on: it ran a real research project completely on its own, for three weeks straight, with no one steering it turn by turn.

Adam: Three weeks. With nobody at the wheel?

Mia: Nobody doing the actual driving, no. A person set the goal and stepped in only when it drifted off track. Everything in between, choosing the method, running the numbers, noticing when something broke and fixing it, the software did itself. This part was updated in October 2026.

Adam: Okay, that's not what I picture when someone says "chatbot."

Mia: It isn't one. And that gap, between a chatbot and whatever that coding program actually is, is the whole subject of this episode, and this whole series. This is part one of five on AI agents at work, and today we're answering the plain question underneath all of it: what an AI agent actually is, in 2026, and why this specific year is when it stopped being a demo and started being something businesses hand real work to. We'll get there in three steps. First, what actually changed between a chatbot and an agent, technically. Then why this year, and not last year. And then the deployments that prove it, including the one that got ahead of itself and had to walk part of it back.

Adam: Start with the technical bit, then. What's actually different under the hood?

Mia: A chatbot, in the plain sense, reads a question and writes an answer. It has no memory between sessions, it can't check anything beyond what you typed in, and it can't touch anything outside that conversation window. An agent is built the same way underneath, usually the exact same underlying model, but wrapped in three things a chatbot doesn't have. Tools it can use, memory that carries across many steps instead of just one reply, and a loop that keeps it working instead of stopping after a single answer. One industry comparison put the whole difference in four words: chatbots read, agents act.

Adam: Tools meaning what, exactly? Like, plug-ins?

Mia: More than that. Give a model the ability to query a database, send an email, edit a file, or actually click around on a screen, and it stops being limited to what it can say. A model operating a mouse and keyboard on a real screen, rather than someone writing custom code for every single app, is what the industry calls computer use. Anthropic shipped the first serious version of that in October 2024. By this year, every major lab has its own take on the same idea.

Adam: So it's not typing suggestions into a box anymore. It's actually driving the software.

Mia: Right. And that's tools. Memory is the second piece: the agent keeps a working record, across the many steps a real task takes, of what it's already tried, what the actual goal is, and what it's learned along the way. Not memory like a diary. Memory like a project file it keeps updating.

Adam: And the loop's the third thing?

Mia: The loop is where the first two actually combine. Anthropic's own engineers call it the agentic loop: the model looks at where the task stands, decides what to do next, takes one action through a tool, looks at the result, and does it again, over and over, deciding for itself when the job is actually done instead of stopping after some fixed number of steps.

Adam: Deciding for itself when it's done sounds like exactly where this could go wrong.

Mia: It is. And a reasoning model, one that plans and checks its own working before it commits to an answer, rather than just running with the first plausible-looking move, makes that loop a lot more reliable. Because most of what goes wrong on a long task like this isn't the model not knowing something. It's the model committing to a bad plan early and never noticing.

Adam: Okay, so tools, memory, a loop that knows when to stop. That's the recipe.

Mia: Almost. There's one more piece, and it's less flashy but it's the reason any of this scaled. Before this existed, if a company wanted an agent to actually touch its own software, someone had to write custom integration code for every tool, for every model, by hand. Anthropic published something called the Model Context Protocol, MCP for short, the following month: an open standard so any tool that speaks it can be picked up by any model that speaks it. Think of it like a USB port. It doesn't matter who made the device or who made the port.

Adam: And that actually caught on?

Mia: Google, Microsoft, OpenAI and Salesforce all built support for it within about a year. In December last year, Anthropic handed control of the standard over to a new foundation under the Linux Foundation, so it's shared infrastructure now, not one company's thing to gatekeep. By this year, there were more than ten thousand of these connections running in production.

Adam: Huh. So the boring standard is doing a lot of the work.

Mia: It usually is. Put all four pieces together, tools to act, memory to persist, a loop that decides when it's finished, and a shared standard so nobody has to wire it all up from scratch, and you don't get a smarter chatbot. You get something you can hand a goal to, walk away from for a real stretch of time, and come back to find either a finished result, or a clear account of what got in the way. That's the entire shift the word "agent" is carrying in 2026. From answering, to acting.

Adam: Okay, so that's what changed technically. Why did it take until this year to matter?

Mia: For that, the cleanest evidence comes from a nonprofit called METR, which has spent years measuring something specific: not how well a model answers test questions, but how long a task you can trust it to finish completely on its own.

Adam: How do you even measure that?

Mia: They call it the time horizon: the length of a task, measured in how long it'd take a skilled person, at which a given model succeeds about half the time. It sounds academic until you realize it's answering the question that actually matters if you're handing something real to one of these: not "is it smart," but "how long can I leave it alone."

Adam: So what's the number?

Mia: In METR's latest measurement, from May this year, Anthropic's Claude Mythos reached a time horizon of at least sixteen hours of continuous, unsupervised work. Back when we first made this part in September 2026, the figure we used was an earlier model, Claude Opus 4.5, at over five hours. Go back to the GPT-4 generation from 2024, and the same measurement reads three and a half minutes.

Adam: Wait, hold on. Sixteen hours versus three and a half minutes?

Mia: Same test, about two years apart. And it's not a one-off jump, it's a trend. Across the years METR's been tracking this, the time a model can be trusted with roughly doubles every seven months. Since then specifically, that doubling has sped up to about once a quarter.

Adam: A doubling every quarter doesn't sound dramatic, day to day.

Mia: It doesn't, until it crosses a line that matters to somebody's actual working day. An hour. A morning. A full day. This year is when several of those lines got crossed, for enough kinds of tasks, that agents stopped being something you sat and watched work, and became something you could plausibly leave running.

Adam: So if it can genuinely run for hours unsupervised now, the obvious question is: who's actually using that, for real work, not a demo?

Mia: That's exactly the turn. And it's also where the one company that got ahead of itself comes in, later. Let's start with the one we opened on.

Mia: That three-week project from the top of the episode was Devin, a coding agent built by a company called Cognition. In September, they published an account of using it to factor a 260-digit number called RSA-260, a benchmark cryptographers have used for decades to measure how much computing effort a method actually needs.

Adam: And it did all of that itself?

Mia: Devin ran the whole thing, picked the mathematical approach, ran the computation, recovered when parts of it failed, across 233 separate working sessions. The human involved only set the goal and redirected it when it drifted, what Cognition calls executive function, rather than doing any of the actual work.

Adam: It's not exactly a task any business needs done, though, is it.

Mia: No, and that's deliberate. It's a stress test: can the thing hold a plan for three weeks without a person rewriting it. And the business numbers around Devin moved almost as fast as the demo. In May, Cognition reported Devin's revenue running at about 492 million dollars a year. By its funding round in September, that had grown to almost 900 million, in four months, at a 48 billion dollar valuation.

Adam: That's not experimentation money. That's production money.

Mia: Right, and they named the teams: engineering groups at companies like NVIDIA and GE Aerospace using it in production, not just software companies working on software.

Adam: Is coding the exception, or does the same thing show up elsewhere?

Mia: Same pattern, different building. Salesforce's Agentforce, which lets companies build agents that act inside their own support and sales systems, reported an annual recurring revenue of about one point two billion dollars, as of the quarter ending in April, up 205 percent from the year before. That's a run rate, not money that landed in one quarter. Customers have also run 7 billion units of agent work through Agentforce and Slack combined to date, 3.2 billion of them in the latest quarter alone.

Adam: Billion, with a B, for units of work. What does a unit even mean there?

Mia: Salesforce doesn't actually spell that out, just that it's their own internal count. Whatever the exact definition, that scale is real, and it wasn't only their own tooling doing the spreading. Slack's version of that MCP standard passed a million active users within six weeks of launch. The standard that made agents technically possible is also what's making them spread fastest once one shows up inside a company that already has one.

Adam: Okay, but you said one of these overreached. Which one?

Mia: Klarna. Their AI assistant, built on OpenAI's models, went live for every customer in February, over two years back, and in its first month it resolved two thirds of its support chats, work OpenAI's own case study on the deployment compared to the workload of around 700 human agents.

Adam: That's a real number, not a demo number.

Mia: It's real, and for a while Klarna leaned hard into the idea that this basically replaced the job. But by last year that framing had visibly cracked. Complaints about quality, and customers wanting an actual person for anything sensitive or complicated, pushed them to rehire and bring guaranteed human agents back into the loop.

Adam: Mia, so the poster child for automated support ended up hiring humans back?

Mia: Pretty much. By last summer, their chief executive had landed on a different way of putting it: that human customer service would almost become, in his own words, a kind of VIP thing. Something you get routed to, not something you default to.

Adam: That's a pretty big walk-back to say out loud, in public.

Mia: It is, and that's actually the useful part. The volume figures didn't disappear, that two-thirds resolution rate is still real. What changed is the honesty about where the line sits. And that pattern, real capability, real revenue, and a visible correction somewhere in the history, is a much better description of where this actually stands than either the pure hype version or the pure skeptic version.

Adam: So where does that leave someone who isn't running a coding company or a support desk, just trying to figure out whether to hand an agent anything at all?

Mia: With a real scoping decision to make, honestly, the same kind of decision you'd make delegating something to someone new on the job, not a toy you try once and judge off a single result. Adam, the thing that separates Cognition and Salesforce from Klarna's early version is exactly what Cognition calls executive function: someone still sets the goal, checks in at defined points, and decides what actually needs a person, instead of assuming the agent's first attempt is the final one.

Adam: Is that just Cognition's marketing, or does it show up elsewhere too?

Mia: It shows up in the survey data too. Enterprise surveys through this year, including McKinsey's global look at AI adoption, keep finding the same gap: plenty of individuals report real productivity gains, but only a minority of companies have actually moved from experimenting with agents to running them at scale in any one part of the business. And the shortfall tracks how mature the oversight is, not how capable the model is.

Adam: So capability isn't really the bottleneck anymore.

Mia: Not the way it was even a year ago. The scoping and the checking-in around it usually still is.

Mia: If you run any kind of small business, a training business, a clinic, a shop with its own back office, the practical version of this is: don't start with the task that would save the most time if it worked. Start with the task where a wrong answer is cheap and easy to check. And if you manage people rather than code, that same idea just moves up a level: it's the difference between handing someone a whole project unsupervised on day one, and handing them something small and checkable, with a moment where you look at it before it goes out the door.

Adam: That's basically what Klarna learned the hard way. Decide in advance what needs a person.

Mia: Which gives you something you could actually try. Pick one task you already do by hand that has a clear right answer, something you could check in five minutes, not five days. Hand only that one to an agent, and build in the same kind of checkpoint Cognition and Salesforce both use: a defined moment where a person looks at the output before anyone else sees it, instead of trusting its own sense that it's finished.

Adam: Small, checkable, and somebody still looks before it ships.

Mia: Exactly. And that's really the question for next time: once you get past the headline numbers, do agents actually earn their keep on the tasks companies are handing them, coding, support, the back office, finance work, or does the measured result turn out harder than the demo promised?

Adam: Because being able to run for sixteen hours, and being worth running for sixteen hours, are not the same claim.

Mia: That was part one of five. Part two is called Where Agents Earn Their Keep, and it's about what the measured results say, once you get past the demos.

Mia: Want a show like this about your own work? You'll find it at walkingpoints dot F M.

Adam: Thanks for walking with us!

Mia: See you on the next walk.

Heard something wrong? Tell us