Part 2: Where Agents Earn Their Keep

Mia and Adam trade the demo stage for the measured evidence on AI agents at work, covering software engineering, customer service, and back office and finance. A controlled study found professional coders got slower while believing they had sped up, a widely quoted customer-service number turns out to mask who actually benefits, and the back office quietly delivers more return than the flashier pilots that make the headlines. It closes on why some tasks keep turning out harder than promised, and three questions worth asking before trusting any agent's numbers.

Tue, 13 Oct 2026 · 17 min

Made for you, by AI, from the sources below.

Listen to it in AI Agents at Work

Transcript

The script the two voices were given, with the sound cues taken out. A voice may say a word a little differently, so the audio is the episode.

Mia: Welcome to Walking Points. Picture sixteen experienced developers, each with years on the exact codebases they were about to work in, given real bugs to fix under a stopwatch, some allowed to use AI tools, some not.

Adam: And the AI helped, presumably.

Mia: It didn't. Using it made them nineteen percent slower, start to finish. Before the test, those same developers had guessed AI tools would cut their time by about twenty-four percent.

Adam: So they expected a big win and got a loss instead.

Mia: A loss they didn't even notice. Afterward, still moving slower with a clock literally running on them, they estimated AI had made them roughly twenty percent faster.

Adam: The measured result and the gut feeling pointed in opposite directions.

Mia: Almost exactly opposite, by nearly the same margin each way. That gap is why this episode exists: what AI agents, systems built to carry out a multi-step task on their own rather than just answer a question, actually deliver once somebody puts a stopwatch on them instead of a demo stage. We're looking at three functions where agents have been let loose at scale: software engineering, customer service, and back-office and finance work.

Adam: Three very different jobs.

Mia: Which is the point. We'll go through software engineering first, because it has the best paper trail, then customer service, where there's one number everyone quotes and it hides a lot, then back-office and finance, where the real payoff turns out to be sitting quietly, nowhere near a slide deck. Last Tuesday we covered what an agent even is, the shift from a system that only answers to one that acts, so this is the next step: what that shift actually earns.

Adam: So, software engineering. Where's the paper trail?

Mia: JPMorgan, because they've published real internal numbers instead of a vendor's slide. Their global chief information officer, Lori Beer, has said the bank's own AI coding assistant is giving software engineers a genuine, double-digit efficiency gain, across a technology workforce of sixty-three thousand people. More recently, the bank has gone further, saying the gains are most visible in software development and back-office operations, and putting a number on it: one and a half to two billion dollars a year in value it attributes directly to AI, measured against specific business lines and budgets, not estimated in the abstract.

Adam: Measured how, though? Attributed to what, exactly?

Mia: To actual teams and actual budgets, which is the unusual part. Most companies wave at a productivity number. This bank is assigning it to real business lines it can check against its own books.

Adam: Is there a number that precise for an ordinary software team, not just one bank?

Mia: There's a broader one. Stanford's AI Index, the university's big annual research report on the state of the field, measured productivity gains of about twenty-six percent specifically on software development tasks. The catch: those gains shrink fast wherever a task needs real reasoning instead of just following a familiar pattern.

Adam: Wait, twenty-six percent gains, and junior developers are struggling to find work?

Mia: Same report, and yes, that's the uncomfortable part. Employment for the youngest software developers has fallen by nearly a fifth in the last couple of years, concentrated in exactly the well-specified, junior-level coding work agents now handle reasonably well. About a third of employers surveyed expect to cut further next year.

Adam: So the productivity gain and the job losses are the same story, told from two different desks.

Mia: There's a follow-up worth adding. The same research group behind that first stopwatch study surveyed hundreds of technical workers about how much value they feel AI adds to their own work. The median answer was something like one and a half to two times their output. Asked to estimate their own speed gains directly, though, they said three times, a bigger number the researchers themselves flagged as inflated, because their earlier work found people overestimate AI's effect by around forty percentage points.

Adam: So even the people studying the overestimate are, on average, overestimating.

Mia: A little, yes. And Adam, worth being clear: that group's own forecast, that the multiplier keeps climbing next year, is a forecast. Nobody's verified a year that hasn't happened yet.

Adam: Fair. What about the benchmark numbers, the ones companies actually put on stage?

Mia: Cognition, the company behind the autonomous coding agent Devin, a system meant to take a coding ticket and finish it with minimal help, published results for its newest model putting it at about seventy-eight percent accuracy on a widely used real-bug benchmark, and around eighty-two percent on a benchmark of command-line tasks. Those sound like great numbers.

Adam: They do.

Mia: They're demo-stage numbers, though. On a separate, tougher benchmark built specifically to resist that kind of gaming, the same model's score drops to about forty-two percent, essentially tied with OpenAI's GPT-5.5 and just a little behind Anthropic's own Claude Opus 4.8. Cognition's own account of everyday use is more modest still: teams running a modest handful of well-scoped tasks a month through it, well short of benchmark speed, with reliability dropping sharply once a ticket turns ambiguous or touches unfamiliar, complicated code.

Adam: So where's the catch? Every gain so far has come with one.

Mia: The catch is what the code looks like afterward. A code-analytics firm called GitClear went through more than six hundred million code changes and tracked quality, not just volume.

Adam: And?

Mia: Copy-pasted code, instead of properly reused code, is up forty-one percent within a single commit compared to a few years ago. Genuinely refactored code, where someone actually cleans up the structure, fell from twenty-one percent of all changes to under four percent. Code that quietly papers over an error instead of handling it is up forty-seven percent. And code that gets rewritten again within two weeks of being written, a rough sign it wasn't right the first time, is up another fifteen percent.

Adam: So it's genuinely faster and genuinely getting messier, at the same time.

Mia: That's GitClear's own read: a real twenty-five percent velocity gain, but a workflow optimized to ship something fast, not to leave the code easy to own three years from now. Think of it like a mortgage: the fast build lowers this month's payment and raises the balance you owe later.

Adam: Except with a mortgage you at least get a statement telling you the balance. Here, nobody sends anyone that bill until the thing actually breaks.

Mia: Exactly, the debt is real, it's just invisible until someone has to work inside that code. If you manage a team that ships software, the thing worth watching isn't whether AI made this sprint faster, it's whether your two-week churn, how much code gets rewritten right after it's written, is quietly climbing. That number tells you whether today's speed is costing you maintainability later.

Adam: Okay. That's engineering. What about customer service?

Mia: There's one number behind almost every executive's slide about it, and it traces back to a well-designed academic study. Economists tracked a generative AI assistant rolled out, at different times, to just over five thousand customer support agents at a large software company. Average productivity, measured as issues resolved per hour, rose fourteen percent.

Adam: That sounds solid. I can already tell there's a catch coming.

Mia: There is. That average hides a split: the newest, least experienced agents improved by thirty-four percent. The most experienced agents barely moved at all. The mechanism seems to be that the assistant spread the best agents' habits to everyone else, so it raised the floor a lot more than it raised the ceiling.

Adam: So it's less that AI makes everyone better, and more that it makes the newest person perform like someone with a lot more experience.

Mia: That's a good way to put it. That study is from 2023, old enough to predate today's fully autonomous customer-service agents, and Stanford's own newer figure for customer support, years later, lands almost exactly on top of the same number.

Adam: Which brings us back to Klarna.

Mia: Right. Klarna's actual public claim was that its assistant did the work of roughly seven hundred human agents, at a time it had already cut its support staff. It later walked that back and rehired people for part of the work, after finding that resolution quality mattered to customers more than raw throughput.

Adam: So the AI wasn't useless. It just got asked to prove something bigger than what was actually measured.

Mia: Exactly. Saying it resolved most incoming chats is one claim. Saying it replaced most of the workforce without losing quality is a different claim, and the second one needs its own evidence. You can't borrow the proof from the first.

Adam: Is there a reason companies keep doing that, treating AI as if it has more judgment than it's actually shown?

Mia: There is, and it's almost psychological. A study published this spring looked at more than twelve hundred managers and found that simply calling the AI system a colleague changed how closely people supervised it. Managers who thought of it that way caught eighteen percent fewer of its errors, individual accountability for mistakes dropped by nine percentage points, and blame pinned on the AI itself went up by eight points.

Adam: So the framing alone made people watch it less closely.

Mia: Right, and that matters, because the AI doesn't actually have the judgment to know when to escalate a problem on its own. Treating it as if it does is exactly how a support queue full of confidently wrong answers goes unnoticed, until it's a headline. If you run a support desk, the practical version of that is: don't measure the assistant by how many chats it closes, measure how many of its closed chats get reopened by the same customer within a few days. That number tells you whether you're buying speed, or buying complaints you just haven't heard yet.

Adam: Which leaves the third piece. Is that where the real money actually is?

Mia: According to the group that's looked hardest at this, yes, and it's the part nobody puts on a keynote slide. A research team at MIT reviewed more than three hundred publicly disclosed AI deployments, ran fifty-two structured interviews, and separately surveyed a hundred and fifty-three senior leaders. The headline finding was that ninety-five percent of enterprise generative AI pilots show no measurable effect on profit or loss, despite tens of billions of dollars in enterprise investment.

Adam: That's a brutal number for anyone who's bought into this.

Mia: It is. But the more useful finding is where the other pilots' returns actually landed. More than half of enterprise AI budgets go to sales and marketing, because a lead-scoring tool or AI-written ad copy gives an executive a good story to tell a board. The biggest measured returns came from somewhere far less glamorous: ending outsourced contracts, cutting agency spend, streamlining finance and operations work that used to run through a vendor or a temp staffing agency.

Adam: So the flashy use case gets the story, and the boring use case gets the actual return.

Mia: That's roughly the researchers' own framing: sales and marketing AI creates a story, back-office automation just deletes a line item. One more detail worth having: pilots built through a specialized vendor partnership succeeded about sixty-seven percent of the time, roughly double the rate of the same thing built in-house.

Adam: That's a strong argument for not building it yourself.

Mia: It's a strong argument for knowing what you're actually good at building. JPMorgan's own back-office numbers point the same way. Their systems now screen market activity and overnight research for private bankers, freeing them up for client conversations, and the bank says that could let a single banker's client coverage grow by up to fifty percent.

Adam: That's specifically a finance function, not a chatbot answering customers.

Mia: Right, and that's exactly the kind of gain that shows up on an internal dashboard, never in a press release. Anthropic's own usage data adds a texture here: computer and mathematical jobs make up thirty percent of the people using its AI assistant for real work, against just four percent of the actual workforce. Heavy technical use is still concentrated in a narrow slice of jobs.

Adam: What about the more routine back-office jobs, the ones we're actually talking about?

Mia: That's the more telling part. Usage among the lowest-paid workers, routine clerical and telemarketing-type roles, drops off sharply on nights and weekends, which suggests it's work assigned by an employer, not something people explore on their own. Usage among higher-paid roles keeps going on evenings and weekends even outside computer jobs, meaning those workers are folding the tools into their own habits.

Adam: So the people paid more are choosing to use it, and the people paid less are being told to.

Mia: That's the shape of it. And Adam, if you sit in finance or operations, here's something concrete to try this week: pull up your outsourced contracts and vendor spend before you greenlight anything customer-facing, and see how much of it could be ended or renegotiated the way these back-office deployments were. That's where the measured returns have actually shown up, not in a flashy pilot.

Adam: Okay, so three functions, three different pictures. Mia, what actually ties them together? Why do some tasks turn out so much harder than the pitch?

Mia: A pattern falls right out once you line them up. The wins, a well-specified coding ticket, a routine support query, a reconciliation with clear rules, share one thing: a human can check the result fast, and the cost of being wrong once is small. The disappointments share the opposite: a vague goal, lots of tools, many steps, and nobody watching closely enough to catch a mistake before it piles up.

Adam: That's basically the colleague-framing point again, isn't it? An agent given a broad goal and told to figure out the rest.

Mia: Exactly that pattern, presented as though it'll reliably work. There's an arithmetic reason it doesn't. Say an agent's individual steps each succeed ninety-five percent of the time, which sounds like a great, demo-worthy number.

Adam: That does sound great.

Mia: A ten-step task built from those steps only succeeds end to end about sixty percent of the time, because the errors compound. Stretch it out to thirty steps, and the same compounding drags it down to about twenty percent.

Adam: So being accurate that often and actually working reliably on someone's real workflow are basically never describing the same thing.

Mia: Never the same thing. It's Devin holding up fine on a handful of clean tickets and struggling on ambiguous ones. It's GitClear's fast-but-messier code. It's Klarna measuring one thing and claiming another. Same underlying math, different costumes.

Adam: So what's the actual test, when someone shows you a demo next week?

Mia: Three questions. What was actually measured, a benchmark or a real deployment. How many steps, and how much ambiguity, sit between the agent's action and someone actually checking it. And what that reported number would look like if you multiplied its per-step reliability out across the whole task, instead of taking it at face value.

Adam: Mia, try that on the next slide someone puts in front of you, and see if the number survives.

Mia: Because that's the gap we started with: developers who were measurably slower and still walked away feeling faster. Everything since has been the same gap, wearing a different costume, function after function.

Adam: That was part two of five. Part three is called When Agents Go Wrong, and it's about why keeping a human in the loop is so much harder than it sounds.

Heard something wrong? Tell us