What happened when real workers were given generative AI

Most of what gets said about AI at work is a forecast. Somebody imagines what it will do to lawyers or programmers or call centres, and the guess travels round the internet until it starts to sound like a fact.

I prefer the studies where researchers watched what happened when real people got the tools. Two of them have become the ones everybody cites, and they're worth reading properly, because the results are stranger and more useful than the headlines. One followed thousands of customer support staff. The other gave GPT-4 to hundreds of management consultants and then set a trap.

The help desk where the newest staff caught up

Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied a Fortune 500 company that sells business-process software. Its customer support team answered questions by chat, and around 80% of the staff were based in the Philippines. Mostly in autumn 2020 and winter 2021, the company rolled out an AI assistant built on GPT-3, in stages. That staggered start is what makes the study useful: for months, some people had the tool and some didn't, while doing the same job.

The assistant read each chat as it happened and suggested what to say. Only the worker saw the suggestions, and they were free to use, edit or ignore them. The researchers looked at about 3 million chats handled by 5,172 people.

On average, people with the assistant resolved 15% more customer issues per hour. That's a solid gain, but the average hides the interesting part. The least skilled fifth of the staff, measured by how they performed before the tool arrived, gained 36%. Less skilled and less experienced workers as a group gained about 30%. The most skilled staff showed no significant change in productivity, and the quality of their conversations dipped slightly.

• Least skilled fifth of staff36%
• Less skilled and newer staff30%
• All staff, average15%

Source: Brynjolfsson, Li and Raymond, Quarterly Journal of Economics, 2025. The most skilled staff showed no significant change.

My favourite finding is about time. Workers with two months on the job and the assistant performed as well as workers with more than six months and no assistant. The tool seems to have packaged what the best people already knew, which phrasing calms an angry customer and which fix usually works, and handed it to the people who hadn't learned it yet. The best people had nothing to gain from a copy of their own habits.

There were side effects too. Customers were more polite to people using the assistant and less likely to ask for a supervisor. Their English became more fluent, especially among staff outside the US. And the gains were biggest on moderately rare problems: ones a newer worker might not have met, but which the AI had seen enough of to help with.

The consultants who got a trick question

The second study is the one with the twist. Fabrizio Dell'Acqua and eight co-authors worked with Boston Consulting Group to run a preregistered experiment on 758 of its consultants, about 7% of the firm's individual-contributor consultants. These are people with degrees from places like Harvard, Yale and Oxford, and the tasks counted toward their bonuses.

Each consultant first did a task without AI, to set a baseline. Then they were randomly split three ways: no AI, GPT-4 (as it stood at the end of April 2023), or GPT-4 plus a prompt engineering overview.

About half of them got a set of 18 tasks that the researchers had pretested to sit inside what GPT-4 could do: developing a new shoe for a fictional footwear company, with questions that tested creativity, analysis, writing and persuasion. Here the AI was a clear win. Consultants using it completed 12.2% more tasks, finished 25.1% more quickly, and produced work that graders scored about 32% higher on average. The ones who had been weaker on the baseline task gained the most, though the strongest gained a good deal as well.

The other half got the trap. They had to tell the chief executive of a fictional company which of its brands had the most potential for growth, using a spreadsheet of figures and a set of interview notes. The spreadsheet was built to look complete. The answer depended on details buried in the interviews, and the task was designed and pretested so that GPT-4, given the instructions and the data, would reach the wrong conclusion.

Without AI, 84.5% of consultants got it right. With GPT-4, 70.6% did. With GPT-4 and the prompting overview, only 60% did. Across the two AI groups that's an average drop of 19 percentage points, among some of the best-qualified knowledge workers you could hope to recruit.

• No AI84.5%
• GPT-470.6%
• GPT-4 plus prompting overview60%

Source: Dell'Acqua et al., Organization Science, 2026 (published version of a 2023 Harvard Business School working paper).

Here's the part that should worry anyone who reviews other people's work. Graders who didn't know the right answer scored each recommendation on how clear and persuasive it was. The consultants with AI wrote more coherent, more convincing recommendations, and that held for the wrong answers as well as the right ones. The AI didn't just lead people astray. It helped them sound sure while it did.

The authors call this a jagged frontier. AI is very good at some tasks and poor at others that look just as hard, sometimes within the same piece of work, and nothing on the surface tells you which side of the line you're on.

What I take from the two studies

I'd draw three lessons, and I hold them more firmly than most forecasts.

First, AI helps most where people are still learning. Both studies found the biggest gains among the less experienced or less skilled, and smaller ones at the top. If you're deciding where to bring it in, start with the work your newer people find hard, not the work your best people already do well.

Second, it can be confidently wrong, and the confidence is the dangerous part. A weak answer that reads badly gets caught. A weak answer that reads beautifully gets sent. The consultants weren't careless; they were given a fluent, plausible recommendation and too many of them went along with it.

Third, the fix isn't a better prompt. The group that got prompting guidance did worse on the trick task than the group that didn't. What saved the right answers was someone reading the interview notes and noticing that the spreadsheet didn't tell the whole story. That needs a person who knows the work and keeps their own judgement switched on.

I should be honest about the limits. The support study covers one firm and one job, and the authors say plainly that it "should not be generalized across all occupations". It used GPT-3, years ago. The consulting study used one firm's staff, tasks designed for the experiment and GPT-4 as it stood in April 2023. Today's models are better, and the frontier has moved. But it's still jagged, and I haven't seen anything that says otherwise.

Where this leaves us at Yarlis

These studies match how we build. Our co-workers do the preparation that new salespeople find hardest: working out which accounts are worth pursuing, why now, and what to say in the first conversation. Then a person from the firm, someone who knows the client and the work, reads it, changes what needs changing, and sends it. The co-worker prepares; the person decides.

Questions

Did AI make experienced workers worse? In the support study, the most skilled staff saw no significant change in issues resolved per hour, small gains in speed and a small decline in conversation quality. In the consulting study, the stronger consultants still gained on tasks inside the frontier. Both AI groups did worse, on average, on the trick task.

Do these results still hold with newer models? Nobody can say for certain from these studies. Both used models that have since been replaced. The useful lesson is about how to work with the tools: find out where they fail and keep a knowledgeable person checking the output.

Sources

• Erik Brynjolfsson, Danielle Li and Lindsey Raymond, Generative AI at Work, The Quarterly Journal of Economics, vol. 140, no. 2, 2025, pp. 889-942. The published version of the support study; all figures here come from it.
• Erik Brynjolfsson, Danielle Li and Lindsey Raymond, Generative AI at Work, NBER Working Paper 31161, April 2023, revised November 2023. The earlier working paper version.
• EurekAlert!, news release on the published support study, 4 February 2025. A plain-English summary of the main results.
• Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani, Navigating the Jagged Technological Frontier, Organization Science, published online 11 March 2026. The published version of the consulting study, open access; all figures here come from it.