Who checks the agent's work, and how to design that review step

The bank BNY has about 140 AI agents it calls “digital employees.” Each one has login credentials, an employee number in the corporate directory and a human supervisor who assigns its work and reviews what it produces. Microsoft published a detailed look at how the bank runs them last week (Microsoft WorkLab).

“All digital employees report to a human manager,” CEO Robin Vince told Business Insider earlier this year. About 100 managers across the firm oversee them.

You don’t need a bank’s compliance department to copy this. Every company putting an agent into real work has to answer the same question: who checks the output, and how? Skip it and you find out the hard way.

How BNY does it

The details in the WorkLab piece are worth reading closely, because they show review built into the work itself.

• The payments agent fixes problems in payments and then submits the corrected payments to a person for review.
• The client-inquiry agent drafts replies and puts them in a queue for a human manager.
• Managers watch a live dashboard that tracks how much each agent processed, how long it took, which transactions it missed and how often its work was approved.
• When reviewers sent work back, the team treated it like coaching a new hire: recalibrate, retrain, try again.

The results BNY reports are strong. Payment validations that took five or six minutes now finish in under 30 seconds, and open payment investigations have dropped by nearly 80%, per the same article. Those are the bank’s own figures. They came with a person checking every payment the agent corrected.

FlowHow a BNY agent's work moves

1. 01Agent does the task
2. 02Work goes to a review queue
3. 03Manager approves or sends back
4. 04Rejections drive retraining

BNY also runs what it calls a Digital Employee Agency, a formal process for deciding when a problem calls for an agent instead of a person or a contractor. It covers onboarding, scorecards and offboarding, per WorkLab. That’s the same paperwork you’d do for a new hire, and it means every agent starts life with a job description and someone who answers for it.

Design the review step before you turn the agent on

Here’s how I’d set up review for any agent going into real work.

Name one owner. Every agent gets one person who is accountable for its output, the same way every employee has a manager. Pick the person who does the task best today. A shared inbox nobody owns isn’t review. The owner also decides when the agent’s instructions change, so fixes don’t come from five different people with five different ideas.

Review everything at first. For the first few weeks, a person approves every piece of output before it leaves the building. That sounds slow. It’s how you learn what the agent gets wrong, and it’s still faster than doing the work from scratch, because approving a good draft takes a fraction of the time of writing one.

Say your agent drafts 200 support replies a week and a good draft takes 30 seconds to read and approve. That’s 100 minutes of one person’s week. Writing those 200 replies from scratch would take most of that person’s week. Review at that cost is a bargain.

Put the review where the work leaves. Check the output at the point it reaches a customer, a system of record or money. An agent can draft, sort and research all day without a checkpoint. It shouldn’t send, post or pay without one.

Make review fast. Ask the agent to show its sources with every draft: the order record it read, the policy section it applied, the numbers it used. A reviewer who can see the inputs can approve in seconds. A reviewer who has to go find them is redoing the work.

Log every rejection with a reason. One line is enough: “wrong return window,” “missed the discount code,” “tone too stiff.” That log is your list of fixes for the agent’s instructions. Read it every week for the first month. The same three or four causes will show up again and again, and fixing them in the instructions is where most of the improvement comes from.

Watch the reviewers, too

One detail in the BNY story surprised me. Reviewers sent back some of the payments agent’s output even though the answers weren’t wrong. They were used to the old process and rejected work that didn’t look the way they expected (Microsoft WorkLab).

Expect the same at your company. A reviewer who has done the task one way for years will reject a correct answer because it’s formatted differently or reached a different way. So when you read the rejection log, ask two questions. Was the agent wrong? Or was the reviewer holding it to a habit nobody wrote down? Fix whichever one it was. Sometimes the instruction needs to change, and sometimes the reviewer does.

Loosen review only on evidence

After a few weeks you’ll have data: how many pieces of work, how many approved untouched, how many edited, how many rejected and why. Use it to decide where review can relax.

ChecklistBefore you reduce review on an agent

• Four weeks of approval data on real work
• Rejection reasons sorted by cause
• The top causes fixed in the instructions
• Approval rate holding steady after the fixes
• High-risk cases still always reviewed

A sensible next step is sampling. Low-risk work, like routine order-status replies, moves to a person checking one in ten. High-risk work, like refunds over a set amount or anything touching a contract, stays at full review. Keep the dashboard running either way. If the approval rate on sampled work drops, go back to full review until you know why.

Don’t let review quietly disappear, either. The more likely failure is a busy reviewer who starts clicking approve without reading because the last hundred drafts were fine. Spot-check the reviewers now and then. Pull ten approved items at random each month and have someone else look at them.

The Forrester analyst who wrote about BNY last week made a related point: the bank built a governed platform and trained its people before it put autonomous agents into core operations (Forrester). Review is part of that groundwork. It’s what lets you trust an agent with more over time.

What to do this week

• For each agent you run or plan to run, write down the name of the one person who owns its output.
• Mark the exact point where its work reaches a customer, a system of record or money, and put the review there.
• Change the agent’s instructions so every draft shows the sources it used.
• Start a rejection log with three columns: date, what was wrong, and whether the cause was the agent or the reviewer’s habit.
• Put a date on the calendar four weeks out to review the log and decide, task by task, where full review stays.

Sources

• The making of a Frontier Firm: How AI is redesigning work at BNY (Microsoft WorkLab, May 12, 2026)
• BNY’s CEO on the firm’s newest crop of managers overseeing its 140 ‘digital employees’ (Business Insider via AOL, March 23, 2026)
• BNY built its digital workforce backward, and it’s working (Forrester, May 12, 2026)