How to run an outsourcing pilot that actually tells you something

A pilot exists to reduce uncertainty. If it is structured so that the provider is almost certain to pass, it has reduced nothing — it has just delayed the moment you find out, to a point where you are contractually committed and it is January.

Send difficult work, not easy work

The instinct is to send clean, simple jobs so the provider has a fair chance. This is exactly backwards. You already know a clean job will come back fine. What you do not know is how a provider behaves when the records are poor, the treatment is ambiguous or the prior year contains a judgement that is not documented anywhere.

Send a representative mix, weighted towards the awkward end. Include at least one client whose file you know is a mess. How the provider handles it — flagging it early and scoping it, or silently absorbing it and producing something plausible — tells you more than ten clean jobs.

Measure review burden, not error rate

Almost every failed outsourcing relationship failed on review burden. The accounts were technically right. The file had no lead schedules, no cross-references and no explanation of judgements, so the partner rebuilt the job to understand it — and the saving evaporated.

So measure the thing that actually matters:

• Time to review. How long did it take your reviewer to reach a conclusion on this file, compared with an equivalent file prepared in-house?
• Questions generated. How many times did your reviewer have to ask what something was, where a figure came from, or why a treatment was chosen?
• Rework rate. How many jobs came back needing substantive correction rather than a partner's stylistic preference?
• Query quality. Were queries batched, evidenced and specific, or dribbled out one at a time with no supporting documentation?

Record these numbers. A vague sense that "it was fine" will not survive contact with a partner who was sceptical from the start.

Do not tell them which jobs are the test

Where practical, a pilot should look like normal work. A provider that assigns its best reviewer to a batch it knows is being assessed is showing you a capability that will not be present at volume. This is not cynicism about providers; it is how any organisation responds to a known assessment.

Test the escalation path deliberately

At some point in a real engagement something will go wrong: a deadline at risk, an ambiguous instruction, a client who has not responded. The pilot is the time to find out what happens. Introduce an ambiguity deliberately and see whether it is escalated, resolved with an assumption, or quietly ignored.

The right answer is escalation with a recommendation. An assumption made silently is the behaviour that produces a significant problem eighteen months later, on a job nobody remembers.

Include a real deadline

A pilot with a comfortable timetable tests technical capability but not delivery discipline. At least one job should carry a genuine deadline, so you observe how the provider communicates as it approaches — proactively, with a status, or only when asked.

Agree the evaluation criteria before you start

Write down, before the first job is sent, what would constitute a pass. Otherwise the evaluation becomes a negotiation with your own sunk cost, particularly if the pilot involved effort from your team.

Reasonable criteria: files reviewable without rebuilding, no more than a defined level of substantive rework, queries batched and evidenced, deadline met, escalation behaved as expected. Share the criteria with the provider — a serious one will welcome knowing what they are being measured against, and one who resists that is telling you something.

Pay for it

A free pilot sounds like a good deal and is not. It creates an obligation dynamic on your side, and on the provider's side it justifies exactly the extra attention that makes the pilot unrepresentative. Pay the standard rate for the work. It keeps the relationship commercial, keeps the assessment clean, and buys you the right to walk away without a conversation about goodwill.

Then run it again in the peak

A pilot in a quiet month tells you what a provider can do with capacity available. A firm's real question is what they can do in January, when every client is a priority. If the first pilot passes, treat the first peak period as a second one — with the same measurement discipline, a limited volume, and a plan for what you do if it does not hold.