How to run an outsourcing pilot that tells you something

A pilot exists to reduce uncertainty. One the provider is almost certain to pass only delays the moment you find out, until you are contractually committed and it is January.

Send the difficult work

The instinct is to send clean, simple jobs so the provider has a fair chance. You learn little from that, because you already know a clean job will come back fine. What you need to see is how a provider behaves when the records are poor, the treatment is ambiguous or the prior year contains a judgement that is not documented anywhere.

Send a representative mix, weighted towards the awkward end, and include at least one client whose file you know is a mess. Whether the provider flags it early and scopes it, or absorbs it silently and produces something plausible, tells you more than ten clean jobs.

Measure review burden, not error rate

Almost every failed outsourcing relationship failed on review burden. The accounts were technically right, but the file had no lead schedules, no cross-references and no explanation of judgements, so the partner rebuilt the job to understand it and the saving disappeared.

So measure review burden:

• Time to review. How long did it take your reviewer to reach a conclusion on this file, compared with an equivalent file prepared in-house?
• Questions generated. How many times did your reviewer have to ask what something was, where a figure came from, or why a treatment was chosen?
• Rework rate. How many jobs came back needing substantive correction rather than a partner's stylistic preference?
• Query quality. Were queries batched, evidenced and specific, or dribbled out one at a time with no supporting documentation?

Record these numbers. A vague sense that "it was fine" will not survive contact with a partner who was sceptical from the start.

Do not tell them which jobs are the test

Where practical, a pilot should look like normal work. A provider that assigns its best reviewer to a batch it knows is being assessed is showing you a capability that will not be there at volume. Any organisation behaves this way when it knows it is being assessed.

Test the escalation path deliberately

At some point in a real engagement something will go wrong: a deadline at risk, an ambiguous instruction, a client who has not responded. The pilot is the time to find out what happens. Introduce an ambiguity deliberately and see whether it is escalated, resolved with an assumption, or ignored.

You want to see it escalated with a recommendation. A silent assumption is what produces a significant problem eighteen months later, on a job nobody remembers.

Include a real deadline

A pilot with a comfortable timetable tests technical capability without testing delivery discipline. Give at least one job a real deadline and watch how the provider communicates as it approaches: whether it sends you a status update unprompted, or only when asked.

Agree the evaluation criteria before you start

Write down, before the first job is sent, what would count as a pass. Otherwise the evaluation becomes a negotiation with your own sunk cost, particularly if the pilot involved effort from your team.

Reasonable criteria: files reviewable without rebuilding, no more than a defined level of substantive rework, queries batched and evidenced, deadline met, escalation handled as expected. Share the criteria with the provider. A serious one will want to know what it is being measured against, and one that resists is telling you something.

Pay for it

A free pilot looks like a good deal. In practice it leaves you feeling obliged, and on the provider's side it justifies exactly the extra attention that makes the pilot unrepresentative. Pay the standard rate for the work. That keeps the relationship and the assessment on a commercial footing, and you can walk away without a conversation about goodwill.

Then run it again in the peak

A pilot in a quiet month tells you what a provider can do when it has capacity to spare. What you need to know is what it can do in January, when every client is a priority. If the first pilot passes, treat the first peak period as a second one, with the same measurement discipline, a limited volume and a plan for what you will do if it does not hold.