A company outsources regression testing to a specialist vendor. The contract is sound: an SLA on cycle time, rates negotiated eighteen per cent below the opening offer, a report after every release. Eighteen months in, the engagement looks exemplary — the vendor executes 640 test cases per release, reports a 97 per cent pass rate, and has never missed a window. Over the same period, production suffered three serious incidents. All three were in areas formally covered by test cases that passed.
At the review meeting someone asks whose fault it was. The vendor demonstrates that it executed exactly what it was given, and executed it flawlessly. The client points out that it is paying for testing and the product keeps breaking. Both are right, and that is the problem: the contract bought the execution of a script while both parties believed it was buying quality. Nobody lied. Nobody had ever written down who was entitled to say the product was not fit to ship.
This article argues something that is uncomfortable for both sides of that contract: QA outsourcing only works when the right to define quality criteria is handed over along with the execution. Keeping that definition in-house — the default configuration, present in most master service agreements we have seen — turns a specialist supplier into a click-executor, and pays click-executor work at a rate that is only justified when you are buying judgement.
What are you actually buying when you outsource testing?
Every QA function contains two entirely different things that outsourcing contracts almost never separate in writing.
The first is execution: walking a test case, running a suite, preparing data, reporting a result. This is work measurable in hours and cases, easy to price, easy to put in an SLA and easy to compare across suppliers — which is precisely why the market compares QA vendors mainly on QA outsourcing rates, the one dimension everyone describes the same way.
The second is definition: deciding what needs testing at all, what counts as a defect, how severe that defect is, and whether the build is fit to ship with that set of open defects. This work cannot be measured in hours, because its value lies in accuracy rather than effort. An hour spent correctly identifying that the risk lives in a data migration rather than in the UI is worth thirty hours of clicking through the UI.
Product quality is produced by the second layer. The first only carries it out. You can execute a set of cases flawlessly and still miss the area where the real risk sits — which is exactly what the opening story describes. A suite that does not cover the scenario in which the system fell over is correct in every individual step and useless as a whole.
Hence a conclusion that looks like a truism until you hold it against real agreements: if the vendor receives only the execution layer, you are buying diligence, not quality. Diligence is available more cheaply.
Why is the hybrid the most common configuration and the worst one?
In discussions of QA operating models — including our own analysis of in-house QA versus outsourced testing — the hybrid model shows up as the sensible compromise: the vendor supplies execution capacity, the client keeps control over what and how things are tested. It reads like the best of both worlds and it sells beautifully, because it asks nobody to extend trust.
It is worth saying plainly: this particular hybrid is the worst available division of labour, and its popularity comes from being politically comfortable rather than from working. It separates knowledge from authority in precisely the wrong direction.
After a year on the account, the vendor knows more about the product’s failure patterns than anyone on the client side. It has seen every defect, every regression, every corner where “this always breaks at release”. That knowledge is its only genuine asset and the only justification for its rate. And it has no right to use it, because the test scope is defined by someone who never watched that history up close.
The client, meanwhile, retains decision rights without the informational basis to exercise them well. Its QA manager writes cases from a specification and from personal intuition formed two years ago, when they were still testing themselves.
The effect is structural, not personal. It cannot be fixed by better communication or a weekly status call, because it is not an information-flow problem — it is a problem of where decision rights sit. The person with the knowledge has no authority, the person with the authority has no knowledge, and the agreement describes only how many cases will be executed. That is how a click-executor is created: not because the vendor is weak, but because the contract contains no moment at which it could use what it knows.
What is the definition of quality made of, and which part never gets handed over?
To discuss handing over the definition you first have to break it into parts, because “the definition of quality” as a whole is too vague to sign. In practice it consists of four components, each of which can independently sit with the client or with the vendor.
The risk model. The answer to what can go wrong and what it would cost. It determines where it is worth looking for defects at all. The risk model decides whether sixty per cent of the effort goes into the payment path or spreads evenly across every screen.
Acceptance criteria. The definition of what “works” means for a given feature. This is the layer where silent divergence happens most often: the client considers PDF export working when a file is produced; the vendor considers it working when it is produced correctly for a 400-line document with non-ASCII characters in the filename.
The severity taxonomy. The rule that turns a finding into a priority. It is the most underrated component of the definition and the one with the strongest influence on realised quality, because severity decides whether anyone fixes the defect before the release.
The right to make a release recommendation. The authority to write “in our assessment this build is not fit to release” and to have that sentence carry weight in the organisation. Without it, the previous three components describe a state rather than driving a mechanism.
A pattern recurs often enough across engagements to deserve a name: clients hand over the first component most readily and the fourth almost never. Commissioning a risk analysis from the vendor is safe — it produces a document. Handing over the right to stop a release is not — it transfers power. And that fourth component is exactly what distinguishes a QA service from subcontracting.
The intermediate version, encountered most often, reads: “the vendor may raise an objection; the decision rests with the client’s release board.” That formula looks reasonable and changes nothing in practice unless the contract also requires that the objection and the reasons for overriding it are recorded. Without that clause, the objection is an opinion, and an opinion creates accountability on neither side.
How do you tell the vendor got execution instead of accountability?
The diagnosis needs no audit and can be done in an afternoon, on data that already exists in your tooling.
Check the authorship of test cases. Open the test management system and filter the author field for cases added in the last six months. If client accounts dominate, the vendor is running someone else’s script. This is the single hardest signal available, because case authorship is the operational trace of who decides what gets tested.
Count severity disputes and see who wins them. Review defects whose severity was changed after being raised. If the changes trend downwards and are consistently made by the client, the client owns the taxonomy regardless of what the contract says. We have seen accounts where downgrades touched more than a third of high-priority findings — at that scale it is not the correction of individual mistakes, it is ownership of the criterion.
Look for a negative recommendation. Read the last twelve release reports and find one sentence in which the vendor did not recommend shipping. If there is none, there are only two possibilities: either the product was flawless twelve times running, or the vendor has no mechanism for writing that sentence. The first does not happen.
See what the monthly report measures. If the first table shows cases executed, cycle time and pass rate, the vendor is reporting activity. If it shows defects that escaped to production, their severity and coverage of the risks in the model, it is reporting outcomes. The gap between activity metrics and outcome metrics in QA is exactly the gap between execution and accountability.
Three of the four signals answering yes means you are paying a judgement rate and receiving throughput.
Why does an SLA on test case volume measure exactly what you do not want to buy?
Writing a number of test cases into the agreement looks like the only objective way to describe the scope of the service. It is also the shortest path to a vendor optimising for something other than what you ordered.
The mechanism is Goodhart’s law in its most literal form: once a measure becomes a target, it stops being a good measure. If billing follows executed cases, a rational supplier increases their number and shortens the unit time. That produces decisions each of which is individually defensible: splitting one complex scenario into four simple ones, keeping alive cases that have not caught a defect in two years, and avoiding exploratory testing because it produces nothing billable. The sum of those decisions is a suite that grows, costs more each quarter and catches less.
Exploratory testing is the expensive casualty here. It is the only technique whose value comes from the tester’s judgement during the work rather than from a step written down beforehand — and the only one that cannot be billed in a “test case as the unit” model. In case-billed contracts it disappears first, and with it goes most of the chance of catching a defect nobody anticipated when the scripts were written.
The alternative is not to abandon measurement but to move the measure one level up, to the outcome. A contract can bill against defects that escaped to production, split by severity, with a tolerance threshold for the worst class. It can bill against coverage of the risks in an agreed risk model rather than coverage of lines of code. It can oblige the vendor to issue a release recommendation and attach a consequence to issuing one inconsistent with its own knowledge. All three are harder to count than a case volume, and all three measure what you are actually buying.
This is where most organisations stop, for a simple reason: an outcome measure requires that the definition has already been handed over. You cannot hold a vendor to escaped defects if someone else wrote the test scope — the vendor will rightly answer that it tested what it was given. Outcome-based contracting and retaining the definition in-house are mutually exclusive, which is exactly why the market clings to activity measures.
What does keeping the definition in-house actually cost?
The case for outsourcing QA is usually a cost case, and the arithmetic is done on the hourly rate. That arithmetic omits a line item which, in the retained-definition model, frequently exceeds the rate arbitrage that motivated the whole exercise.
A client-side headcount nobody put in the model. Someone has to write and maintain the test cases, arbitrate severities, answer the vendor’s questions and prepare scope for every release. On a mid-sized product this is typically between 0.6 and 1.0 FTE of an experienced QA manager — an expensive person. If the reason for outsourcing was that in-house QA cost too much, this line consumes a large share of the saving before anything else is counted.
Vendor waiting time. In a model where every ambiguity requires a client ruling, the vendor’s team waits. That time is billed, because the vendor sells team availability. It rarely exceeds the low teens as a percentage, but it is charged at the highest rate in the contract.
The cost of the second pass. Divergence between the client’s acceptance criteria and the vendor’s understanding only surfaces in the result, and ends in repeating part of the cycle. Each repetition costs twice: once in hours, once in release delay.
The cost of an incident that falls between definitions. The most expensive line and the only one that never appears in any comparison, because it is not formally a QA cost. A production incident in an area no scenario covered is an event for which neither the vendor (it delivered the scope) nor the client’s QA manager (they could not foresee everything) is accountable. Distributed accountability is no accountability, and the bill arrives anyway — as downtime, lost transactions or service credits owed to your own customers.
Sum those four items, compare them with the rate arbitrage that motivated the arrangement, and the retained-definition model very often costs more than keeping QA in-house — and almost always more than outsourcing with the definition handed over. That calculation is worth running on your own numbers before renewal, with the same discipline you would apply to a total cost of ownership comparison for QA.
What exactly do you sign when you hand the definition over?
Handing over the definition of quality is not a declaration of trust; it is a set of specific clauses. The five below are enough to change the character of the relationship, and none of them requires renegotiating rates.
A risk model workshop signed by both parties. A one-off exercise of a few days producing a list of risk areas with an attached business impact. The client brings knowledge of impact, the vendor knowledge of likelihood. From then on that document is the basis of test scope and the only thing the client approves before each release — instead of approving a list of cases.
A vendor-authored test strategy, approved once. The vendor writes how it intends to cover the risks in the model. The client approves the strategy, not individual cases. This is the operational core of the handover: from that moment the author of the scenarios is the party that knows the failure patterns.
A vendor-owned severity taxonomy with an appeal path. The vendor classifies; the client has a narrow, documented right of appeal with reasons and a register entry. The appeal register incidentally becomes the best maturity metric available: a handful of appeals after a year means the taxonomy has converged; dozens mean the parties still are not talking about the same thing.
A named release recommendation. Before every release the vendor issues an unambiguous recommendation — ship or do not ship — signed by a specific person. The client keeps the full right to ship against it, but must do so deliberately and leave a trace. That single clause does more for quality than any change to test scope, because it converts blurred co-responsibility into two clean, separated decisions.
One outcome measure in the commercials. At minimum one billing element tied to defects that escaped to production. It need not be a penalty — a threshold above which a joint risk model review is triggered is enough. The point is that outcomes enter the commercial conversation at all. A structured IT outsourcing vendor evaluation scorecard is a good place to attach these clauses during selection rather than after signature.
What must never be handed over — where does the boundary actually run?
The thesis of this article is strong but it is not absolute, and pretending otherwise would be bad advice. There is a line past which delegation stops being a transfer of competence and becomes an escape from a decision.
Business risk appetite stays with the client and is not negotiable. Whether you ship with an open medium-severity defect because a marketing campaign launches on Monday is not a question about quality. It is a question about how much risk the firm wants to absorb in exchange for a specific benefit, and no supplier has either the standing or the mandate to answer it. The vendor supplies an unambiguous statement of what you are risking. The ruling belongs to the client.
Regulatory accountability is personal and non-transferable. In supervised domains — medical devices, parts of financial services, product safety — accountability for releasing to use rests on a named function inside the manufacturer and cannot be assigned by contract. The vendor can prepare the entire evidence pack; the signature stays with the client.
Product trade-offs belong to the product. Whether a feature ships in a reduced form or slips a month is a strategic decision. QA supplies data to it; it does not make it.
The boundary therefore runs exactly here: the vendor receives the definition of technical quality, the client retains the definition of business risk. Without that split, this article’s thesis collapses into a proposal to put the firm into external administration, which would be bad advice regardless of the vendor’s quality. With it, the thesis becomes something far more modest and achievable: moving engineering judgement to where the knowledge is, while keeping the business decision where the ownership is.
When is retaining the definition the right call?
There are three situations in which keeping the definition of quality client-side is justified. In all three the practical conclusion is the same, and it is uncomfortable for QA service providers.
A product with a short remaining life. If the system has eighteen months ahead of it — because it is being decommissioned, migrated, or is a seasonal campaign — the cost of transferring risk knowledge never pays back. The vendor will reach a useful risk model roughly when the product stops existing.
Extreme concentration of risk knowledge. Some organisations hold all knowledge of what can go wrong in one person’s head, never written down. Handing over the definition then requires extracting it first, which is a separate multi-month project. Until it completes, the definition necessarily stays with the client.
Regulatory accountability covering the whole, not a fragment. In a small number of domains the regulator requires that acceptance criteria themselves be established by the responsible entity. There, delegation is legally excluded rather than merely risky.
The conclusion in each case is identical: if you cannot hand over the definition, do not buy a QA service — buy capacity. Contract engineers who work inside the client’s team, under its process and its definition, billed for availability rather than for a quality outcome. That is an honest model with predictable economics. The dishonest one is the third: buying a QA service at a judgement rate while denying the supplier access to judgement. It costs like a service, performs like capacity, and has no owner when something goes wrong.
How do you run the handover inside an existing contract?
The handover requires neither terminating the agreement nor re-tendering. The sequence below fits in roughly ninety days and most of its steps are a variation notice plus a process change.
Days 1–15: baseline measurement. Four numbers: the share of test cases authored by the client, the number of severity changes made by the client, the number of negative release recommendations in the last year, and the number of defects that escaped to production, split by severity. Without this baseline you will not be able to show later that the change achieved anything.
Days 16–30: risk workshop. Two or three days with the product owner, an architect and the vendor’s team. The output is a list of risk areas with business impact expressed in money or in downtime. Both parties sign it.
Days 31–50: vendor-authored test strategy. The vendor writes how it covers the risks. The client approves the document as a whole and stops approving individual cases from that point. This is the hardest step organisationally, because it removes from the client’s QA manager the activity that defined their role — and it requires openly redefining that role as owner of the risk model.
Days 51–70: severity taxonomy and appeal path. A joint rewrite of severity definitions into at most four levels with unambiguous criteria. Classification rights move to the vendor; the client receives a documented appeal path with a deadline and a requirement to give reasons.
Days 71–90: release recommendation and outcome measure. A variation introduces the obligation of a named recommendation before every release plus one outcome measure in reporting. After the first three releases, repeat the baseline measurement.
There is one signal that the handover worked, and it usually appears between the second and fourth month: the vendor declines to recommend a release for the first time, and the organisation holds that release. Until that happens, everything is a declaration. In multi-vendor estates it is worth aligning this mechanism from the start with how quality and risk are governed across suppliers, so recommendations do not cancel each other out.
What does the maturity map for handing over the definition look like?
The map below lets you locate your own contract in a few minutes. The levels are not stages you must pass through in order — you can move from 1 straight to 3 if a risk model already exists.
| Level | Who writes cases | Who sets severity | Who recommends release | What the contract measures | Typical outcome |
|---|---|---|---|---|---|
| 0 — subcontracting | client | client | nobody formally | man-hours | vendor replaced yearly, risk knowledge accumulates nowhere |
| 1 — measured execution | client | client | nobody formally | case volume, cycle time | suite grows, detection falls, exploratory testing disappears |
| 2 — consultative hybrid | jointly | client, after vendor raises | vendor informally, no record | cases plus time SLA | severity disputes, diffuse accountability, ownerless incidents |
| 3 — technical definition handed over | vendor, per risk model | vendor, client may appeal | vendor by name, client decides | risk coverage plus escaped defects | vendor starts declining recommendations, quality becomes visible |
| 4 — outcome partnership | vendor | vendor | vendor, with contractual consequence | outcome measures only | joint risk model review after an incident instead of a blame hunt |
Most QA outsourcing contracts we have reviewed sit at levels 1 and 2. Level 4 is realistic only in a mature relationship and requires both sides to accept shared consequences — not every organisation wants that, and not every one needs it. Level 3 is enough for the service to start doing what it is paid for.
What are the key takeaways?
Outsourcing QA is buying judgement, not throughput. Judgement is produced by defining: what to test, what counts as a defect, how severe it is, and whether the product is fit to ship. A contract that leaves that entire layer with the client buys execution at the price of judgement and disappoints both parties systematically, without either breaching the agreement.
Three things are worth doing this week if the description fits your contract. First, check test case authorship over the last six months — one query in the tool, and the hardest signal that exists. Second, search the last year of reports for a single negative release recommendation; its absence does not mean there were no grounds. Third, add the client-side headcount that maintains the definition to the outsourcing bill and check whether the rate arbitrage survives the correction. A UAT planning checklist helps here, because user acceptance is where divergent definitions surface first.
If those three steps show you are paying for judgement you do not allow to speak, you have two honest options: hand the technical definition of quality to the vendor while keeping the business risk decision, or stop buying a service and start buying capacity. The third option — the current configuration — is the most expensive of them.
At ARDURA Consulting we run quality assurance engagements in both models, and we open every commercial conversation with a question this industry asks far too rarely: once the contract is signed, who is entitled to say “this is not fit to release”? Let’s talk about your contract before you renew it unchanged for another year.