Studio of 2 vs agency of 30: what SaaS build quality depends on

SaaS build quality is the share of what a team ships that works for users the first time, and its strongest predictors are how many people touch each part of the code and whether each part has an owner. The headcount on a vendor's website predicts neither. A studio of two and an agency of thirty can both ship a product that breaks every week. Both can also ship one that doesn't.

This piece is for the founder or CTO holding two proposals, one from a small studio and one from a large agency, priced far enough apart that the gap needs a reason. The reason most often given is quality: a big team reads as depth, review and backup, a small one reads as risk. The research on software defects measures something narrower, and you can check it on either vendor before you sign. If your question is which scope fits which size, start with our small studio vs big agency comparison. This article stays on quality.

What does the data say about team size and defects?

QSM, a software estimation firm that has benchmarked completed projects since 1978, has studied team size several times. In 2012 Kate Armel compared 1,060 IT projects completed between 2005 and 2011, split into teams of 4 or fewer full-time people (median 2.1) and teams of 5 or more (median 8.5). On small projects, the larger teams cost 3 times as much and delivered 2 times as many defects. On large projects, they cost 4 times as much and delivered 3 times as many defects. The extra people bought very little schedule.

The pattern repeats across QSM's samples. Doug Putnam's 1997 study of 491 projects found teams of 3 to 7 people optimal for medium-sized systems. A later sample of 390 applications compared teams of fewer than 4 people with teams of 9 or more: the larger teams cut the schedule by about 30%, raised cost by 350% and produced 500% more defects to fix in testing, according to QSM's summary of its four team-size studies.

One caveat changes how you read all of it. QSM measured the team on the project. The size of the company that employed the team was outside the question. A 30-person agency can put 4 people on your product. A 2-person studio can bring in contractors until 9 people are committing to your repository. The number that matters is the team that touches your code, and it rarely appears on page one of a proposal.

Why is headcount the wrong number to compare?

Headcount is easy to read, and it answers a different question: how much work a vendor can run at once. Quality depends on how that work is divided. Melvin Conway described the mechanism in 1968, in How Do Committees Invent?: organisations that design systems are "constrained to produce designs which are copies of the communication structures" of those organisations.

A product built by a design department, a frontend team and a backend team tends to show seams where those teams meet. A product built by two people tends to show seams where those two people stopped talking. Size moves the seams around. Every team leaves some, and the five measures below tell you where yours will be.

Five things SaaS build quality depends on

1. How many people touch each part of the code

In 2008 Nachiappan Nagappan, Brendan Murphy and Victor Basili studied 3,404 binaries in Windows Vista, more than 50 million lines of code, to find which measures predicted the binaries that would fail after release. Eight organisational measures, such as how many engineers edited a binary and how far apart they sat in the reporting chain, identified failure-prone binaries with 86.2% precision and 84.0% recall. Code churn reached 78.6% and 79.9%. Code complexity reached 79.3% and 66.0%. Test coverage reached 83.8% precision but only 54.4% recall. How the people were organised said more about where failures would appear than any property of the code itself.

2. Whether each part has one owner

Christian Bird and colleagues at Microsoft Research followed up in 2011 with Don't Touch My Code, covering Windows Vista and Windows 7. They split the contributors to each binary into major contributors, with at least 5% of its changes, and minor contributors, with less. In Vista, the number of minor contributors correlated with pre-release failures at 0.86 and with post-release failures at 0.70. The share of changes held by the top owner correlated negatively, at minus 0.49 for both. Components that many people touched a little failed more. Components with a clear owner failed less.

A small team gets ownership almost by default, because there are few people to share it with. A large team can have it too, by design: one named owner per module, and every change from outside the module reviewed by that owner.

3. Who leaves before the work is done

Nagappan's measures also counted engineers who had touched a binary and then left the company. The person who inherits code they didn't write doesn't know why it was written that way, and the gaps show up as defects. This risk favours neither size. In a studio of two, one departure takes half the knowledge with it. In a large agency, people move between accounts as new projects start, and a move off your account is a departure from your codebase. Ask both vendors how they handle it, and get the answer written into the contract.

4. How many handoffs sit between an approved design and the shipped screen

Every handoff is a point where someone who didn't make a decision restates it. A designer passes a file to a frontend developer, who passes an API request to a backend developer, who passes a build to QA. Each step can be done well. Each step is also one more chance for the screen in production to drift from the one you approved. Count the handoffs in each proposal's process, and ask what each one is there for.

5. Who writes the code and who reviews it

The people named in a pitch are not always the people who commit. That holds for any vendor that sells more work than its senior people can do by hand, small or large. Review is where seniority reaches the code: a senior reviewer on every change to billing, authentication and data access catches the defects that cost the most to fix later. Ask who reviews, on which parts of the product, and whether a change can merge without that review.

Where does a team of two fall short?

The research that favours small teams also sets their limits. Two people can't run four streams at once, so a launch that needs a web app, two native apps and a data pipeline on the same date needs more hands. Two people rarely cover every specialism in depth either: a penetration test, a formal accessibility audit or a 24-hour on-call rotation usually comes from outside. And ownership concentrated in two people is risk concentrated in two people. A small studio that takes quality seriously names the specialists it brings in and tells you what happens to your product if one of its people stops.

Where does a team of thirty earn its size?

A large agency earns its headcount when the work splits into parallel streams, each run by a small team with its own owner. Thirty people organised as five teams of six behave, on the evidence above, like five small teams with a coordination cost on top. Thirty people kept as one pool and staffed onto whatever is urgent this week behave like the large teams in the QSM data. The proposal won't tell you which of the two you are buying. The questions below will.

How do you check quality on any vendor before you sign?

1. Name the people who will commit to the repository. Ask for names and roles, and whether those names go into the contract.
2. Ask for the ownership map. Who owns billing, authentication, the data model and the design system? One name for each.
3. Ask for the review rule. Which changes need a second reviewer, and who that reviewer is.
4. Ask what happens when someone leaves or moves off the project. Notice period, overlap, written handover. The clauses to look for are in our eight contract clauses to check.
5. Ask which delivery numbers they track. DORA, the long-running research programme on software delivery, measures instability with two ratios: change fail rate, the share of deployments that need immediate intervention, and deployment rework rate, the share of unplanned deployments caused by an incident in production. A vendor that tracks either can show you a trend. A vendor that tracks neither is estimating its own quality by feel.
6. Ask to see the history of a comparable project. A commit log with names hidden still shows how many people touched each module and how often fixes followed a release.

A short paid discovery phase is a cheap place to see these answers in practice before the main contract. The questions for the first call itself are in our guide to hiring a product engineering studio.

Where we stand

We are a small studio, so weigh this section with that in mind. The evidence favours a team on your code that is small, stable and owned, whoever employs it, and a small studio can fail that test as easily as a large agency. When a brief needs more parallel streams than a small team can run, we say so on the first call.

Sources

• Nagappan, Murphy, Basili: The Influence of Organizational Structure on Software Quality (Microsoft Research, ICSE 2008)
• Bird, Nagappan, Murphy, Gall, Devanbu: Don't Touch My Code: Examining the Effects of Ownership on Software Quality (Microsoft Research, ESEC/FSE 2011)
• QSM, Kate Armel: Small Teams Deliver Lower Cost, Higher Quality (2012)
• QSM: 4 Key Studies on Team Size
• Melvin E. Conway: How Do Committees Invent? (Datamation, 1968)
• DORA: Software delivery performance metrics