Study appendix
Everything in the experiment
The whole experiment in one place — the job, the screening tool, and all 22 resumes. To the recruiters, this didn't look like a tidy set: the tool shuffled the order every session — and in the second round displayed an applicant count of more than 400 — so they triaged the way they would on a real, high-volume role. Six of the 22 were the controlled comparison — built to be equally qualified, so the only difference was how each resume was written and laid out. The rest were realistic borderline and non-fit applicants that made the pile feel real.
How the pile worked
- One job. Every resume was written against the same fictional role — a Marketing Manager at a deliberately forgettable mid-market software company in Austin, with six stated requirements. The full job description is further down this page.
- Six equal fits. Six candidates met all six requirements at a similar level. The only thing that differed was how each resume presented it. Those six are the experiment; they are Candidates A–F on the result chart.
- Sixteen other resumes. The rest were built to be realistic: two genuine borderline cases, twelve clear non-fits — too junior, wrong function, overqualified, and so on — and a separate two-resume test described further down. So the recruiter had to actually screen rather than pick from a short list.
- Random order, high-volume feel. The tool shuffled the order every session; in the second round it also displayed a 400-plus applicant count, so recruiters triaged the way they would under real volume.
All 22 candidates are fictional, built for the experiment — and so are their employers and the hiring company; any resemblance to a real company's name is coincidental. Tap any resume below to open the full document.
The screening tool
What the recruiters actually used
We built a screening tool that behaves like a real applicant-tracking system. Recruiters read the job description, worked down a list of applicants, opened resumes, and filed each one — shortlist, maybe, unqualified, or not a fit. That filing is the outcome we report.
The job
The role they were screening for
Every resume in the pile was written against this job description. The six equal candidates all satisfy the required list; the decoys each miss it in a specific way.
Marketing Manager
Northbeam Software · Austin, TX (hybrid, 3 days/week in office) · Full-time
Northbeam Software builds workflow and analytics tools for mid-market B2B teams — about 220 people, profitable, growing across North America. The marketing team is small, and they want a Marketing Manager to own demand generation and scale the pipeline.
What you’ll do
- Own and execute multi-channel demand generation campaigns (paid, email, content, webinars) to drive qualified leads for sales
- Manage marketing automation and CRM workflows (HubSpot) — nurture sequences, lead scoring, funnel reporting
- Plan and run the content calendar across blog, email, and social, partnering with product marketing on messaging
- Track and report campaign performance and pipeline contribution; own the marketing dashboard and present to leadership monthly
- Manage external agencies and freelancers
- Partner closely with sales on lead quality and pipeline goals
What we’re looking for (required)
- 4–7 years of B2B marketing experience (SaaS / tech preferred)
- Hands-on demand generation with measurable pipeline or revenue impact
- HubSpot (or comparable marketing automation)
- Analytics and reporting — owning dashboards and funnel / ROI metrics
- Budget ownership — has managed a marketing / paid-media budget
- Sales partnership — has worked directly with a sales team in a pipeline-driven org
Nice to have
- Experience marketing to operations, logistics, or technical buyers
- Familiarity with account-based marketing (ABM)
- Basic design skills (Figma, Canva) for quick campaign assets
Tier 1 · the experiment
The six equally-qualified candidates
Each of these meets all six requirements at a similar level — 5 to 7 years, $2.4M–$3.1M in pipeline, a $500K–$750K budget, HubSpot, a sales team, the same city. The only variable is presentation, and the shortlist counts are the result. These are Candidates A–F on the chart.
Tier 2 · borderline
The genuine “maybe”s
Two candidates who are close but not clean fits — and the gap is content, not format. They keep the pile honest: a recruiter can’t simply shortlist everyone who looks polished.
Tier 3 · clear non-fits
The realistic rejects
Twelve candidates who don’t fit the role, each for a specific, believable reason. They aren’t cartoonish bad resumes — they’re the kind of near-misses that fill a real pile and take real time to rule out.
A separate test
The twin pair
A second experiment shared the same pile: one deliberately weaker candidate, shown in our format and in a plain format. We are not drawing a conclusion from it — the two versions didn’t state years of experience the same way, so the difference isn’t clean. The full reasoning is in the study’s methodology.