Software engineering hiring starts with a scorecard template, not a job ad.
On this page
For each thing you test.
Write it before the job ad.
Decide what the person must do in year one. Then what evidence proves they can.
Design the question last.
If you cannot write the strong answer in advance, the question is not ready.
Most hiring problems are decided the moment a team writes a job ad.
Before anyone agreed what a strong candidate looks like.
The fix is boring and it works.
What does an interview scorecard template contain?
Three written levels for every line you score, describing evidence rather than a rating.
Strong describes what the candidate did. Weak describes what they left out.
Now 2 interviewers have something to disagree about that is not taste.
Take “system design”. Useless as a scorecard line on its own.
Useful version:
Strong. Clarified requirements before designing. Estimated scale with numbers. Named a trade-off nobody prompted them about.
Chose a simpler option and said why.
Mixed. Produced a workable design. Needed prompting to consider failure modes. Explained the choice when asked but did not volunteer it.
Weak. Started drawing components without establishing requirements. Could not say what breaks first under load.
Why does writing the scorecard template first change the questions?
Writing the scorecard template first exposes bad questions.
You find questions whose strong answer is “has seen this before”. That is a recall test you did not mean to run.
You find questions where you cannot write a strong answer at all. Those get cut.
At Zalando this removed about a third of our questions before anyone used them.
The order that works
1. What does this person need to do in the first year? 3 to 5 concrete things. Not “be a great engineer”.
Things like “own the ingestion pipeline”. Or “be the person the team asks about Kafka”.
2. What evidence would convince you they can? For each item, what would you need to see. This is the scorecard template.
3. Which question produces that evidence? Now design the interview. This is the last step, not the first.
4. Now write the job ad. It practically writes itself, and it is accurate, which means fewer mismatched applications.
How do you know your scorecard template is real?
Give the scorecard template to 2 engineers who did not write it. Have them score the same recording independently.
Scores more than one level apart mean the scorecard template is not specific enough. Fix it and repeat.
Scoring the same candidate separately is the calibration exercise you should run anyway. It takes an afternoon.
What does this fix in software engineering hiring?
Four things. Debriefs stop being arguments about taste and become arguments about evidence, which converges.
Feedback to candidates becomes possible, because you wrote down what you were looking for.
New interviewers ramp faster instead of guessing at an unwritten bar.
Bias narrows too, because a written standard is harder to bend unconsciously.
Bias narrows rather than disappears. A written standard is still a human one.
The number that justifies the afternoon
Unstructured interviews explain about 4% of on-the-job performance.
Structured technical ones explain 26%.
An afternoon on the scorecard is the difference between those 2 figures.
On every hire you make afterwards.
Where do teams give up?
Teams write the scorecard template for 1 role and never for the next one. Keep them, because most roles are variations.
The second scorecard template takes an hour and the fifth takes 20 minutes.
Treat the first one as a project and you pay that cost on every role.
Where do you keep them?
Keep scorecards in the same place as the job ads, not in someone’s drive.
Date the scorecard and name its author.
A year later you will want to know who agreed to that bar.
A scorecard nobody can find gets rewritten from memory every time.
Rewriting from memory every time is how a team stops writing scorecards at all.
Want this built into the team, calibration included? That is the workshop sprint.
Audit at 500 euro, workshop sprint at 2,500.
Common questions
How long does an interview scorecard template take to write?
An afternoon for the first role. An hour for the second, 20 minutes by the fifth. Most roles are variations on one you have written. The cost people resist is the meeting where the team agrees what the job is. That meeting is the point.
How many lines should a scorecard have?
Four to 6, one per thing the person must do in year one. A 14-field rubric does not get filled in, and engineers are right to refuse it. If a line does not map to something the role requires, cut it. Do not score it out of habit.
Can you reuse a scorecard across levels?
Reuse the lines, rewrite the evidence. Senior and staff are tested on the same dimensions with a different bar under each. Naming an unprompted trade-off is strong at senior and expected at staff. Writing both columns side by side also settles levelling disputes fast.
What is a scorecard?
A scorecard is the written list of signals one interview round tests. Each signal carries the evidence that counts as strong. The scorecard is filled in during the round. Six lines per round is enough. A scorecard written before the question makes two interviewers comparable.
Who should write the interview scorecard template?
The hiring manager drafts it, and the panel argues with the draft. A scorecard template written alone gets ignored by everyone who did not agree to it. One hour with the 4 interviewers surfaces every disagreement about the bar. That is the meeting doing its job.
The short version
Decide what the person must do. Decide what evidence proves they can. Then design the question.
If you cannot write the strong answer in advance, the question is not ready.
Read next
Why Technical Hiring Is Broken
Most loops measure things nobody agreed on in advance. Here is what that costs, and why I started SmartHiring.
Why Your Interview Loop Rejects Good Engineers
Unstructured interviews explain about 4% of job performance. Structured ones explain 26%. Here is where the signal leaks out, and what to fix first.
What a Published Salary Range Changes
Publishing the band changes who applies and what your team finds out later. Including the parts that are uncomfortable.
Want this fixed in your loop?
Book a free 20-minute audit call. I’ll show you where your interview quality breaks and what to fix first.