Interview calibration is a session where interviewers score the same real anonymised scorecards.
Interviewers score independently, then reveal their ratings at the same time.
Interviewers then argue about the differences. Quarterly is enough for a stable team.
One measurement tells you it worked: the spread of ratings on the same candidate.
On this page
- What is interview calibration?
- How do you run an interview calibration session?
- What comes out of an interview calibration session?
- What does a usable scorecard line look like?
- How do you know interview calibration worked?
- Why does interview calibration get skipped?
- Common questions
- The short version
Two interviewers, same candidate, same 45 minutes. One writes “strong hire”. The other writes “no”.
Two honest interviewers disagreeing is a measurement problem. Your loop’s output partly depends on who was free that week.
Huffcutt and colleagues pooled 125 reliability estimates in 2013.
Interrater reliability measures how far 2 interviewers agree, from 0 to 1.
Different interviewers in separate rounds reached .44. Panels watching the same interview reached .74.
Interviewer calibration is the fix. It costs about 2 hours a quarter.

What is interview calibration?
Interview calibration is a session where interviewers score the same real evidence independently.
Compare afterwards. Then argue about the differences until the standard is shared.
Interviewer calibration is neither a lecture on interviewing nor a document.
The arguing is the mechanism. Skip it and you have a written bar nobody applies the same way.
How do you run an interview calibration session?
A calibration session starts with 3 anonymised scorecards from real past interviews. Strip the names.
Include at least 1 where the panel disagreed at the time.
Everyone scores alone. Reveal all ratings simultaneously. Start with the widest gap.
Separate disagreement about the evidence from disagreement about the standard. Reveal the actual outcome last.
An interview calibration session is 2 hours and 6 to 10 interviewers. 10 silent minutes per scorecard.
Ratings go on the board at the same time. Sequential reveals anchor the room on whoever spoke first.
Afterwards, write down what changed. Usually 2 or 3 sentences about what a given rating requires.
The written change is the actual output, and it is worth more than the meeting.

What comes out of an interview calibration session?
Four findings recur in almost every interview calibration.
People are scoring different things. One scores whether the candidate reached the answer, another how they got there.
“Senior” means different things per team. The most senior person’s standard is the one everyone else has been guessing at.
Some interviewers never say no. A round that never rejects carries no information, and its owner should know.
The evidence in scorecards is thin. Half of a typical scorecard is adjectives and conclusions with no observation attached.
What does a usable scorecard line look like?
A usable scorecard line has three parts. What you asked, what the candidate did, and what you concluded.
A line with all 3 survives a debrief and an interview calibration session. A line with 1 part is an adjective.
The format itself is in write the scorecard before you write the job ad.
Chose a queue over direct calls without prompting, and named the failure it protects against. When I doubled the load he identified the consumer as the bottleneck before I asked.
Could not explain why the retry count was 3. Asked twice, in different ways. Suggests the code was produced without the decision being made.
Compare with what usually gets written:
Good communication skills. Solved the problem. Seems senior. Would work well with the team.
Nothing in the second set can be argued with or checked. It does not belong in a decision.

How do you know interview calibration worked?
Interview calibration worked if the spread of ratings on one candidate narrows across the panel.
Track it per interviewer too. The person always 1 rating above everyone else is using a different standard.
The remaining disagreements should be about genuinely borderline people. Borderline candidates are what disagreement is for.
Quarterly is enough for a stable team. Monthly for one hiring heavily.
Everyone who interviews attends, the 5-year veterans most of all. New interviewers calibrate before their first real round.
Why does interview calibration get skipped?
Interviewer calibration costs 2 hours of senior engineering time a quarter. It produces no candidates.
Compare it against what 1 bad hire costs. 12 months of salary, and your best engineers’ time.
The arithmetic is in the real cost of a bad engineering hire.

Common questions
How often should a panel calibrate?
A stable panel should calibrate quarterly. During heavy hiring or interviewer onboarding, it calibrates monthly. Two sessions a year keeps a panel roughly aligned. Less than that and the standard drifts back. Every interviewer recalibrates against whoever they last sat in a debrief with.
What material should you calibrate on?
Calibrate on 3 anonymised scorecards from real past interviews, with names stripped. Include at least 1 where the panel disagreed at the time. Invented material produces polite agreement. Real disagreement is the thing you are there to examine. Pick the cases that were genuinely hard.
Why does interviewer calibration get skipped?
Interviewer calibration costs 2 hours of senior engineering time per quarter. It produces no candidates. Compare it against what 1 bad hire costs. 12 months of salary, and your best engineers’ time. And a resignation the team blames on the bar. That comparison is what gets it scheduled.
What if one interviewer is always harsher than the rest?
A harsh interviewer is using a different standard. The session exists to find out which one. Ask what evidence would have moved them to the next rating up. Often the answer reveals they are scoring a level above the role. Show them the written level, then score the same answer again together.
How are interviews scored?
Each interviewer rates the signals their round owns. The scale is agreed by the team in advance. Every rating carries a quote or an action observed. Scores are submitted before the debrief and locked. Without interviewer calibration those scales drift. One candidate then collects a strong hire and a no hire.
The short version
Interviewer calibration is a session where interviewers score real anonymised scorecards independently.
The argument about the differences follows.
Reveal ratings simultaneously. Start with the widest gap.
Separate disagreement about evidence from disagreement about the standard.
Write down what a rating requires. Track the spread. Watch for the interviewer who never says no.
I run this session for teams, either once or across a hiring quarter.
Shadowing and calibration, or a workshop sprint.
The debrief that follows the interviews is its own meeting.
Excel, 3 sheets
Calibration session kit
The 2-hour agenda, a rating grid for 3 scorecards across up to 10 interviewers with the spread calculated per candidate and per interviewer, and the sheet for what changed.
Leave a work email: the file opens right here, and one hiring stage, taken apart, follows every week. One click to leave.
Read next
The Behavioral Interview for Engineers, With 3 Questions That Cannot Be Rehearsed
Every candidate has a rehearsed answer to "tell me about a time you disagreed". Here are the 3 questions that survive rehearsal, and the follow-ups that separate a true story from a prepared one.
Structured Interview Questions, With the Strong Answer Written First
Unstructured interviews explain 4% of later performance. Structured ones explain about 18%. The difference is 3 questions per round and a strong answer written first.
Want this fixed in your loop?
SmartHiring trains engineering interview panels on their own open roles. Elman Huseynov, a Technical Interview & Calibration Trainer in Berlin, runs every engagement. One workshop writes the scorecards for your roles, then calibration runs until the panel agrees. It starts with a free 30-minute call.
No pitch if the loop is already fine. See an example audit report.