.webp?dpl=dpl_DNXyzGHGUwQfFxeATajTfJeGNMDH)
Why do my interviewers score the same candidate differently?
Two people interviewed one candidate on the same afternoon. One came back with a strong hire and the other with a pass, and now you are in a debrief trying to work out which of them is right. Usually neither is wrong. They scored different things, on scales they each carry in their heads. This page names the five causes and gives you a method that removes them before the next candidate walks in.
The five reasons the scores come back far apart
Almost every wide split traces to one of these, and most trace to two or three at once.
They asked different questions
One interviewer asked about strategy, the other asked about a launch that missed. Each scored the answers they heard. They were never scoring the same candidate, only the same name.
There is no written scale
“Strong hire” and “hire” mean whatever each person has learned to mean over their career. Two people can agree on everything they heard and still pick different words for it.
The scale has numbers but no anchors
A 1 to 5 column with nothing under it is a mood. Without a sentence describing what a 3 sounds like, a 3 from a generous interviewer and a 3 from a tough one are different scores wearing the same digit.
Scores are written after the conversation
If people talk first and fill in the scorecard afterwards, the scorecard records the meeting, not the interview. The first strong opinion in the room moves every score written after it.
The loudest voice decides
When the founder or the most senior person says “I loved her” before anyone else has spoken, the quieter interviewer with the sharper evidence rounds toward it. The decision was made by rank.
None of these is a people problem. Both interviewers did their job with the tools they were handed, and the tools were an empty notes field and a word like “strong.” The gap is in what was written down before the loop started, which was nothing.
A 5 and a 2 on the same candidate
The company and the people are invented; the shape of the disagreement is the common one.
A twenty-five person company is hiring its first marketing manager. Dana, the founder, and Leo, who runs sales, each spend forty-five minutes with the same candidate. The scorecard is a shared doc with four rows and a 1 to 5 column. On the row labelled “owns outcomes,” Dana writes a 5 and Leo writes a 2.
Dana asked how the candidate would approach the first ninety days. The answer was organized and confident: audit the funnel, pick two channels, set a pipeline target. Dana heard someone who thinks in outcomes. Leo asked about the last campaign that missed its number. The candidate described the campaign well and, when Leo asked what the target had been and what they changed after, gave a general answer about learning and iterating. Leo asked twice more and did not get a number. Leo heard someone who has not owned one.
Neither of them misheard anything. They scored two different questions under one label, on a scale where nobody had written what a 3 was, and the debrief opened with Dana saying “I really liked her,” so Leo spent the meeting defending a 2 instead of reading out what he heard. Written before the loop, the row would have said what “owns outcomes” means for this role: names a target they were accountable for, says whether they hit it, and describes what they changed when they did not. Both interviewers would have asked for that, both would have noticed the missing number, and the split would have been a 2 and a 2 with one clear follow-up for the final round.
How to make the scores comparable
Five steps, and all but one happen before the first interview. The aim is to put every candidate on the same scale, so that when two interviewers disagree, you can see exactly why.
1. Write the role outcomes and the scale before the loop starts
Decide what this hire has to deliver in the first year, then write what you need to hear in an interview to believe a candidate can do it. That is the scorecard. Do it before the first call, because after the first call you are writing it around a person.
2. Give every interviewer the same guide and the same anchored scale
Each interviewer covers assigned areas with written questions, and each area is scored 0 to 4 with a sentence for every number describing what that level of evidence sounds like. A 0 means not enough information, never “bad.” The anchors are the part doing the work; the digits are just how you write them down.
3. Collect the scores before anyone talks
Every interviewer submits their own scorecard, with the evidence behind each score, before the debrief. This is the cheapest rule on the page and the most skipped. It is the only way a score can be a record of the interview.
4. Run the debrief on evidence, not impressions
Open with the disagreements. For each one, ask what each person actually heard, not how they felt. The hiring manager speaks last. How to run an interview debrief walks through the meeting itself, with the questions in order.
5. Treat disagreement as information about the interview
A 4 and a 1 on the same competency tells you one interviewer asked a question the other did not, or the anchor is ambiguous, or one candidate answer was never tested with a follow-up. Fix that for the next candidate. The disagreement did its job.
How to build an interview scorecard covers step one and two in detail, with a worked card. How to run an interview debrief covers step four, with the kickoff questions in order. Where the scorecard sits in the rest of the loop is in how to build a hiring process.
What to do with an interviewer’s gut feeling
Gut feeling is a signal to check, not a score. A veteran who says “something is off” is usually right that something is off, and usually cannot yet say what. The scorecard is how you find out. Ask what they heard that produced the feeling, and in most cases there is a specific answer that sounded fine and was never tested with a follow-up.
Write down what the instinct is reacting to, then go and check it. Follow-up questions are the checking tool: what exactly did you do, what was the number, who else was involved, what happened next. How to ask follow-up questions lays out the seven directions a follow-up can go. An interview guide with those follow-ups written under each question is what turns a hunch into a score another person can read.
What this looks like in Yardstick
Everything above works in a shared document if the person running the loop has the time to write the guide, the anchors, and the follow-ups for every role. Most do not, which is why the scorecard ends up as four rows and an empty column.
In Yardstick you describe the role and AI designs the interview guide for it before anyone is scheduled: which areas each interviewer covers, the questions with the follow-ups already under them, and a scorecard with a written anchor for every score. The draft is wrong in places and faster to fix than to write. Every candidate then has the same important conversation, and every interviewer scores from the same card, so a 4 and a 1 mean the two of them heard different evidence rather than used different scales. Each interviewer submits their own scorecard, and the scores line up side by side for the debrief.
Two built-in agents take the chasing off you. The chase missing scorecards agent sends reminders to interviewers who have not submitted and escalates the final missing card to the hiring manager. Once the assigned scorecards are complete, the prepare a panel decision brief agent turns them into one brief for the team to review, so the debrief opens on the evidence instead of on whoever speaks first. Interview scorecard software has the details of the scorecards themselves.
Frequently asked questions
Is it normal for interviewers to disagree about a candidate?
Yes, and some of it is useful. Two interviewers who covered different areas should come back with different evidence, and a candidate who is strong on the work and weak on collaboration should produce a split score. The disagreement to fix is the one where two people heard the same thing and wrote down different numbers, or scored two different questions under the same label. A written scale with anchors and one interview guide per role removes that kind, and leaves the kind you want.
Should everyone on the panel ask the same questions?
Every candidate should get the same interview, which is different from every interviewer asking the same questions. Split the areas across the panel so each interviewer owns two or three and goes deep on them, with follow-up questions written in advance. What must not change is the set of questions from one candidate to the next, because that is what makes the scores comparable.
Does a scorecard replace an interviewer's judgment?
No. The scorecard is where the judgment gets written down in a form other people can check. When an experienced interviewer has a bad feeling, the right response is to ask what they heard that produced it, and usually there is a specific answer that never got a follow-up. The instinct found the gap; the scorecard makes sure the gap gets probed instead of argued about.
Why won't my interviewers fill in their scorecards?
Usually because the card asks them to type free-form notes into an empty box, and they cannot see what the notes are for. A card with the questions they asked, a scale with a sentence under each number, and a place for the evidence takes five minutes and is visibly used in the debrief. In Yardstick the scorecard for each interview carries those anchors, and the chase missing scorecards agent sends reminders to interviewers who have not submitted and escalates the final missing card to the hiring manager.
What does Yardstick do about interviewer disagreement?
Yardstick designs the interview guide for each role with AI before anyone is scheduled: the areas, the questions with follow-ups, and a scorecard with a written anchor for every score, which you edit before the first interview. Every interviewer scores from the same card, each interviewer's scorecard is their own, and the scores line up side by side for the debrief. When the assigned scorecards are in, the prepare a panel decision brief agent turns them into one brief for the team to review. Your first three Jobs are free.
Put every candidate on the same scale before the next loop.
Describe the role, get the interview guide and the anchored scorecard for it, and hand both interviewers the same questions and the same scale. Your first three Jobs are free.