Selection: interviews, assessments and validity

Leo describes how Kestrel hires engineers. A recruiter screens résumés for strong schools and well-known previous employers. Candidates then meet four engineers, one at a time, each of whom asks whatever questions they think best and reports a feeling of yes or no. If most say yes, an offer goes out. Leo admits the results are mixed: some hires have been outstanding, others have struggled, and two engineers who were rejected by Kestrel became stars at a competitor. Worse, he cannot say which part of the process works, because nothing has ever been measured. Dr. Rahimi says Kestrel's process is typical, and that a century of research on selection offers a clear answer to how it could be better.

What selection is trying to do

Selection is the process of choosing among candidates the ones most likely to perform well and stay. It is a prediction problem: the organization observes candidates for a few hours and tries to forecast years of behavior. Every selection method is a predictor, and its value depends on how well it predicts the criterion, usually job performance and sometimes retention, safety or other outcomes. Two technical properties matter. Reliability is consistency: a method that gives different results for the same candidate on different days, or from different interviewers, cannot predict well. Validity is accuracy: the degree to which scores on the method relate to the outcome it is supposed to predict.

Criterion-related validity is usually expressed as a correlation between predictor scores and later performance, from 0 for no relationship to 1 for perfect prediction. In selection research, correlations are modest by the standards of physics. A validity of 0.3 is useful and 0.4 is strong. Dr. Rahimi explains why modest correlations matter: when many candidates apply for few positions, even a moderate improvement in prediction lifts the average performance of those hired noticeably, and the gain accumulates over every hire for years. Content validity, the extent to which a method samples the actual work, is a second form of evidence, and it is why work samples and job knowledge tests based on careful job analysis are easy to defend.

How well do common methods predict?

For decades, the standard summary of selection research was a 1998 review by Frank Schmidt and John Hunter, which ranked general mental ability tests and work samples highest, with validities above 0.5, and structured interviews close behind. In 2022, Paul Sackett and colleagues re-examined the corrections used in that work and concluded that earlier estimates had been inflated, especially for cognitive ability. In their revised estimates, structured interviews came out on top, followed closely by job knowledge tests and work samples, with cognitive ability tests lower than previously believed and unstructured interviews well behind.

The ranking tells Leo that his process relies on the weakest methods. Résumé screening for prestigious schools and employers mostly measures where candidates have been, which is related to ability but also to background and luck. Years of experience, often the main filter, predicts performance only weakly beyond the first year or two. And his four unstructured interviews are, in effect, four independent conversations, each with low reliability, combined by a vote that gives the most confident interviewer the most weight.

Structured interviews

Structured interviews are the most practical improvement for most organizations because almost every employer interviews anyway. Structure has several elements. Questions are derived from job analysis and asked of every candidate in the same way. Behavioral questions ask what the candidate did in past situations: tell me about a time you had to fix a production problem under time pressure. Situational questions ask what the candidate would do in a realistic future situation. Every answer is rated on an anchored scale, with examples of poor, acceptable and excellent answers written in advance. Interviewers take notes and rate independently before discussing. Each interviewer covers different competencies rather than repeating the same conversation, and the final decision combines ratings mechanically rather than by persuasion.

Interviewers resist structure, because it feels mechanical and because people trust their own judgment of others. Research on intuition is not kind to that trust. Unstructured interviews are vulnerable to first impressions, similarity to the interviewer, halo effects in which one striking trait shapes every rating, and the influence of irrelevant cues such as handshake, accent or appearance. Daniel Kahneman described the problem of noise: different interviewers judging the same candidate disagree far more than they realize. Structure reduces both bias and noise. Dr. Rahimi adds that structured interviews also give candidates a fairer experience, because every candidate has the same chance to show what they can do.

Tests, work samples and assessment centers

Job knowledge tests measure the technical knowledge a role needs, such as electrical codes for an electrician or pharmacology for a nurse. Work samples ask candidates to perform part of the job: a coding exercise, a mock sales call, a forklift test, a simulated patient handover. They predict well, are hard to fake and are perceived as fair by candidates, but they must be designed for each job and can be expensive to run. Assessment centers combine several exercises, such as in-basket tasks, group discussions and role plays, observed by trained assessors; they are used mainly for managerial roles.

Cognitive ability tests measure general reasoning, verbal and numerical ability. They predict performance across many jobs, especially complex ones, and are cheap to administer. Their drawback is that average scores differ between some demographic groups, so using them as a hard cut-off can produce adverse impact, discussed below. Personality measures, mostly based on the five-factor model, predict less on average; conscientiousness is the most consistent predictor across jobs, and other traits matter for specific roles, such as extraversion for sales. Integrity tests predict counterproductive behavior such as theft. Situational judgment tests present realistic dilemmas and ask candidates to choose the best response.

References and background checks

Reference checks are widely used and weakly predictive in their usual form, because candidates choose referees who will praise them and many employers will confirm only dates and titles. Structured reference checks, which ask several referees the same specific questions about past behavior and rate the answers, do better. Background checks verify identity, education, licenses and, for some roles, criminal records or driving records.

In the United States, background checks obtained from consumer reporting agencies are regulated by the Fair Credit Reporting Act, which requires disclosure and consent and a specific process before rejecting a candidate based on a report. Many states and cities, including Illinois and Chicago, have ban-the-box laws that bar questions about criminal history until later in the process, and the EEOC has long warned that blanket exclusions for criminal records can have a disparate impact. Illinois also prohibits using conviction records unless there is a substantial relationship to the job or an unreasonable risk. Drug testing rules vary widely by state, especially for cannabis, which many states, including Illinois, have legalized for adult use.

Adverse impact

A selection method has adverse impact when it selects members of one group at a substantially lower rate than another. It is not in itself illegal, but it requires the employer to show that the method is job-related, and it is a signal to look for alternatives that predict as well with less impact. The common screening rule is the four-fifths rule: if the selection rate for any group is less than 80 percent of the rate for the group with the highest rate, the method is flagged for review. With small numbers, the rule can mislead, and statistical tests of the difference are used as well.

The four-fifths rule comes from the Uniform Guidelines on Employee Selection Procedures adopted by federal agencies in 1978. Federal enforcement of disparate impact has narrowed since 2025, and a June 2026 Department of Justice opinion challenged the EEOC's disparate impact guidance, but disparate impact remains part of Title VII for private plaintiffs and of several state laws, including Illinois's. An employer that can show a method was developed from job analysis, validated for the job and chosen over alternatives with similar validity is in a far stronger position than one that cannot.

There is a well-known tension between validity and adverse impact. Cognitive ability tests are valid but tend to show group differences; structured interviews, work samples and personality measures show smaller differences. Combining methods, and weighting them sensibly, can keep most of the validity while reducing impact. Dr. Rahimi stresses a point often missed: unstructured interviews are not a safe alternative. They have lower validity and are open to the very biases the law prohibits; their impact is simply harder to measure.

The value of better selection

How much is better selection worth? Utility analysis, developed by Hubert Brogden, Lee Cronbach and Goldine Gleser in the mid-twentieth century, gives an estimate. The gain from using a method is the number of people hired, times the years they stay, times the validity of the method, times the standard deviation of performance in money terms, times the average standardized score of those hired, minus the cost of the method. The standard deviation of performance in money terms is often estimated at about 40 percent of annual pay for professional jobs. The average score of those hired depends on the selection ratio, the share of applicants hired: the more selective the employer, the higher it is.

# Utility of moving Kestrel's engineering hiring to structured interviews plus a work sample
from statistics import NormalDist
nd = NormalDist()

hires_per_year, years_stay = 20, 3.0
pay = 150_000
sd_y = 0.40 * pay                      # money value of one standard deviation of performance
r_old, r_new = 0.19, 0.45              # unstructured interviews vs combined structured methods
applicants = 400
ratio = hires_per_year / applicants    # selection ratio
cut = nd.inv_cdf(1 - ratio)
z_mean = nd.pdf(cut) / ratio           # average standardized score of those hired
extra_cost = applicants * 250 + 60_000 # assessment time per applicant plus design

gain = hires_per_year * years_stay * (r_new - r_old) * sd_y * z_mean - extra_cost
print(f"selection ratio {ratio:.0%}, average z of hires {z_mean:.2f}")
print(f"estimated gain per year of hiring: {gain:,.0f}")

# Adverse impact check on the new work sample (four-fifths rule)
groups = {"group A": (180, 18), "group B": (120, 9)}   # applicants, passed to offer
rates = {g: p / a for g, (a, p) in groups.items()}
top = max(rates.values())
for g, r in rates.items():
    print(f"{g}: selection rate {r:.1%}, ratio to highest {r / top:.2f}",
          "flag for review" if r / top < 0.8 else "within four-fifths")

selection ratio 5%, average z of hires 2.06
estimated gain per year of hiring: 1,770,699
group A: selection rate 10.0%, ratio to highest 1.00 within four-fifths
group B: selection rate 7.5%, ratio to highest 0.75 flag for review

The estimate is large because the inputs compound: twenty hires a year, three years each, a substantial rise in validity, a high standard deviation of performance for skilled engineers and a selective process. Dr. Rahimi urges caution with the precise figure, since the standard deviation of performance is an estimate and the validity of a new process must be checked locally. Still, even if the gain were a quarter of the estimate, the new process would pay for itself many times over. The adverse impact check shows that the work sample's selection rate for group B is about three quarters of the rate for group A, below the four-fifths threshold, so Kestrel should review the exercise: is a time limit stricter than the job requires, or does it assume knowledge of tools that only some candidates have used?

Combining information and making the decision

Research since Paul Meehl's 1954 book on clinical versus statistical prediction has found that combining information by a consistent formula usually predicts better than combining it by holistic judgment, even when the judges are experts. The reason is consistency: people weigh the same information differently on different days and are swayed by vivid details. The practical lesson is to decide in advance which methods will be used and how they will be weighted, to have each assessor score independently, and to use discussion to resolve missing information rather than to overturn scores. Kahneman suggested a further safeguard: delay the overall intuitive judgment until all the structured scores are in.