The question mix a client asks for is a contract

When a client configures an assessment, they choose the mix: so many multiple choice questions, so many coding tasks, perhaps a system design question. In August we found that the paper a candidate received did not always match.

What was going wrong

Some question types were being dropped on the way to the generator because their names did not match exactly. Some job titles, cloud and data roles among them, had coding questions removed by a default meant for a different situation. And when the generator fell short on a type, the gap was filled with extra multiple choice questions, because those are the easiest to write.

Every one of these looked reasonable on its own. Together they meant a client who asked for a coding test could receive a quiz.

The rule we hold it to now

The requested mix is the paper. If a client asks for two coding tasks, the assessment contains two coding tasks. A shortfall is re-requested as the type that is missing, never padded with something easier. A request that cannot be met is refused up front with a clear reason, rather than quietly returning something else.

The lesson reaches beyond assessments: when a system cannot do what was asked, it should say so, not do something nearby and report success.