Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

A new audit of six models using 180 invented-entity questions and six prompt conditions found that validating an automatic scorer with a single overall accuracy or F1 score can hide prompt-specific mistakes, with an overall F1 of .916 in a balanced audit of 288 responses.