Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone

The paper critiques the prevalent practice of evaluating alignment in machine learning solely through model-level assessments, arguing that such evaluations do not adequately reflect deployment-relevant alignment. It emphasizes the need for alignment claims to be tied to the specific level of evidence collection, whether that be model-level, response-level, interaction-level, or deployment-level. Two studies are presented to support this argument, highlighting the limitations of existing alignment benchmarks.