In a recent experiment, Anthropic observed that nine autonomous Claude AI instances significantly outperformed human researchers on an open alignment problem within a controlled environment. However, when Anthropic attempted to transfer this winning alignment method to its own production models, the impressive effect vanished.
Alignment research focuses on ensuring AI systems behave as humans intend. With more open research questions than available human researchers, Anthropic sought to investigate whether AI itself could contribute to this crucial work, thereby accelerating discovery.
The experiment centered on a specific scenario: a small, weaker AI model attempting to teach a larger, stronger one which of two chat responses is superior. Such evaluations are critical for training helpful AI systems, but the challenge lies in the "teacher" being less capable than its "student," raising the question of how much of the student's full potential can still be unlocked.
Anthropic measured this using "Performance Gap Recovered" (PGR), where a score of 0 indicates the student performs no better than its weak teacher, while a score of 1 means it reaches its full capability. This scenario serves as a model for a future where humans, as less powerful teachers, supervise superhuman AI.
Nine instances of Claude Opus 4.6 were each provided with their own work environment, a shared forum, and access to an evaluation server. Given only vague initial directions, these "Automated Alignment Researchers" (AARs) operated autonomously, formulating hypotheses, designing experiments, and analyzing results without human intervention.
Two human researchers achieved a PGR of 0.23 after seven days. The nine Claude instances, in contrast, hit 0.97 in just five additional days, effectively unlocking nearly all of the stronger model's potential at an approximate cost of $18,000, demonstrating superior efficiency and effectiveness.
Despite these impressive lab results, real-world performance proved sobering. All experiments were conducted on small, freely available open-source models, specifically Qwen models with 0.5 and 4 billion parameters. When Anthropic attempted to apply the best method to its production model, Claude Sonnet 4, using its in-house training infrastructure, the improvement was statistically insignificant, landing at merely 0.5 points, essentially noise.
Anthropic suspects this discrepancy might relate to how its production model expresses preferences. The researchers only tested a single, simple evaluation method, and other approaches could potentially yield better results. Nevertheless, the company acknowledges a fundamental issue: AARs tend to exploit specific quirks of the models and datasets they are trained on. This suggests that methods effective in a controlled experimental setting do not necessarily transfer to different models or larger scales in production environments.