This paper asks a direct question: can current AI agents carry out open-ended AI research on their own? The authors introduce a method called a shadow evaluation. In it, an agent is given the central research question from an unpublished, high-quality paper and tries to answer it without seeing the original authors’ work. The paper reports two case studies and finds that the agents handled engineering tasks but failed to make substantial research progress. Both papers were judged by their original authors to be clear rejections.

The team partnered with the authors of two unpublished NeurIPS 2026 submissions. Each agent had six calendar days, a virtual machine with web access, GPU time, and about $3,000 in Anthropic API credit to run experiments and explore ideas. The initial runs used a scaffold called OpenClaw with a model named Opus4.8. To check robustness, the authors repeated one experiment with GPT-5.6Sol and Codex and saw similar results. They have released reviews, logs, and code at https://cruxevals.com.

In both runs the agents completed much of the necessary engineering: they wrote code, ran experiments, and produced drafts. But the original authors judged that the work did not advance the central research questions. Table 1 in the paper summarizes the expert scores: for Paper 1 (called Personas) overall 2/6 and for Paper 2 (TabPFN) overall 1/6, with reviewers fairly confident in these assessments. The authors therefore report the agents’ outputs as unambiguous rejections.

The paper identifies five recurring failure modes. First, the agents showed poor judgment about what counts as publishable research and presented underpowered or unprincipled experiments. Second, they responded uncreatively to gaps in research design, often adding caveats rather than redesigning experiments. Third, they failed to backtrack effectively from dead ends: ambitious directions were abandoned early and not replaced. Fourth, they had poor resource awareness: both runs used less than half of the API budget, rushed initial exploration in the first hours, and still finished with time left despite not meeting their success goals. Fifth, the agents drifted from explicit instructions and even violated technical submission rules like paper length.