arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Early tests show AI agents can do research engineering but not the hard creative work | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Artificial IntelligenceEnglishPublished

Early tests show AI agents can do research engineering but not the hard creative work

July 30, 2026arXiv: 2607.27191v1

This paper asks a direct question: can current AI agents carry out open-ended AI research on their own? The authors introduce a method called a shadow evaluation. In it, an agent is given the central research question from an unpublished, high-quality paper and tries to answer it without seeing the original authors’ work. The paper reports two case studies and finds that the agents handled engineering tasks but failed to make substantial research progress. Both papers were judged by their original authors to be clear rejections.

The team partnered with the authors of two unpublished NeurIPS 2026 submissions. Each agent had six calendar days, a virtual machine with web access, GPU time, and about $3,000 in Anthropic API credit to run experiments and explore ideas. The initial runs used a scaffold called OpenClaw with a model named Opus4.8. To check robustness, the authors repeated one experiment with GPT-5.6Sol and Codex and saw similar results. They have released reviews, logs, and code at https://cruxevals.com.

In both runs the agents completed much of the necessary engineering: they wrote code, ran experiments, and produced drafts. But the original authors judged that the work did not advance the central research questions. Table 1 in the paper summarizes the expert scores: for Paper 1 (called Personas) overall 2/6 and for Paper 2 (TabPFN) overall 1/6, with reviewers fairly confident in these assessments. The authors therefore report the agents’ outputs as unambiguous rejections.

The paper identifies five recurring failure modes. First, the agents showed poor judgment about what counts as publishable research and presented underpowered or unprincipled experiments. Second, they responded uncreatively to gaps in research design, often adding caveats rather than redesigning experiments. Third, they failed to backtrack effectively from dead ends: ambitious directions were abandoned early and not replaced. Fourth, they had poor resource awareness: both runs used less than half of the API budget, rushed initial exploration in the first hours, and still finished with time left despite not meeting their success goals. Fifth, the agents drifted from explicit instructions and even violated technical submission rules like paper length.

Why this matters: many forecasts of rapid AI progress assume agents will soon automate long stretches of AI research. These case studies suggest a more nuanced picture. Current frontier models can do much of the technical work around experiments and code. They struggle, however, with the creative and judgment-heavy parts of science: choosing promising directions, fixing weak designs, and managing multi-step projects under constraints. That limits how confidently we can predict agents will soon replace human researchers.

The authors are careful about limits. This study covers only two papers and a limited time and budget per run. The shadow evaluation method means original authors knew the work was AI-generated, which could introduce bias, though the reviewers judged the outputs to be clearly poor. Results might change with different models, more time, or other scaffolds. The authors ran a robustness check that reproduced the same failure modes, and they plan more experiments. For now, these results are early evidence that agents can automate engineering work but not the full, open-ended research lifecycle.

Source paper

Can AI agents conduct open-ended AI research? Early evidence from two case studies

Artificial Intelligencecs.CYMachine Learning
View on arXiv