Last week in AI evaluation and agentic AI
Most agent benchmarks hand the agent a complete brief. Real requests often do not arrive that way, and a paper posted to arXiv on 13 October 2026 measures what that gap costs.
AM-Bench, from Kaiyu Huang and colleagues, takes 48 tasks from Terminal-Bench 2.0 and Harness-Bench and rewrites each one six ways. Something is left vague or left out entirely: the goal (what done looks like), the context (the facts needed to get there), or a constraint (a rule the agent has to respect). The authors then ran five models through four agent harnesses, the wrapper that gives a model its tools and its loop, for 20,160 runs in total.
Two results stood out to me.
When a constraint was missing, the paper reports a success rate of 0%. Agents still finished tasks, and every finished run failed the constraint checks. A grader that only asks whether the task completed would have counted those runs as wins.
Rankings moved too. The two models that led with the full brief fell to fourth and fifth on the underspecified average, according to the authors. Swapping one harness for another raised one model's success while lowering another's.
The authors also estimated how much asking a clarifying question would recover, and it was uneven across the three gaps.
My take: a leaderboard score describes a model, a harness and a perfectly written prompt together. Change any one of them and you are holding a different number.
Quick check before you scroll: In AM-Bench, which of the three kinds of missing information did asking a clarifying question barely fix?
If your network builds with AI, a repost puts this in front of people who don't follow me yet.
Full breakdown + the answer: frankduah.me/learnings/2026-10-19-last-week-in-ai-evaluation-and-agentic-ai
New here? I post a bite-size AI / ML concept like this every day. Follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#AI #LLM #AIAgents #MachineLearning
The answer
Constraints. The authors estimate that clarification lifted compliance by only 4.5 points, leaving it around 30%, while a goal gap recovered about 15 points of success. My read: an agent rarely asks about a rule it never knew existed, which is why it finishes confidently and still breaks it.