https://agentdiagonstics.com/ Newly added papers, benchmarks, and tools on AI agent diagnostics and optimization, curated by CausaMind. en Wed, 21 Oct 2026 12:00:00 GMT https://arxiv.org/abs/2505.22954 https://agentdiagonstics.com/resources/darwin-godel-machine Wed, 21 Oct 2026 12:00:00 GMT Optimization A self-improving coding agent that iteratively rewrites its own codebase and keeps a change only if coding benchmarks improve — replacing the Gödel machine’s impossible proof obligation with empirical validation. It maintains an open-ended archive of agent variants and grows a branching tree of agents rather than a single lineage; over its runs it raised its own SWE-bench score from 20.0% to 50.0% and Polyglot from 14.2% to 30.7%, sandboxed and with human oversight. https://arxiv.org/abs/2410.04444 https://agentdiagonstics.com/resources/godel-agent Wed, 21 Oct 2026 12:00:00 GMT Optimization A self-referential agent that can inspect and rewrite any part of its own code at runtime — including the routine that rewrites it — applying modifications by monkey-patching the running process. It outperforms hand-designed and meta-learned agents on reasoning and math benchmarks at a fraction of their search cost. https://arxiv.org/abs/2506.10943 https://agentdiagonstics.com/resources/self-adapting-language-models Wed, 21 Oct 2026 12:00:00 GMT Optimization A framework that lets a language model adapt its own weights: the model generates a “self-edit” — training data and update directives — and finetunes on it, while an outer RL loop trains it to write edits that improve post-update performance. https://arxiv.org/abs/2507.21046 https://agentdiagonstics.com/resources/survey-of-self-evolving-agents Wed, 21 Oct 2026 12:00:00 GMT Optimization The first systematic survey of self-evolving agents: what evolves (model, memory, tools, architecture), when, how, and where — plus the evaluation metrics and benchmarks built for evolving agents, and the open problems in safety and scalability. https://arxiv.org/abs/2610.27911 https://agentdiagonstics.com/resources/mmenterprise-bench Wed, 21 Oct 2026 12:00:00 GMT Evaluation A 1,613-question benchmark over evolving multimodal workplace records — documents, chats, figures — testing evidence reasoning, cross-source and temporal conflict resolution, and unanswerability. Across ten proprietary and open-source multimodal models, even the strongest struggle with conflicting or multi-image evidence, and larger retrieval budgets raise recall without consistently raising answer accuracy; code and data are released. https://arxiv.org/abs/2610.27465 https://agentdiagonstics.com/resources/recoverability-and-verifier-strength Wed, 21 Oct 2026 12:00:00 GMT Optimization Defines two measurable quantities that decide whether a self-improvement step can run unattended: verifier strength, the agreement between an automated verifier and a trusted reference, and recoverability, how cheaply a bad change can be undone. On a working platform, the LLM judge silently returned the same score regardless of input in 9 of 37 settings, and only changes with a retained old version could be rolled back without a human; the rule permits prompt, tool, and skill edits unattended but keeps weight updates under human sign-off. https://arxiv.org/abs/2610.27406 https://agentdiagonstics.com/resources/more-memory-worse-agents Wed, 21 Oct 2026 12:00:00 GMT Agent design Errors in a memory store's abstractions are regenerated through an abstraction-execution-re-abstraction cycle, persisting longer than direct reuse alone would explain; the paper names this feedback loop error reproduction. Its Anti-Persistence design accumulates raw trajectories but reconstructs task-specific guidance on demand and discards it after use, achieving higher held-out success at the lowest average LLM-call cost among the memory designs compared. https://arxiv.org/abs/2603.03116 https://agentdiagonstics.com/resources/beyond-task-completion Tue, 20 Oct 2026 12:00:00 GMT Evaluation An audit of tau-bench with procedure-aware evaluation: re-scoring reported successes against the procedures agents actually followed finds 27–78% of them corrupt — success without the required steps. Under gated Pass^4, headline scores collapse (GPT-5 0.58 → 0.24; Mistral 0.46 → 0.03). Because the audit cannot recall every corrupt run, these rates are lower bounds. https://arxiv.org/abs/2610.23599 https://agentdiagonstics.com/resources/trajectories-to-differentiated-skills Mon, 19 Oct 2026 12:00:00 GMT Agent harness Treats each skill as four coordinated parts (Meta, Experience, Tool, Script) updated separately from success and failure evidence. Finds most real skill packages blur these roles; the split method topped all 36 benchmark, model and harness settings tested. https://arxiv.org/abs/2610.21412 https://agentdiagonstics.com/resources/jevagentharness Mon, 19 Oct 2026 12:00:00 GMT Agent harness Frequent small decisions inside a harness, like which tool next, can go to a cheap decision model instead of an LLM call. A 4B model trained on post-mistake states made agents much cheaper and faster with little success loss. https://arxiv.org/abs/2610.21458 https://agentdiagonstics.com/resources/provenance-marks-copied-harnesses Mon, 19 Oct 2026 12:00:00 GMT Agent harness Harnesses leak, and a plagiarist can launder one through a self-evolver. Measured marks mostly fail to survive; survival depends on the rewriting model. The authors' alternative keys a mark to the ordering of near-tied entries in a task-critical lookup table. https://arxiv.org/abs/2610.22869 https://agentdiagonstics.com/resources/ms-eval Mon, 19 Oct 2026 12:00:00 GMT Agent harness An evaluation kit for main agents that delegate to specialised sub-agents over MCP: benchmark, execution harness, delegation metrics and a composite index. Specialisation helped in practice; delegation raised cost, helping most open models but sometimes hurting closed ones. https://arxiv.org/abs/2610.21170 https://agentdiagonstics.com/resources/memscope Mon, 19 Oct 2026 12:00:00 GMT Agent design A patch agent reads execution traces, runtime memory state and the memory architecture's own code to locate failures and apply small bounded repairs instead of rewrites: gains on GAIA, xBench and unseen ALFWorld with a median code change of 4.4%. https://arxiv.org/abs/2610.22469 https://agentdiagonstics.com/resources/testing-planning-claims Mon, 19 Oct 2026 12:00:00 GMT Agent design Agent planning claims checked against hidden PDDL ground truth. The best setup solves 78% of solvable tasks offline but 40% online and recognises unsolvable tasks under 10% of the time; differences trace to the model more than to prompting or architecture. https://arxiv.org/abs/2610.23255 https://agentdiagonstics.com/resources/sample-mix-or-verify Mon, 19 Oct 2026 12:00:00 GMT Optimization How to split a fixed test-time budget between more samples, model mixing and verification when the verifier itself errs. SPLIT uses a small labelled pilot and an exact success-probability formula that accounts for correlated verifier mistakes. https://arxiv.org/abs/2610.21791 https://agentdiagonstics.com/resources/rsiflow Mon, 19 Oct 2026 12:00:00 GMT Optimization A meta-agent reads agent trajectories and feedback to diagnose bottlenecks, picks a component-specific improvement workflow, and distils results into skills and principles that evolve its own harness. Best average across six benchmarks and two backbones in the paper's tests. https://arxiv.org/abs/2610.23574 https://agentdiagonstics.com/resources/darwin-claw Mon, 19 Oct 2026 12:00:00 GMT Optimization Instead of replaying raw trajectories, this framework consolidates experience hierarchically: fine-grained error attributions distilled into updatable prototype vectors that steer skill and memory refinement. Reports gains on WildClawBench. https://arxiv.org/abs/2610.21893 https://agentdiagonstics.com/resources/gacc-bench Mon, 19 Oct 2026 12:00:00 GMT Evaluation 500 goals and 2,571 questions test whether a compressor keeps what matters for a stated goal under a length budget. Even the best of ten compressors trails full context; the right goal helps, the wrong goal actively hurts. https://arxiv.org/abs/2610.23254 https://agentdiagonstics.com/resources/ignored-not-refused Mon, 19 Oct 2026 12:00:00 GMT Evaluation Across 33,000 trajectories from eleven small agents, most failed injections were silently ignored rather than refused, and system-prompt changes alone swung attack success widely. Argues injection evals should report how attacks fail, not just whether they succeed. https://arxiv.org/abs/2610.22511 https://agentdiagonstics.com/resources/auditkit Mon, 19 Oct 2026 12:00:00 GMT Evaluation One result representation for benchmark accuracy, judge scores, runtime behaviour, tool-use traces and retrieval quality, keeping verified outcomes separate from trajectory diagnostics and marking unobservable evidence as unscored rather than failed.