APforge

One engineering investigation, three versions of the agent

Every phase below runs real engine code and real tool executions. The naive agent (V1) guesses instruments, fails, reflects, remembers, and updates its strategy — by V3 it solves everything with zero wasted calls, including tasks it has never seen.

Live learning loop — turn by turn

Fresh session on the real benchmark. V1’s strategy is naive, so the first turns pick the wrong instrument; the loop reacts with evaluation, reflection, memory, and strategy deltas until the same task types succeed. Click a turn to see its full trace.

Persistent tool-use memory (accumulated during the session)

Strategy state after the session (prior + learned delta)

V1 → V2 → V3 across all four suites

Real executions of the evolution benchmark, measured on instances. Cost is a modeled per-instrument charge attached to real call counts; latency is wall-clock time around each real call.

Learned strategy & persistent memory changes

From the benchmark’s training phases: how tool selection shifts between phase A (early, decoy-heavy) and phase B (converged to the correct instrument), plus the live session’s memory and strategy state.

Unseen-task generalization

V1/V2/V3 measured on unseen instances — bundles the agent never executed during training. If the learned tool-use strategies transfer, later versions must solve novel instances with no retraining.

Real call traces: the same scenario before and after learning

The actual per-call records recorded by the benchmark for one held-out scenario — first when the agent was naive (V1), then after both learning phases (V3).

Where these numbers come from