Motion forecasting models are mainly evaluated by trajectory-level accuracy metrics. However, these metrics do not directly evaluate which surrounding agents a model relies on for prediction—that is, agent-level reliance. We adopt leave-one-out (LOO) agent removal as a model-agnostic, output-based, linear-cost intervention. It measures how sensitive a model's forecast error is to removing a single agent; we call this quantity agent-removal sensitivity. We formalize this as the LOO Reliance Audit protocol. We evaluate five architecturally distinct motion models on Argoverse 2 and the Waymo Open Motion Dataset, validate the protocol along four aspects—faithfulness, non-reducibility, stability, and convergence—and then derive three findings. Across the five architectures, agent-reliance rankings exhibit near-zero agreement (mean pairwise Spearman ρ ≈ 0.02), consistently reproduced across both benchmarks (F1). LOO reliance is only partially aligned with human-annotated causal agents: in 17.4% of scenes, no model selects any majority-labeled causal agent as its top-1 (F2). In addition, within-model agreement between attention and LOO rankings is very low, suggesting that attention currently does not function as a reliable proxy for agent-removal sensitivity (F3). Overall, current motion forecasting evaluation supports trajectory accuracy but does not support claims about what models rely on for prediction. We propose adding to existing benchmarks the two progress axes defined by the audit—how well a model's reliance aligns with human-annotated causal agents and how well attention and LOO rankings agree within the same model. Code and per-scene LOO scores are publicly released.
The audit hides one agent from the model input, re-runs the forecast, and measures how much its error moves against the same recorded future. For a scene \(S\) and a surrounding agent \(a_i\), the LOO score of \(a_i\) is
\[ \Delta_i \;=\; \mathrm{minADE}_6\big(f(S \setminus \{a_i\}),\, y\big) \;-\; \mathrm{minADE}_6\big(f(S),\, y\big). \]Masking \(a_i\) sets its trajectory coordinates to zero and turns off its valid flag at every timestep, so every evaluated model skips it as absent; the map and all other agents are unchanged. A positive \(\Delta_i\) means the forecast got worse without \(a_i\) and a negative one that it got better; a zero score means no effect under this cost, not necessarily an identical output. We use reliance only as shorthand for this quantity, not as a causal effect, semantic importance, or internal reasoning.
Before reporting findings, we check the measurement along four aspects; each check rules out one way the scores could mislead.
Rules out: the LOO ranking is arbitrary.
Removing the top-5 LOO agents hurts more than a random 5, which hurts more than the bottom 5, in 67% (AV2) / 63% (Waymo) of scenes on average over the five models; chance 1/6.
Rules out: LOO merely re-encodes distance or attention.
Ranking the same agents by distance or by attention gives about 0.20, near chance.
Rules out: small input changes reshuffle the ranking.
Within-model ρ against the original ranking: 0.47 (noise), 0.57 (dropout), 0.55 (dummy injection) on AV2.
Rules out: single-agent deletion disagrees with multi-agent marginal attribution.
LOO top-1 matches Monte Carlo Shapley top-1 in 49.4% (AV2) / 48.9% (Waymo) of scenes; chance ≈ 4%.
Mean cross-model Spearman ρ between LOO rankings is 0.026 on AV2 (max 0.055) and 0.021 on Waymo (max 0.032). The same near-zero level holds across LOO ablation forms, scene stratifications, within-architecture seed ablations, and input perturbations, and retraining each architecture with two additional seeds (15 Waymo checkpoints) confirms that it is not an artifact of a single training run. Under the same mask and outcome, the models rank agent-removal sensitivity very differently despite similar accuracy; this does not identify their internal reasoning.
Against Waymo CausalAgents labels (41,095 annotated scenes), Prec@1—the probability that a model's LOO top-1 belongs to the human-annotated causal set—shows a 2.3–3.5× lift over chance (0.137) for every model. Yet in 17.4% of scenes (\(n = 7{,}135\)) no model selects any majority-labeled causal agent as its top-1, and in 0.65% all five select the same non-causal agent. A mismatch with this reference is not by itself an error: the model may rely on a valid predictive proxy, the annotation may omit a relevant cue, or the label itself may be uncertain. The alignment is partial at the scene level too: when only one model in a pair picks the causal agent, it has the lower minADE6 in 51.3% of cases, near chance.
Two Waymo scenes from the 0.65% in which all five models select the same non-causal agent as top-1. The human-annotated causal agent is spatially and temporally related to the focal vehicle, yet no model picks it as top-1. Dashed trajectories show each agent's future 6-second path.
For the four models with extractable agent-to-agent attention, attention and LOO rankings are effectively unrelated within each model (Spearman ρ ∈ [0.003, 0.033]): what a model says it attends to and what its forecast is sensitive to are separate dimensions, so attention does not yet work as a proxy for agent-removal sensitivity.
Attention top-1 and LOO top-1 point to different agents; unselected causal candidates are also shown.
The rate at which a model's LOO top-1 aligns with human causal labels. It is added alongside minADE, not in place of it: at similar accuracy, closer alignment with a public human reference counts as progress. We propose it not because a mismatch is an error, but because agreement with a public reference is auditable and comparable across models.
The two signals are currently effectively unrelated, meaning attention does not yet work as a reliance proxy. Raising this agreement would make attention a checked proxy for agent-removal sensitivity; closing the gap would not invalidate the LOO instrument.
@inproceedings{park2026beyond,
title = {Beyond Accuracy: Auditing Agent-Removal Sensitivity in Motion Forecasting},
author = {Park, Geonyeong and Park, Byounghun and Kim, Nayoung and Kim, Kyungmin and Hwang, Soonmin},
booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
year = {2026}
}