Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents

We consider a victim agent $\mathcal{A}$ that solves a sequence of tasks while maintaining a persistent experience memory. For the $i$-th task $x_{i}$, the victim produces an execution trajectory $\tau_{i}\sim\pi_{\mathrm{vic}}(\cdot\mid x_{i},\mathcal{M}_{i-1})$, where $\mathcal{M}_{i-1}$ denotes the persistent experience state available to the victim before processing $x_{i}$. The trajectory $\tau_{i}$ abstracts the agent’s execution trace, including its actions, tool interactions, and final output.

After task completion, the victim distills the task and its trajectory into a reusable experience $e_{i}=\Phi(x_{i},\tau_{i})$, where $\Phi$ denotes the victim’s native experience-extraction process. The resulting experience may take the form of a reflection, procedural rule, or reusable skill, and is incorporated into the persistent memory, yielding the updated memory state $\mathcal{M}_{i}$.

Let $\mathcal{M}_{0}$ denote the victim’s memory state before the attack, which may be empty or contain pre-existing experiences. The adversary submits a sequence of benign tasks $\mathcal{T}_{\mathrm{adv}}=(t_{1},\ldots,t_{n}).$ Following the victim’s self-evolution process, each task $t_{i}$ induces an experience $e_{i}^{a}$. We use $\mathcal{M}_{n}$ to denote the post-attack memory state after all attacker-induced experiences have been incorporated.

At the target stage, the victim receives a safety-sensitive query $q$ in a fresh session, where the prior interaction context is unavailable but the persistent experience memory remains. Let $J_{\mathrm{harm}}(q,\mathcal{M})\in[0,1]$ denote the victim’s harmful-compliance score under target query $q$ and memory state $\mathcal{M}$. We define the experience-induced safety degradation as

The attack is subject to an atomic benignness constraint: each submitted task and each induced experience must be benign when evaluated in isolation,

where $B(\cdot)$ denotes a benignness auditor.

We consider a gray-box adversary that interacts with the victim through its task interface and has read-only observability of the experiences generated through its own interactions.

Given an initial safety-sensitive query $q_{0}$, EvoBreak constructs an adaptive sequence of attack-stage interactions and a final target-stage query. Because the adversary controls the submitted tasks but not the experiences ultimately distilled by the victim, a predetermined decomposition of $q_{0}$ or a fixed task sequence may become ineffective when the generated experiences are incomplete, unexpected, or redundant.

EvoBreak therefore adopts a closed-loop strategy that replans after each interaction based on the experiences. The attack consists of two stages: experience-conditioned acquisition and experience-aware target reformulation.

To determine which target-relevant requirements remain insufficiently represented by the experiences generated so far, EvoBreak introduces an experience residual $r_{i}$, an explicit textual planning state jointly generated with the next action:

where $\mathcal{T}$ denotes the space of candidate tasks. The residual $r_{i}$ summarizes the target-relevant knowledge, procedures, or constraints that remain absent in the accumulated experiences. It is inferred solely from $q_{0}$ and $\mathcal{H}_{i}$, rather than from a predefined decomposition of the target.

If $a_{i+1}\neq\textsc{Stop}$, the selected action is submitted to the victim as the next task, denoted by $t_{i+1}=a_{i+1}$. The victim executes the task and generates a new experience $e_{i+1}^{a}$. EvoBreak then updates the observable history as $\mathcal{H}_{i+1}=\mathcal{H}_{i}\oplus\left(t_{i+1},e_{i+1}^{a}\right)$ and replans from the updated history. This closed-loop process enables EvoBreak to revise its attack trajectory according to the experience actually produced at each interaction. The experience-acquisition stage terminates when EvoBreak selects Stop. Let $k$ denote the number of completed interactions at termination.

Training experience-conditioned planning requires targets whose local requirements can be elicited separately yet must be jointly composed. Existing safety datasets largely consist of standalone harmful queries, offering little control over their decomposition and dependency structures. We therefore introduce BreakGym, a scalable structure-first pipeline for synthesizing structurally controllable training targets.

BreakGym defines each structural configuration as

where $d_{\mathrm{dec}}$ specifies the decomposition principle, $d_{\mathrm{top}}$ specifies the composition topology, and $d_{\mathrm{tgt}}$ specifies the safety-sensitive target domain. Given a sampled configuration $c$, BreakGym constructs a training instance $\xi=(q_{0},G,c),$ where $q_{0}$ is a synthetic safety-sensitive query and $G$ is a latent scaffold encoding its local requirements and dependency relations. The scaffold specifies the internal organization of $q_{0}$ without prescribing the interaction trajectory of EvoBreak. It is accessible only during data construction and training.

Stage 1. Structure Sampling. Given a sampled configuration $c$, BreakGym constructs a latent scaffold $G=(\mathcal{V},\mathcal{L})$, where each node $v\in\mathcal{V}$ represents a local target requirement, and $\mathcal{L}$ encodes the structural relations among the requirements according to the sampled composition topology.

Stage 2. Target Instantiation. BreakGym instantiates the scaffold within the sampled target domain to generate a coherent safety-sensitive query $q_{0}\sim p_{\mathrm{syn}}\left(\cdot\mid G,d_{\mathrm{tgt}}\right)$. Because the local requirements and their dependencies are specified before generation, the structure of the query is explicitly controllable. BreakGym can vary domains, contexts, and entities to scalably generate diverse decomposable queries.

Stage 3. Scenario Validation and Filtering. BreakGym validates whether each synthesized query faithfully reflects its scaffold while remaining coherent and natural as a standalone request. Queries whose local requirements are indistinguishable, weakly complementary, or inconsistent with the specified dependency structure are discarded. The remaining instances with similarity to downstream evaluation prompts are removed to prevent data leakage.

EvoBreak exposes attack as a sequential and trainable process. As shown in Figure 1, BreakGym supports a cascaded optimization workflow: rejection-sampling SFT first initializes the sequential attack policy, followed by Hint-guided GRPO for further exploration under outcome-level feedback.

We then perform standard supervised fine-tuning on the complete attack trajectories:

where $o_{i}^{\zeta}$ denotes the $i$-th model-generated output in rollout $\zeta$, and $\mathcal{C}_{i-1}^{\zeta}$ denotes its preceding trajectory context.

At each interaction step, the attack model first generates an initial residual–action pair. The critic evaluates this decision using the latent scaffold $G$ and returns no hint when the decision is appropriate. Otherwise, it provides a local corrective hint, based on which the attack model regenerates the decision. Only the final decision is retained for rollout construction. The hint critic and the latent scaffold are used only during training and are unavailable at inference time.

Let $S_{\mathrm{atk}}(\zeta)\in\{0,1\}$ indicate whether the attack succeeds. For each rollout $\zeta$, we define the trajectory-level reward as

where $0<\lambda<1$. The first term rewards attack success, while the second measures the average benignness.

For each target, we sample a group of $M$ rollouts $\{\zeta_{j}\}_{j=1}^{M}$ and normalize their rewards within the group to obtain the group-relative advantages $A_{j}$. We then optimize EvoBreak using the standard GRPO objective:

where $\rho_{j}$ denotes the policy ratio for rollout $\zeta_{j}$, $\pi_{\mathrm{ref}}$ is the reference policy, and $\beta$ controls the KL penalty.

Table 1 systematically compares EvoBreak with five competitive baselines across two victim models, two self-evolving frameworks, three pre-evolution domains, and two safety benchmarks. We highlight three main observations.

First, EvoBreak consistently achieves the strongest attack performance across all settings. Its overall average ASR reaches 86.12%, outperforming the strongest baseline, ReNeLLM, by 24.19 percentage points. The consistent gains over both query-level and memory-oriented attacks demonstrate that benign experience composition is an effective attack surface for self-evolving agents.

Second, EvoBreak is robust to heterogeneous self-evolution mechanisms and pre-evolution domains. It maintains consistently high ASR on both SE-Agent and ReasoningBank, as well as across mathematics, code, and general reasoning domains. This robustness suggests that EvoBreak does not rely on a specific experience representation or initial memory distribution. Instead, its experience-conditioned acquisition process continually replans based on the experiences actually generated by the victim, adapting the attack trajectory to different evolutionary settings.

Third, EvoBreak generalizes effectively across victim backbones. Unlike attacks that primarily exploit model-specific prompt vulnerabilities, EvoBreak leverages the general experience accumulation and reuse mechanism of self-evolving agents. Its experience-aware target reformulation further aligns the final query with the accumulated benign experiences, facilitating their joint reuse and weakening the victim’s refusal boundary.

Figure 3 reports the average ASR and benignness on Llama-3.1-8B-Instruct across self-evolving frameworks and pre-evolution domains. EvoBreak achieves the best effectiveness–benignness trade-off on both benchmarks. Query-level attacks directly rewrite or obfuscate the target request, but must still expose sufficient harmful intent within a single query, limiting either ASR or benignness. AgentPoison achieves stronger attack performance through direct injection of malicious memory records, requiring privileged access and making the attack more detectable.

In contrast, EvoBreak’s high benignness is supported by its training and adaptive planning design. BreakGym teaches the attacker to acquire complementary requirements through separate interactions rather than exposing the complete harmful intent in a single task. Rejection-sampling SFT filters out trajectories containing non-benign tasks or experiences, while Hint-guided GRPO explicitly rewards benign intermediate interactions. Moreover, experience-conditioned planning adapts each subsequent task to the experiences actually generated by the victim, maintaining target relevance without resorting to overtly harmful requests. These mechanisms allow EvoBreak to achieve high benignness while preserving strong attack effectiveness.

To evaluate the contribution of each component, we conduct systematic ablations, with results reported in Table 2.

First, training and optimization substantially improve attack effectiveness. The untrained variant still achieves non-trivial success, indicating that experience acquisition and target reformulation are effective even without specialized training. Rejection-sampling SFT provides strong supervision from successful and benign trajectories, while GRPO further improves performance; the additional gains from Hint-guided GRPO show that local corrective guidance benefits intermediate planning under sparse trajectory-level rewards.

Second, experience-conditioned replanning is critical. A fixed task decomposition cannot account for the experiences actually distilled by the victim, which may be incomplete, redundant, or off-target. By planning over the realized experience state, EvoBreak continually targets uncovered requirements and constructs a more complementary experience set.

Third, experience acquisition and target reformulation are complementary. Acquisition constructs the target-relevant experience basis, whereas reformulation activates and composes these distributed experiences at the target stage. Removing either component reduces the attack to unsupported query rewriting or poorly activated experience accumulation, demonstrating the importance of coupling experience construction with experience activation.

To examine whether BreakGym provides effective supervision for training EvoBreak, we conduct an SFT-only comparison using different training sources, including AdvBench (35), Do-Not-Answer (29), WildJailbreak (15), and Jailbreak_LLMs (23). All variants share the same framework, backbone model, and action space, and GRPO is excluded to isolate the effect of supervision data.

As shown in Figure 4, BreakGym provides the strongest supervision among all training sources. Unlike existing safety datasets, which mainly contain standalone harmful requests, BreakGym explicitly organizes complex targets through decomposition principles and composition topologies. This structured construction better matches EvoBreak’s need to identify uncovered requirements and acquire complementary experiences across interactions, thereby providing more effective supervision for experience-conditioned planning.