TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill GenerationConference: The ACM International Conference on the Foundations of Software Engineering; July 12–16, 2027; Shenzhen, ChinaProceedings of the ACM International Conference on the Foundations of Software Engineering, July 12–16, 2027, Shenzhen, ChinaCCS: Security and privacy Software security engineeringCCS: Software and its engineering Software verification and validation

Automated skill generation seeks to transform task experience and external knowledge into reusable procedural assets that improve an agent without updating its model parameters. Execution guided methods evaluate candidate skills through downstream agent performance. Skill Set Optimization constructs transferable skills from trajectories associated with high environment rewards (Nottingham et al. 2024). EvoSkill analyzes failed executions and retains generated skill folders that improve performance on a held out validation set (Alzubi et al. 2026). These methods expose candidate skills to executable agent environments and use observed task outcomes to guide subsequent generation. Their selection objectives are defined by functional rewards or answer correctness. A candidate can therefore receive a positive refinement signal after producing the expected result even when its trajectory contains unauthorized file access, dangerous command execution, or unintended network communication. Their evidence coverage also follows fixed task partitions and task specific verifiers. Environment conditioned branches and security relevant behaviors outside the exercised scenarios provide no signal to the generation process.

Another direction evaluates multiple properties of the generated artifact before deployment. SkillNet constructs skills from trajectories, repositories, documents, and user instructions, and assesses their safety, completeness, executability, maintainability, and cost (Liang et al. 2026). Its safety assessment primarily applies predefined rubrics through an LLM evaluator, while sandbox execution establishes whether bundled code and tool invocations are executable. This design incorporates Safety Reliability into skill quality assessment and can identify risks visible in the generated instructions and executable components. Its safety evidence remains centered on properties observable from the artifact. State dependent tool arguments, action composition across multiple steps, and persistent effects created during agent execution remain outside the refinement signal.

TRUSS combines static and dynamic verification in a single loop. Static checks validate consistency among the declared task, required capabilities, executable components, data access, and expected side effects. Admitted candidates are then executed by a shadow agent in a CEE. Functional outcomes and safety properties are evaluated on the same trace, distinguishing successful execution from unsafe intermediate behavior. Observed information flows and side effects become refinement evidence. This joint feedback enables TRUSS to preserve verified task improvements while removing vulnerabilities that become visible only after the skill interacts with an execution environment.

Agent skills form a security critical extension layer because their natural language instructions and bundled resources influence agents with access to both system command execution and external web resources. The effective authority of a skill follows the permissions exposed by its host agent. In local coding agent deployments, this authority may include access to sensitive user data and execution under the local user account (Liu et al. 2026a). Large scale analysis has found that 26.1% of examined skills contain at least one vulnerability, with data exfiltration and privilege escalation among the most prevalent categories (Liu et al. 2026b). Behavioral studies show that benign requests can activate harmful instructions embedded in skills or their execution context. Skill Inject demonstrates severe skill file attacks across frontier agents, while SkillSafetyBench reproduces such failures in executable environments (Schmotz et al. 2026; Jin et al. 2026). SkillSpector identifies risks visible before installation, whereas runtime verification captures behaviors that emerge after invocation (Paz et al. 2026).

Agent providers recognize the same execution boundary. Anthropic warns that malicious skills may direct an agent to exfiltrate data or perform unintended actions (Anthropic 2025).OpenAI identifies sandboxing, approval policies, and network controls as central safeguards for coding agents that execute model generated actions (OpenAI 2026). These findings motivate Safety Reliability as a property of the complete execution trajectory. TRUSS incorporates this property directly into skill generation, so runtime violations guide revision before a candidate enters the final skill repository.

Let $c$ denote the source corpus from which a skill is generated, and $G$ denote the skill generator. A generated skill $Sk$ is a standardized artifact whose procedural content and executable resources can be loaded by a fixed agent $\mathcal{A}$, which interprets the artifact while interacting with its execution environment.

We study two generation regimes that differ in the information available to $G$. Under the task conditioned regime, $G$ receives $c$ together with a disclosed task specification $t$, yielding $Sk=G(c,t)$, while $p$ represents the distribution of held out instances associated with $t$. Under the task agnostic regime, $G$ receives $c$, yielding $Sk=G(c)$, while $p$ represents the distribution of downstream tasks that remains hidden during generation. In either regime, $x\sim p$ denotes an individual task instance used to evaluate the generated skill.

For an instance $x$, we mark the complete execution trajectory produced by $\mathcal{A}$ after loading $Sk$ as $\tau^{Sk}(x)$ (and $\tau^{\emptyset}(x)$ is the corresponding trajectory under the empty skills). We define the functional effectiveness of $Sk$ as its expected utility gain over the empty skill condition:

where $U(x,\tau)\in[0,1]$ is the externally evaluated task utility of trajectory $\tau$ on $x$.

For safety reliability, let $J$ denote the number of safety properties considered by the framework, with $j\in\{1,\ldots,J\}$ identifying one property. Let $Z_{j}(x,\tau)\in\{0,1\}$ be the violation indicator whose value equals one when trajectory $\tau$ violates property $j$ on instance $x$, with zero indicating compliance. The expected risk associated with property $j$ is

Let $\mathbb{S}_{G}$ denote the space of skill candidates that $G$ can produce from $c$ under the selected generation regime. The overall skill generation objective combines functional effectiveness with the mean safety risk across the $J$ properties

where $\lambda\geq 0$ controls the influence of safety risk on candidate selection, while division by $J$ keeps its contribution comparable when the property set changes in size. This objective characterizes generation quality through the task improvement induced by a skill together with the safety reliability observed throughout its execution.

Figure 1 presents TRUSS as an iterative collaboration among a Generator, a Checker, and a Refiner. The Generator constructs a candidate Skill from the available source context. The Checker evaluates the candidate along two evidence dimensions. Static inspection examines the Skill artifact before execution, while runtime inspection observes the behavior induced when an agent loads the admitted Skill. Each stage evaluates both functional obligations and applicable safety properties.

The Checker first derives the obligations implied by the declared purpose and implemented capabilities of the candidate. Static evidence resolves obligations that can be determined from the artifact and transfers behavior dependent obligations to supervised execution. A Model Executor then exercises the Skill through brokered tools. An Intermediate Breaker evaluates every requested action before an allowed action reaches the Disposable Sandbox. The resulting trace supports runtime function and safety checks.

All conclusions are assembled into a Function and Safety Record whose entries retain their supporting artifact spans or execution events. The Refiner uses unresolved obligations and violation evidence to revise the candidate. Each revision receives a new identity and reenters the complete checking procedure. A candidate becomes the final output when the record establishes the required function and resolves every applicable safety obligation under the evaluated scenarios.

The Generator constructs the first candidate Skill $Sk^{(0)}$ from the information available at generation time. The source corpus $c$ supplies the domain knowledge from which the Skill derives its procedure. Under task conditioned generation, the disclosed task specification $t$ additionally provides the intended behavior and its observable completion criteria. Under task agnostic generation, the Generator derives the expected usage context from $c$, while downstream tasks remain hidden until evaluation. These regimes determine the functional evidence available to the subsequent Checker.

The Generator produces a complete Skill package as the candidate unit. The package contains the primary instruction document together with the executable resources referenced by its procedure. This package boundary is necessary because the behavior induced by a Skill arises from the interaction between its natural language guidance and its machine executable content. A procedure may describe an appropriate operation while an associated script introduces a different capability. The Checker therefore receives the complete package produced during the same generation instance.

At the handoff to the Checker, TRUSS records the generation regime and source provenance, freezes the candidate package, and assigns the package a content digest. Static findings, runtime traces, and subsequent revisions are associated with this exact candidate identity. The common candidate interface also permits different foundation models and coding agents to instantiate the Generator while preserving the checking and refinement procedure. The resulting $Sk^{(0)}$ enters Static Check as a Raw Skill whose functional claims and security implications require evidence before certification.

The Refiner receives a candidate when its Function and Safety Record contains a failed obligation or an obligation whose required evidence cannot be established. The record serves as a structured revision specification. A static finding identifies the applicable criterion and the artifact span that supports the conclusion. A runtime finding identifies the triggering request, the broker decision, the resulting observation, and its causal position in the execution trace. A functional failure associates the expected task state with the state observed after execution. Every entry remains bound to the content digest of the candidate from which its evidence was obtained.

For each revision, the Refiner constructs an evidence context around the unresolved entries. The corresponding artifact content and execution events define the revision targets, while satisfied function and safety entries provide preservation constraints. This joint context allows one revision to account for the interaction between utility and safety. Removing an unsafe operation may invalidate a required task procedure, while restoring functionality may activate a capability governed by an additional property. The Function and Safety Record makes both consequences available during the same revision decision.

The Refiner edits the evidence bearing content and produces a new candidate $Sk^{(k+1)}$. The revised package receives a new content digest and reenters Static Check from the beginning. Static inspection recomputes the implemented capabilities and applicable property scope because a revision may change the behavioral surface of the Skill. Candidates admitted by the new Static Gate proceed through a fresh supervised execution, producing a trace associated exclusively with the revised digest.

The refinement cycle terminates when the Checker produces a complete record in which the required functional outcomes are established and every applicable safety obligation is resolved under the evaluated scenarios. TRUSS then emits the corresponding candidate as the certified Skill. Exhausting the revision budget produces an uncertified candidate together with the residual record, preserving the evidence responsible for the decision.

RQ1 evaluates whether the evidence collected by TRUSS can distinguish vulnerable Skills from their clean counterparts. We use all 84 matched pairs from SkillInject (Schmotz et al. 2026). Each pair contains one clean Skill and one counterpart carrying a controlled injection, which yields 168 artifacts. We compare the intermediate LLM checker, the Static Checker, and the complete TRUSS configuration that combines static inspection with runtime evidence.

Table 2 shows that a direct LLM judgment provides weak vulnerability detection. Its recall reaches only 19.05%, indicating that most injected Skills remain unidentified, while its precision of 44.64% also shows that its positive judgments contain substantial noise. Static inspection raises precision by 36.91 percentage points and recall by 75.00 percentage points, producing an F1 score of 87.36%.

The complete framework resolves the remaining errors and reaches 100.00% on all three metrics. Relative to the Static Checker, runtime evidence contributes an additional 18.45 percentage points in precision and 5.95 percentage points in recall. These gains show that execution evidence resolves cases for which artifact level inspection alone leaves either an incorrect warning or an undetected vulnerability.

RQ2 evaluates the repaired Skills on all 155 cases from SkillSafetyBench  (Jin et al. 2026). Every case is executed before and after repair under two Codex configurations whose target models are GPT 5.5 and GPT 5.4. The official attack verifier determines attack success, while the official task verifier determines task completion.

Let $A_{0}$ and $A_{1}$ denote attack success before and after repair, respectively, and let $T_{1}$ denote completion by the repaired Skill. Since the reported attack elimination values use all 155 cases as their denominator, we define $\mathrm{AER}=P(A_{0}=1\land A_{1}=0)$. Attack regression is $P(A_{0}=0\land A_{1}=1)$. Safe task completion is $\mathrm{STC}=P(A_{1}=0\land T_{1}=1)$.

As shown in Table 3, GPT 5.5 initially triggers 60 attacks, corresponding to an ASR of 38.71%. Repair eliminates 30 of these attacks and reduces ASR to 19.35%. GPT 5.4 initially triggers 72 attacks. Repair eliminates 26 attacks and reduces ASR from 46.45% to 29.68%. Neither configuration introduces an attack in a previously safe execution, yielding zero attack regression.

The functional results reveal a substantial model effect. With GPT 5.5, the repaired Skills complete 98 tasks, of which 82 are completed safely. The corresponding task completion and safe task completion rates are 63.23% and 52.90%. With GPT 5.4, 62 tasks are completed and only 37 satisfy the safety condition, producing rates of 40.00% and 23.87%. The 29.03 percentage point difference in safe task completion shows that the target model materially affects the utility retained after repair.

Task completion alone includes 16 unsafe completions under GPT 5.5 and 25 under GPT 5.4. Safe task completion therefore provides the primary joint measure of repair quality because it credits a completed task only when the repaired Skill also suppresses the attack.

RQ3 evaluates whether the complete framework can generate Skills that improve task performance while satisfying the required safety properties. We conduct the experiment on all 187 samples in SkillGenBench (Zhou et al. 2026) and compare three conditions under the same benchmark tasks and evaluation procedure. No Skills measures the performance of the underlying agent without additional Skill support. Intermediate LLM Generator evaluates the Skills produced by the generation model before the complete verification and refinement procedure is applied. The TRUSS condition evaluates the final Skills produced by the complete framework. Improvement is reported as the absolute percentage point difference from the No Skills condition.

Table 4 reports the results over all 187 benchmark samples. The agent without Skills completes 17.11% of the tasks, corresponding to 32 successful samples. Loading the Skills produced by the intermediate LLM generator increases Effectiveness to 29.95%, corresponding to 56 successful samples and an improvement of 12.83 percentage points. The complete TRUSS framework reaches 52.94%, corresponding to 99 successful samples and an improvement of 35.83 percentage points over No Skills. Consequently, the complete framework solves 67 more samples than the underlying agent and 43 more samples than the intermediate generator.

The Security rate follows the same progression. No Skills satisfies the safety evaluation on 50.80% of the samples, corresponding to 95 safe executions. The intermediate generator raises this rate to 75.40%, which represents 141 safe executions and a gain of 24.60 percentage points. TRUSS passes the safety evaluation on all 187 samples, reaching 100.00% Security. This result amounts to a gain of 49.20 percentage points over No Skills and 24.60 percentage points over the intermediate generator.

The intermediate generator establishes that automatically produced Skills can improve the underlying agent, as its Effectiveness exceeds No Skills by 12.83 percentage points. Its Security result also shows that generated guidance can steer the agent toward safer behavior, although 46 samples still fail the safety evaluation. These results characterize the capability of the generation model before the generated artifacts receive the complete framework treatment.

Applying TRUSS produces a further Effectiveness gain of 22.99 percentage points over the intermediate generator while converting the remaining 46 unsafe outcomes into safe executions. The simultaneous increase in both measurements shows that the framework improves functional outcomes while enforcing the safety obligations used in the evaluation. The comparison between the intermediate generator and TRUSS captures the combined contribution of framework level inspection, execution evidence, and refinement. Their individual contributions require the component ablation reported separately.