Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation2184Journal: TOGCCS: Computing methodologies Motion processingCCS: Computing methodologies AnimationCCS: Computing methodologies Neural networks

Xiaogang Peng Affiliation: Northeastern University , 360 Huntington Ave , Boston , MA , 02115 , USA email: peng.xiaog@northeastern.edu , Zeyu Han Affiliation: Northeastern University , USA email: han.zeyu@northeastern.edu , Zichong Meng Affiliation: Northeastern University , USA email: meng.zic@northeastern.edu , Yiming Xie Affiliation: Northeastern University , USA email: ymxyimingxie@gmail.com , Jihua Zhu Affiliation: Xi’an Jiaotong University , China email: zhujh@xjtu.edu.cn , Gang Hua Affiliation: Amazon , USA email: ganghua@gmail.com and Huaizu Jiang Affiliation: Northeastern University , USA email: h.jiang@northeastern.edu

Refer to caption — Figure 1. Our approach enables: (a) whole-body interaction of a variable number of rigid objects from text prompts; (b) interaction with articulated objects exhibiting diverse joint dynamics (e.g., laptop, drawers, and knobs); and (c) text-driven whole-body motion synthesis guided by sparse object waypoints or full object motion trajectories.

Abstract.

Daily activities require humans to coordinate whole-body motion with the motion of surrounding objects. Despite recent progress in human-object interaction (HOI) generation, most existing methods assume interactions with a single rigid object and do not extend well to scenarios involving a variable number of objects or articulated objects with diverse joint mechanisms. We propose surface keypoint trajectories as an object motion representation: for each rigid component, whether a standalone object or one part of an articulated assembly, we track a small set of non-collinear surface points over time. This representation handles multi-object coordination and diverse articulation mechanisms directly from point dynamics without requiring explicit joint-type specification. To model when and where each body region contacts each object, we introduce a spatio-temporal contact distance field that extends distance-based contact modeling to whole-body, multi-object, and articulated settings. We factorize HOI generation into three stages: generating object motions from text or waypoints, predicting the contact distance field, and synthesizing whole-body motion with contact-guided optimization. Experiments on ParaHome, HIMO, ARCTIC, and OMOMO demonstrate better or comparable performance to existing methods across single-object, multi-object, and articulated interaction settings.

Keywords:

Human-Object Interaction Synthesis, Motion Representation

1. Introduction

Everyday activities such as cooking, cleaning, and furniture assembly require humans to coordinate whole-body motion with the movement of surrounding objects. Synthesizing such human-object interactions (HOIs) is a fundamental challenge in computer graphics and vision, with broad applications in video games, virtual and augmented reality, embodied AI, and robotics. The goal is to generate natural and contextually appropriate motions for both the human body and the objects being manipulated.

Previous works have made notable progress in HOI generation conditioned on various input modalities, including textual descriptions (5; 33), sparse object waypoints/trajectories (19; 20), object geometry and target grasps (43; 50), past motion history (53), and videos (18). However, most existing methods face two key limitations when applied to more complex real-world interactions.

(i) Multi-object and articulated interaction generation. Most existing methods focus on interactions with a single rigid object. However, multi-object interactions introduce additional complexity, as the number and combination of objects involved varies across activities, requiring the generation model to handle flexible object configurations. Moreover, the dominant object representation, global $SE(3)$ transformations (20; 19; 5; 33), inherently assumes rigid-body motion. Although per-part $SE(3)$ transformations can describe articulated objects (36; 58), they require explicit joint-type specification, with separate handling needed for each mechanism (revolute, prismatic, screw). While recent benchmarks such as HIMO (28) and ParaHome (17) have begun to capture multi-object and articulated interactions, the generation methods trained on them still require pre-specifying the number of objects (28). Furthermore, existing methods either generate human and object motions jointly (5; 33), which struggles as the number of objects grows, or require object trajectories as input (20), limiting their applicability. As a result, generation methods that handle a variable number of objects and diverse articulation mechanisms, while also producing object motions directly from text, remain limited.

(ii) Contact modeling. Accurately capturing when and where each body part contacts each object is essential for physically plausible interactions, especially when multiple objects are involved. Yet existing contact representations are either binary labels indicating presence or absence of contact (5; 33), which lack fine-grained spatial information; static affordances that do not evolve over time (22; 4); or hand-centric correspondences that do not extend to whole-body interactions (19; 36). ManipNet (57) introduces distance-based spatial sensors between hands and objects, but these are local to the hand and do not generalize to whole-body, multi-object settings. ROG (55) constructs an Interactive Distance Field for HOI generation, but handles only single-object interactions, measures proximity from sparse interior skeletal joints rather than the body surface, and uses a limited number of object keypoints. None of these approaches provides fine-grained, temporally dynamic correspondence between whole-body markers and multiple object surfaces, the kind of signal needed to guide generation when different body parts engage with different objects at different times.

To address limitation (i), we propose surface keypoint trajectories as an object motion representation. For each rigid component, whether a standalone object or one part of an articulated assembly, we sample a small set of non-collinear surface points from its mesh and track their 3D positions over time. A key property is that three non-collinear points suffice to uniquely determine a rigid transformation via the Kabsch algorithm (14), so the representation is lossless for rigid motion while operating entirely in Euclidean space. Different articulation mechanisms (revolute, prismatic, screw, as illustrated in Fig. 2) each produce distinctive point trajectory patterns that a generative model can learn directly from data, without requiring explicit joint-type specification. To handle a variable number of rigid components across different interactions, each component occupies a fixed-size slot that is zero-padded when unused and masked during training, allowing a single model to handle one object, multiple objects, or multi-part articulated assemblies without architectural changes.

To address limitation (ii), following prior work (50; 58), we represent the human body using surface markers on SMPL-X (31). This allows us to introduce a spatio-temporal contact distance field that captures, for every time step, the proximity between each body marker and each object surface point (1 indicates contact and 0 not). Unlike the binary, static, or hand-centric contact representations discussed above, our field provides whole-body coverage across multiple objects and their articulated parts. While ROG (55) introduces a distance field for single-object interactions, our formulation extends this concept to multi-object and articulated settings, computes distances from body surface markers rather than interior skeletal joints where contact physically occurs, and uses finer sampling on both the human and object sides. By providing fine-grained marker-to-surface correspondences at every time step, the field specifies which body part should contact which object region and when, enabling precise contact-guided optimization during body motion synthesis.

The contact distance field serves as the key intermediate representation in our factorized generation pipeline: given a text prompt, we first generate object keypoint trajectories (Stage I), then predict the contact distance field conditioned on the generated object motions (Stage II), and finally synthesize whole-body motion with contact-guided optimization using the predicted field (Stage III). This factorization decouples object dynamics, contact prediction, and body synthesis into well-defined sub-problems, allowing each stage to focus on a specific aspect of the interaction while remaining tractable for multi-object and articulated scenarios. We evaluate our method on four benchmarks that collectively span single-object, multi-object, and articulated interaction settings: ParaHome (17), ARCTIC (6), HIMO (28), and OMOMO (20). Our approach achieves state-of-the-art performance, consistently outperforming existing methods in both motion quality and interaction accuracy.

To summarize, our contributions are as follows.

•

We propose surface keypoint trajectories as an object motion representation that enables a single generative model to handle a variable number of rigid and articulated objects without explicit kinematic specification. We show this representation outperforms $SE(3)$ -based alternatives and naturally supports multi-part composition.
•

We introduce a spatio-temporal contact distance field that extends distance-based contact modeling to whole-body, multi-object, and articulated settings, capturing fine-grained, temporally evolving body-object correspondences. Predicted as a standalone intermediate representation, it provides richer supervision than binary contact labels and outperforms alternative contact formulations.
•

Experiments on ParaHome, OMOMO, ARCTIC, and HIMO demonstrate state-of-the-art performance in both motion quality and interaction accuracy, across single-object, multi-object, and articulated interaction settings.

2. Related Works

Representations for Human and Object Motion in HOI Generation. Most existing HOI generation methods represent the interaction state by combining a skeleton-based human representation (e.g., joint positions (8; 29), joint rotations (51; 20)) with an object state parameterized by global $SE(3)$ transformations (e.g., continuous 6D rotations (61) plus 3D translations (28; 20; 19; 5; 7; 53)). This mixed representation introduces a geometric mismatch: human motion resides in Euclidean space while object pose lies on the non-Euclidean Lie group $SE(3)$ , making it difficult for generative models to learn spatial correlations such as contact and coordinated motion. Furthermore, $SE(3)$ -based object parameterizations assume rigid-body motion and do not naturally extend to articulated objects. Recent efforts have incorporated articulated objects by adding a 1D joint angle to the 6-DoF global pose (36; 58), but this requires explicit joint-type specification and does not extend to objects with multiple articulated parts or diverse joint mechanisms (e.g., prismatic, screw). Uni-Inter (23) addresses the representation mismatch by encoding humans, objects, and scenes into a shared voxel-based occupancy volume; however, it requires object trajectories as input and relies on discrete voxelization. In contrast, we represent objects using surface keypoint trajectories that handle multi-object and articulated interactions without explicit joint-type specification, and adopt surface markers on SMPL-X (31) for the human body following (50; 58), placing both in Euclidean space.

Human-Object Interaction Generation. Recent works incorporate scene geometry or object motion priors to guide human motion synthesis, improving the modeling of human-scene and human-object interactions (11; 60; 47; 49; 59; 13; 27). Meanwhile, there has been growing interest in text-driven HOI generation, where many approaches jointly synthesize human motion and dynamic object motion (5; 48; 19; 41; 54; 39; 33; 52). However, most existing methods focus on interactions with a single object in isolation. HIMO (28) is among the first to explore text-driven multi-object HOI generation, but requires the number of interacting objects to be pre-specified, limiting flexibility. For articulated objects, existing approaches either restrict dynamics to revolute articulation (36; 58), covering only a narrow subset of everyday mechanisms, or focus on bimanual hand manipulation without synthesizing full-body motion (1; 58; 10). Our method handles a variable number of rigid and articulated objects within a single model, enabling whole-body HOI generation across diverse interaction settings.

Articulated and Multi-Object Interactions Generation. HIMO (28) is among the first to explore text-driven multi-object HOI generation; however, it requires the number of interacting objects to be pre-specified, which limits modeling flexibility. For complex articulated objects, existing approaches either restrict object dynamics to revolute (hinge) articulation (36; 58), covering only a narrow subset of everyday mechanisms, or focus on bimanual hand manipulation without synthesizing full-body motion (1; 58; 10). In contrast, our method enables whole-body HOI generation with diverse object dynamics in richer multi-object settings, yielding more realistic and flexible human–object interaction synthesis.

\begin{overpic}[figures/framework_v4.1.pdf] \put(144.0,8.0){\small{$f^{o}$}} \put(248.0,18.0){\small{$f^{d}$}} \put(350.0,28.0){\small{$f^{h}$}} \put(60.0,-10.0){\small{Stage1: Object Motion Generation} \hskip 9.24994pt{Stage2: Contact Distance Field Prediction} \hskip 9.24994pt{Stage3: Body Motion Synthesis}} \end{overpic}

Figure 3. Overview of the proposed framework. Our pipeline consists of three stages. Stage 1 (Object Motion Generation) employs an object-motion diffusion model

f^{o}

to generate object surface keypoint trajectories conditioned on the initial HOI state, the text prompt, and object geometry encoded with Basis Point Sets (BPS). Stage 2 (Contact Distance Field Prediction) uses a diffusion model

f^{c}

to predict a human–object contact distance field based on the generated object motion, the text prompt, and the object geometry. Stage 3 (Body Motion Synthesis) applies a marker-motion diffusion model

f^{h}

to synthesize the final human motion conditioned on the text prompt, the generated object motion, and the predicted contact distance field. We further perform contact optimization to alleviate contact and penetration artifacts, and finally recover the human and object meshes.

Contact Modeling for HOI Generation. Modeling contact between the human body and objects is essential for physically plausible interaction synthesis. Early approaches predict binary contact maps or contact likelihoods over the human body (35; 19; 5), object surfaces (1; 12; 2; 58), or both (45; 56), but such binary signals lack the spatial precision to recover which body part touches which object region. Several works explore richer contact representations: some model static paired contact patterns (22; 4), while others capture dynamic correspondences only for a limited set of body parts (e.g., bimanual hand manipulation) (19; 39; 36). ManipNet (57) introduces distance-based spatial sensors between hands and nearby object surfaces, but these are local to the hand and restricted to single-object settings. ROG (55) constructs an Interactive Distance Field between 24 skeletal keypoints and object surface keypoints with spatial and temporal attention, representing the closest prior work to ours. However, ROG handles only single-object interactions, and computing distances from interior skeletal joints rather than the body surface limits contact precision. In contrast, our spatio-temporal contact distance field extends distance-based contact modeling to whole-body, multi-object, and articulated settings, computing distances from 138 body surface markers to 384 surface points per rigid component.

3. Methodology

3.1. Surface Keypoint Representation

We represent object motion using surface keypoint trajectories and human motion using SMPL-X surface markers, placing both in Euclidean space so that spatial relationships such as contact and proximity reduce to point-to-point distance computations.

Object Representation. Most existing methods represent object states using global $SE(3)$ transformations, which assume rigid-body motion and require explicit joint-type specification when extended to articulated objects (36; 58). Instead, we model each scene as a collection of rigid components, where a rigid object constitutes a single component and an articulated object consists of multiple components (e.g., a drawer unit has a fixed body and several movable drawers, each as a separate component). For each rigid component, we sample $K$ non-collinear surface keypoints from its canonical mesh using farthest point sampling (38), where $K\geq 3$ is the minimum required to uniquely determine a rigid transformation. Because each component is rigid, $K$ non-collinear points fully determine its pose, making the representation lossless for rigid motion while operating entirely in Euclidean space and avoiding the manifold constraints of $SE(3)$ parameterizations. At each time step $t$ , the object state is represented by the global 3D positions of these keypoints. A key property of this representation is that different articulation mechanisms (revolute, prismatic, and screw joints) each produce distinctive keypoint trajectory patterns (as illustrated in Fig. 2), which a generative model can learn directly from data without requiring explicit joint-type specification.

In addition, existing multi-object methods require pre-specifying the number of objects at training time (28). To handle a variable number of rigid components, we define a maximum capacity $N$ and represent the full object state at time $t$ as $\mathbf{O}_{t}\in\mathbb{R}^{N\times 3K}$ . When fewer than $N$ components are present, unused slots are padded with zeros and masked out during training and inference. The rigid transformation of each component can be recovered in closed form: given the canonical keypoints $\mathbf{O}_{0}^{i}\in\mathbb{R}^{3K}$ and predicted keypoints $\mathbf{O}_{t}^{i}\in\mathbb{R}^{3K}$ for the $i$ -th rigid component, we use the Kabsch algorithm (14) to recover its rotation and translation as

(1)

\mathbf{R}_{t}^{i},\mathbf{t}_{t}^{i}=\argmin_{\mathbf{R},\mathbf{t}}\|\mathbf{R}\mathbf{O}_{0}^{i}+\mathbf{t}-\mathbf{O}_{t}^{i}\|^{2}.

Human Representation. Following prior work (50; 58), we represent the human body using $M$ surface markers sampled on the SMPL-X body mesh (31), including dense markers on the palms for fine-grained hand contact. At each time step $t$ , the human state is represented by the global 3D positions of these markers, denoted as $\mathbf{H}_{t}\in\mathbb{R}^{M\times 3}$ . We use $M=138$ markers in our experiments. For rendering and evaluation, we train a lightweight fitting model that maps predicted marker positions to SMPL-X parameters, enabling recovery of the full human mesh. Please refer to the Appendix for more details.

We note that while both humans and objects are represented as surface points, the object keypoint representation exploits a property specific to rigid bodies: a small number of non-collinear points fully determine pose, enabling lossless, joint-type-agnostic motion modeling that has no analog in the deformable human setting. These design choices are validated empirically in Sec. 4.3.

3.2. Factorized HOI Generation

Jointly modeling object motion, contact dynamics, and full-body kinematics is highly complex, making it difficult for a single end-to-end model to learn all components well. We therefore decompose the problem into three sequential stages: object motion generation, contact distance field prediction, and human body motion synthesis. Fig. 3 illustrates our pipeline. Specifically, we formulate each stage as a conditional diffusion model based on the Transformer architecture (46). We adopt Flow Matching (21; 25) for its efficacy and training simplicity, minimizing the mean squared error (MSE) between predicted and ground-truth velocities in a latent space. For brevity, we will explain how the diffusion models work in the Euclidean space in the rest of this section, which can be easily extended to the latent space. More details are provided in the supplementary material.

Stage I: Object Motion Generation. Stage I predicts object keypoint trajectories $\hat{\mathbf{O}}_{1:T}=\{\hat{\mathbf{O}}_{t}\}_{t=1}^{T}\in\mathbb{R}^{T\times N\times 3K}$ for all $N$ rigid components across $T$ time steps¹¹ 1 Throughout this paper, we use $\hat{\cdot}$ (e.g., $\hat{\mathbf{O}}_{1:T}$ ) to denote quantities that are predicted by or depend on a neural network’s output.. The generation is conditioned on four inputs: (1) an instruction embedding $\mathbf{e}$ derived from either a text prompt or sparse waypoints; (2) per-object Basis Point Set (BPS) features $\mathbf{b}$ that encode each object’s geometry in its canonical pose (37); (3) initial human markers $\mathbf{H}_{\mathrm{init}}\in\mathbb{R}^{L_{\mathrm{init}}\times 3M}$ ; and (4) initial object keypoints $\mathbf{O}_{\mathrm{init}}\in\mathbb{R}^{L_{\mathrm{init}}\times N\times 3K}$ , where $L_{\mathrm{init}}$ denotes the number of initial frames (4 in our implementation) that specify the HOI’s starting location.

We denote the noisy object motion as $\bm{\epsilon}^{o}\in\mathbb{R}^{T\times N\times 3K}$ . We flatten the object and time dimensions into $TN$ tokens, enabling our Transformer diffusion model $f^{o}$ to handle a variable number of objects. The object trajectories are generated as

(2)

\hat{\mathbf{O}}_{1:T}=f^{o}(\mathbf{e},\,\mathbf{b},\,\mathbf{H}_{\mathrm{init}},\,\mathbf{O}_{\mathrm{init}},\,\bm{\epsilon}^{o}).

$\hat{\mathbf{O}}_{1:T}$ will serve as conditioning for both Stage II and Stage III.

Stage II: Contact Distance Field Prediction. Before generating human motion, Stage II predicts a spatio-temporal contact distance field that captures when and where body markers should contact each object, as shown in Figure 4. By decoupling contact prediction from motion synthesis, we allow the model to reason explicitly about body-object correspondences, which is crucial for multi-object and multi-part interactions where different body parts may engage with different objects or parts at different times.

		2 objects/components								3 objects/components
Dataset	Method	Motion					Interaction			Motion					Interaction
		FID $\downarrow$	$R_{prec}$ $\uparrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$	FID $\downarrow$	$R_{prec}$ $\uparrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$
ParaHome	Real motion (reference)	0.00	0.727	7.78	0.0039	0.15	-	-	-	0.00	0.679	7.47	0.0031	0.05	-	-	-
	HIMO-Gen (28)	14.83	0.580	8.23	0.3258	8.15	0.534	0.815	0.570	21.61	0.509	7.03	0.4437	7.13	0.634	0.809	0.857
	Ours	4.49	0.707	8.38	0.0035	0.72	0.669	0.896	0.536	6.09	0.598	7.57	0.0017	0.40	0.680	0.906	0.776
HIMO	Real motion (reference)	0.000	0.729	11.905	0.0007	0.08	-	-	-	0.267	0.713	9.755	0.0004	0.10	-	-	-
	HIMO-Gen (28)	8.019	0.570	10.387	0.0021	0.89	0.612	0.698	0.670	4.467	0.556	9.777	0.0038	0.43	0.656	0.712	0.563
	Ours	6.572	0.612	11.021	0.0011	0.17	0.731	0.912	0.519	2.273	0.628	9.812	0.0015	0.19	0.751	0.883	0.621

Table 1. Quantitative results on ParaHome and HIMO for multi-object and articulated HOI generation. We compare against HIMO-Gen (28). Best results are highlighted in bold. Note that lower penetration (Pene) does not always imply better contact, as it may also indicate no contact at all.

We define the contact distance field between a subset of human markers and dense object surface points. For the human, we select $M_{c}$ markers from the full set of $M$ markers, covering body regions that frequently participate in interactions (e.g., hands, torso), and denote them as $\mathbf{H}^{c}_{t}\in\mathbb{R}^{M_{c}\times 3}$ at time $t$ . For objects, we use $Q$ denser surface points per rigid component ( $Q\gg K$ ) to capture finer geometric detail, and denote them as $\mathbf{S}_{t}\in\mathbb{R}^{NQ\times 3}$ at time $t$ . The contact distance field $\mathbf{D}_{t}\in\mathbb{R}^{M_{c}\times NQ}$ encodes the proximity between each marker-surface pair (1 indicates contact and 0 not), computed as

(3)

\mathbf{D}_{t}=\sigma\!\left(\frac{\tau-\mathtt{dist}(\mathbf{H}^{c}_{t},\mathbf{S}_{t})}{\alpha}\right),

where $\mathtt{dist}(\cdot,\cdot)$ computes pairwise Euclidean distances, $\sigma(\cdot)$ is the sigmoid function, $\tau$ is a contact threshold, and $\alpha$ controls sharpness.

This formulation differs from prior contact representations in several important ways. Binary contact labels (5; 33) are extremely sparse, resulting in weak gradients that make it difficult for the model to learn fine-grained contact transitions. ManipNet (57) introduces distance-based spatial sensors between hands and objects, but these are local to the hand and restricted to single-object settings. ROG (55) proposes an Interactive Distance Field between skeleton joints and object boundary keypoints, but does not model fine-grained contact and is also limited to a single object. Signed distance fields (SDFs) encode inside/outside geometry, which is useful for penetration penalties but does not directly capture interaction-aware proximity between body and object surfaces over time. In contrast, our contact distance field is fully spatio-temporal, fine-grained, and captures whole-body correspondences across multiple objects. We validate this choice empirically in Sec. 4.3. Visual results are shown in Fig. 4.

Given initial noise $\bm{\epsilon}^{d}\in\mathbb{R}^{T\times M_{c}\times NQ}$ and the generated object trajectories $\hat{\mathbf{O}}_{1:T}$ from Stage I, our Contact Distance Field Diffusion model predicts

(4)

\hat{\mathbf{D}}_{1:T}=f^{d}(\mathbf{e},\,\mathbf{b},\,\hat{\mathbf{O}}_{1:T},\,\bm{\epsilon}^{d}).

Similar to Stage I, we flatten the marker and time dimensions into $TM_{c}$ tokens for the Transformer. The predicted $\hat{\mathbf{D}}_{1:T}$ serves as contact priors for optimization in Stage III.

Stage III: Human Motion Synthesis with Contact Optimization. The goal here is to generate the human motion sequence $\hat{\mathbf{H}}_{1:T}=\{\hat{\mathbf{H}}_{t}\}_{t=1}^{T}\in\mathbb{R}^{T\times M\times 3}$ conditioned on the predicted object trajectories $\hat{\mathbf{O}}_{1:T}$ from Stage I, and further refine the motion using the predicted contact distance field $\hat{\mathbf{D}}_{1:T}$ from Stage II. Given initial noise $\bm{\epsilon}^{h}\in\mathbb{R}^{T\times M\times 3}$ , our Body Motion Diffusion model generates

(5)

\hat{\mathbf{H}}_{1:T}=f^{h}(\mathbf{e},\,\mathbf{b},\,\mathbf{H}_{\mathrm{init}},\,\mathbf{O}_{\mathrm{init}},\,\hat{\mathbf{O}}_{1:T},\,\bm{\epsilon}^{h}).

While the diffusion model produces plausible motions, it may not precisely satisfy the predicted contacts, and directly optimizing $\mathbf{H}_{1:T}$ to minimize contact violations can push the result off the learned manifold, leading to unrealistic artifacts. To address this, we adopt Diffusion Noise Optimization (DNO) (15), a technique also employed in recent HOI methods (36; 39), which optimizes the initial noise $\bm{\epsilon}^{h}$ rather than the output motion. The key idea is to treat the diffusion model as a differentiable decoder: we define a loss measuring contact violations on the generated motion, backpropagate gradients through the entire denoising process, and iteratively update the noise to find a latent sample that decodes into motion with smaller contact errors. Because the output is always decoded through the diffusion model, it remains on the learned motion manifold, preserving realism while improving contact accuracy.

We optimize the noise $\bm{\epsilon}^{h}$ by minimizing

(6)

\mathcal{L}=\mathcal{L}_{\text{contact}}+\lambda_{\text{pen}}\mathcal{L}_{\text{pen}}

where $\mathcal{L}_{\text{contact}}$ encourages predicted contact pairs (identified by thresholding $\hat{\mathbf{D}}_{t}$ ) to be spatially close, and $\mathcal{L}_{\text{pen}}$ penalizes human markers that penetrate object interiors using the precomputed object SDF. Visual examples of the predicted contacts are shown in Fig. 4. The detailed loss formulations are provided in the Appendix. At each denoising step, we decode the current noise to obtain $\hat{\mathbf{H}}_{1:T}$ , compute the loss, and backpropagate through the diffusion model to update $\bm{\epsilon}^{h}$ . This optimization is applied at inference time to ensure contact accuracy. When faster inference is desired, the optimization can be reduced or skipped entirely, with a modest trade-off in contact quality.

4. Experiments

4.1. Setup

Datasets. We use the ParaHome (17) dataset to evaluate HOI generation with multiple or articulated objects, which contains 486 minutes of motion sequences spanning 22 object categories. We additionally evaluate on the HIMO (28) benchmark for multi-object interactions and the ARCTIC benchmark (6) for articulated-object interactions, following their respective task settings. We also train and evaluate our model on the single-object interaction dataset OMOMO (20), which includes human-object interaction motion for 15 objects, with a total duration of approximately 10 hours. We recover the hand motions for OMOMO following the method of (52).

Evaluation metrics. We evaluate different methods using the following metrics, considering both motion and interaction quality.

Method	FID $\downarrow$	$R_{prec}$ $\uparrow$	FS $\downarrow$	Diversity $\rightarrow$	IV $\downarrow$	ID $\downarrow$	CR $\uparrow$
Real motion (reference)	-	0.516	0.002	8.052	4.68	11.47	0.085
CoDA (36)	$\textbf{2.178}^{\pm.013}$	$0.479^{\pm.003}$	0.003	7.562	5.25	12.87	0.086
HIMO-Gen (28)	$3.064^{\pm.017}$	$0.388^{\pm.002}$	0.004	7.432	8.84	13.97	0.084
Ours	$2.331^{\pm.014}$	$\textbf{0.495}^{\pm.003}$	0.003	7.892	5.37	11.98	0.089

Table 2. Quantitative results on the ARCTIC benchmark dataset for articulated-object HOI generation. We compare our full method with our method’s variants and the baseline CoDA (28). The best results are highlighted in bold.

Motion Quality. We use Fréchet Inception Distance(FID) to quantify the distributional discrepancy between real and generated motions using a pretrained motion encoder. R-Precision (R_prec) measures the semantic alignment between generated motions and their corresponding text prompts. Diversity (Div) evaluates the range of variation across generated samples, reflecting motion richness. Additionally, we report the Foot Sliding Score (FS), computed as a weighted average of accumulated foot translation in the XY plane following prior work (16). The score is measured in centimeters (cm), with lower values indicating better generation quality. Finally, we report Jerk_obj to assess object motion quality, where lower jerk values indicate smoother and more natural movements.

Interaction Quality. We assess the quality of human-object interactions by evaluating Temporal Contact Accuracy ( $C_{acc}^{tem}$ ) and Body Contact Accuracy ( $C_{acc}^{body}$ ), computed against the ground-truth spatial-temporal contact labels. To evaluate physical plausibility, we compute the Penetration Score (Pene) using the signed distance field (SDF) of the object mesh, following the prior works (19). More details are provided in the supplementary material.

4.2. Comparisons with Existing Methods

Baselines. We compare against HOI-Diff (33), CHOIS (19), ROG (55), and HOIDiNi (39) for single-object interactions. For multi-object scenarios, we follow the official HIMO-Gen (28) implementation and train separate models for interactions with two and three objects, respectively, since it does not support a variable number of objects. For fair comparisons, we train all baseline methods using their original representations with the same length of initial HOI state and a generation length of $T{=}120$ .

Quantitative Results. Table 1 reports quantitative results on ParaHome and HIMO for multi-object and articulated interactions. On ParaHome, our full model achieves the best overall performance, obtaining state-of-the-art FID and R-Precision, indicating high motion quality and strong text alignment. Moreover, improved $C_{acc}^{tem}$ , $C_{acc}^{body}$ , and competitive Pene score demonstrate more physically plausible and coherent human-object interactions. On HIMO, our single model outperforms HIMO-Gen, which requires separate models for 2-object and 3-object scenarios. On ARCTIC (Table 2), we achieve competitive performance with CoDA (36), demonstrating that surface keypoint trajectories can handle articulated objects without explicit joint-type specification.

Method	Motion					Interaction
Method	FID $\downarrow$	$R_{prec}$ $\uparrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$
Real motion (reference)	0.00	0.566	8.28	0.0002	0.44	-	-	-
HOI-Diff (33)	11.94	0.318	6.13	0.2074	58.01	0.446	0.915	0.477
CHOIS (19)	9.11	0.461	7.39	0.0047	73.07	0.589	0.905	0.543
ROG (55)	3.27	0.483	7.87	0.0036	16.05	0.719	0.898	0.586
HOIDiNi (39)	4.87	0.512	7.95	0.0047	7.95	0.698	0.828	0.671
Ours	3.46	0.529	7.98	0.0004	2.95	0.886	0.896	0.613

Table 3. Quantitative results on the OMOMO dataset for single-object HOI generation. We compare our full method with our method’s variants and existing baselines. The best results are highlighted in bold.

We further evaluate on the single-object OMOMO benchmark, where our approach shows better or comparable performance with other models, as shown in Table 3.

Qualitative Results. We present qualitative comparisons with baseline methods in Figure 7. The baselines often produce inaccurate contacts and unstable object motions, especially for complex articulated objects.

Perceptual User Study. We conduct a perceptual user study comparing our method against baselines on both ParaHome and OMOMO. For each sample, participants evaluate the generated animations on two aspects: text alignment (how well the motion matches the text description) and interaction quality (how natural and plausible the human-object interaction appears). Details are in the supplementary material. As shown in Figure 6, our method is consistently preferred on both criteria across both datasets.

Object Motion Representation	Text	Waypoints
Object Motion Representation	Jerk_obj $\downarrow$	Jerk_obj $\downarrow$	$T_{obj}$ $\downarrow$	$O_{obj}$ $\downarrow$
6D rot + trans (33)	4.51	4.02	10.13	1.11
9D rot matrix + trans (19)	7.23	4.58	9.78	1.05
Keypoints (Ours)	3.75	2.86	7.51	0.97

Table 4. Ablative results of different object motion representation on the OMOMO dataset. Best results are highlighted in bold.

4.3. Ablation Studies

We ablate the two representation choices central to our contributions: the surface keypoint representation for objects and the contact distance field formulation. We also provide ablation studies about our factorized pipeline in the supplementary material.

Object Motion Representation. Table 4 compares different object motion representations on OMOMO under text-only and waypoint conditioning (33; 20), while keeping the Stage I architecture and training settings identical. Under both conditions, keypoint trajectories consistently outperform 6D rot+trans (33) and 9D rotation-matrix+trans (19), achieving the lowest jerk, translation error, and orientation error. As shown in Figure 6, the advantage is particularly evident for articulated objects.

Design Choice	ParaHome (2 obj.)		ParaHome (3 obj.)		OMOMO
Design Choice	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}\uparrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}\uparrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}\uparrow$
Euclidean Distance (55)	0.612	0.857	0.642	0.854	0.848	0.858
Binary Label	0.565	0.802	0.594	0.806	0.795	0.804
Contact Pairs (39)	0.589	0.821	0.612	0.813	0.808	0.815
Ours	0.653	0.902	0.683	0.901	0.893	0.902

Table 5. Ablation results on the impact of different contact representations on ParaHome (2–3 objects) and OMOMO. Best results are highlighted in bold.

Effect of Contact Representations. As shown in Table 5, the proposed distance field achieves the best results across all metrics on ParaHome and OMOMO, confirming that it provides a more informative and learnable interaction cue than the alternatives. Binary labels suffer from extreme sparsity, resulting in weak gradients that make it difficult for the model to learn fine-grained contact transitions. Raw Euclidean distances, are absolute and unnormalized, increasing learning complexity and sensitivity to variations in object size. Contact Pairs (39) encode hand-object contact through a fixed number of selected contact locations on the object surface, resulting in relatively sparse contact cues.

4.4. Limitations and Discussions

The noise optimization in Stage III adds computational overhead (approximately 4 minutes per 124-frame sequence), which limits real-time deployment; developing feed-forward alternatives for contact refinement is a promising direction. Physical artifacts such as foot floating and minor penetrations remain present in our results, as they do across existing HOI generation methods, given the fundamental challenges of the HOI generation task; integrating physics-based constraints or learned collision handling could help mitigate these in future work.

5. Conclusion

We presented surface keypoint trajectories as an object motion representation for human-object interaction generation that handles a variable number of rigid and articulated objects without explicit joint-type specification. We also introduced a spatio-temporal contact distance field that extends distance-based contact modeling to whole-body, multi-object, and articulated settings, predicted as a standalone intermediate representation in a factorized three-stage pipeline. Experiments on ParaHome, HIMO, ARCTIC, and OMOMO demonstrate better or comparable performance to existing methods across single-object, multi-object, and articulated interaction settings.

References

Cha et al. (2024) J. Cha, J. Kim, J. S. Yoon, and S. Baek Text2hoi: text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1577–1585. Cited by: §2, §2, §2.
Chu et al. (2025) H. Chu, X. Deng, Q. Lv, X. Chen, Y. Li, J. Hao, and L. Nie 3d-affordancellm: harnessing large language models for open-vocabulary affordance detection in 3d worlds. arXiv preprint arXiv:2502.20041. Cited by: §2.
Chung et al. (2014) J. Chung, C. Gulcehre, K. Cho, and Y. Bengio Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. Cited by: Appendix B.
Cseke et al. (2025) A. Cseke, S. Tripathi, S. K. Dwivedi, A. S. Lakshmipathy, A. Chatterjee, M. J. Black, and D. Tzionas PICO: reconstructing 3d people in contact with objects. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1783–1794. Cited by: §1, §2.
Diller and Dai (2024) C. Diller and A. Dai CG-hoi: contact-guided 3d human-object interaction generation. Cited by: §1, §1, §1, §2, §2, §2, §3.2.
Fan et al. (2023) Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges ARCTIC: a dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §4.1.
Ghosh et al. (2023) A. Ghosh, R. Dabral, V. Golyanik, C. Theobalt, and P. Slusallek IMoS: intent-driven full-body motion synthesis for human-object interactions. In CGF, Cited by: §2.
Guo et al. (2022) C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng Generating diverse and natural 3d human motions from text. In CVPR, Cited by: Appendix A, Appendix B, §2.
Henry et al. (2020) A. Henry, P. R. Dachapally, S. Pawar, and Y. Chen Query-key normalization for transformers. arXiv preprint arXiv:2010.04245. Cited by: Appendix D.
Huang et al. (2025) M. Huang, F. Chu, B. Tekin, K. J. Liang, H. Ma, W. Wang, X. Chen, P. Gleize, H. Xue, S. Lyu, et al. HOIGPT: learning long-sequence hand-object interaction with language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7136–7146. Cited by: §2, §2.
Huang et al. (2023) S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S. Zhu Diffusion-based generation, optimization, and planning in 3d scenes. In CVPR, Cited by: §2.
Jian et al. (2023) J. Jian, X. Liu, M. Li, R. Hu, and J. Liu Affordpose: a large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14713–14724. Cited by: §2.
Jiang et al. (2024) N. Jiang, Z. He, Z. Wang, H. Li, Y. Chen, S. Huang, and Y. Zhu Autonomous character-scene interaction synthesis from text instruction. In SIGGRAPH Asia 2024 Conference Papers, pp. 1–11. Cited by: §2.
Kabsch (1976) W. Kabsch A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A 32 (5), pp. 922–923. External Links: Document, Link Cited by: §1, §3.1.
Karunratanakul et al. (2024) K. Karunratanakul, K. Preechakul, E. Aksan, T. Beeler, S. Suwajanakorn, and S. Tang Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1334–1345. Cited by: §3.2.
Karunratanakul et al. (2023) K. Karunratanakul, K. Preechakul, S. Suwajanakorn, and S. Tang Guided motion diffusion for controllable human motion synthesis. In ICCV, Cited by: §4.1.
Kim et al. (2025) J. Kim, J. Kim, J. Na, and H. Joo ParaHome: parameterizing everyday home activities towards 3d generative modeling of human-object interactions. In CVPR, Cited by: §1, §1, §4.1.
Li et al. (2026) H. Li, H. Yu, J. Li, and J. Wu ZeroHSI: zero-shot 4d human-scene interaction by video generation. In International Conference on 3D Vision (3DV), Cited by: §1.
Li et al. (2023a) J. Li, A. Clegg, R. Mottaghi, J. Wu, X. Puig, and C. K. Liu Controllable human-object interaction synthesis. External Links: 2312.03913 Cited by: Appendix A, Appendix B, §1, §1, §1, §2, §2, §2, §4.1, §4.2, §4.3, Table 3, Table 4.
Li et al. (2023b) J. Li, J. Wu, and C. K. Liu Object motion guided human motion synthesis. TOG. Cited by: Table 2, Appendix E, §1, §1, §1, §2, §4.1, §4.3.
Lipman et al. (2022) Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §3.2.
Liu et al. (2023) S. Liu, Y. Zhou, J. Yang, S. Gupta, and S. Wang Contactgen: generative contact modeling for grasp generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20609–20620. Cited by: §1, §2.
Liu et al. (2025) S. Liu, Y. Liang, J. Wang, S. Du, C. Zhang, and X. Li Uni-inter: unifying 3d human motion synthesis across diverse interaction contexts. In SIGGRAPH Asia, Cited by: §2.
Liu et al. (2022a) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003. Cited by: Appendix C.
Liu et al. (2022b) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003. Cited by: §3.2.
Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In ICLR, Cited by: Appendix D.
Luo et al. (2024) Z. Luo, J. Cao, S. Christen, A. Winkler, K. Kitani, and W. Xu Grasping diverse objects with simulated humanoids. arXiv preprint arXiv:2407.11385. Cited by: §2.
Lv et al. (2024) X. Lv, L. Xu, Y. Yan, X. Jin, C. Xu, S. Wu, Y. Liu, L. Li, M. Bi, W. Zeng, et al. HIMO: a new benchmark for full-body human interacting with multiple objects. In European Conference on Computer Vision, pp. 300–318. Cited by: Appendix A, §1, §1, §2, §2, §2, §3.1, Table 1, Table 1, Table 1, Table 1, §4.1, §4.2, Table 2, Table 2, Table 2.
Meng et al. (2024) Z. Meng, Y. Xie, X. Peng, Z. Han, and H. Jiang Rethinking diffusion for text-driven human motion generation. arXiv preprint arXiv:2411.16575. Cited by: §2.
Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: an imperative style, high-performance deep learning library. NeurIPS. Cited by: Appendix D.
Pavlakos et al. (2019) G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985. Cited by: §1, §2, §3.1.
Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195–4205. Cited by: Appendix D.
Peng et al. (2023) X. Peng, Y. Xie, Z. Wu, V. Jampani, D. Sun, and H. Jiang Hoi-diff: text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553. Cited by: Appendix A, §1, §1, §1, §2, §3.2, §4.2, §4.3, Table 3, Table 4.
Pennington et al. (2014) J. Pennington, R. Socher, and C. Manning GloVe: global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans (Eds.), Doha, Qatar, pp. 1532–1543. External Links: Link, Document Cited by: Appendix B.
Petrov et al. (2024) I. A. Petrov, R. Marin, J. Chibane, and G. Pons-Moll TriDi: trilateral diffusion of 3d humans, objects, and interactions. arXiv preprint arXiv:2412.06334. Cited by: §2.
Pi et al. (2025) H. Pi, Z. Cen, Z. Dou, and T. Komura CoDA: coordinated diffusion noise optimization for whole-body manipulation of articulated objects. Advances in Neural Information Processing Systems. Cited by: Appendix E, §1, §1, §2, §2, §2, §2, §3.1, §3.2, §4.2, Table 2.
Prokudin et al. (2019) S. Prokudin, C. Lassner, and J. Romero Efficient learning on point clouds with basis point sets. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4332–4341. Cited by: §3.2.
Qi et al. (2017) C. R. Qi, L. Yi, H. Su, and L. J. Guibas Pointnet++: deep hierarchical feature learning on point sets in a metric space. NeurIPS. Cited by: §3.1.
Ron et al. (2025) R. Ron, G. Tevet, H. Sawdayee, and A. H. Bermano HOIDiNi: human-object interaction through diffusion noise optimization. arXiv preprint arXiv:2506.15625. Cited by: Appendix A, §2, §2, §3.2, §4.2, §4.3, Table 3, Table 5.
Shazeer (2020) N. Shazeer Glu variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: Appendix D.
Song et al. (2024) W. Song, X. Zhang, S. Li, Y. Gao, A. Hao, X. Hou, C. Chen, N. Li, and H. Qin HOIAnimator: generating text-prompt human-object animations using novel perceptive diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 811–820. Cited by: §2.
Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, pp. 127063. Cited by: Appendix D.
Taheri et al. (2022) O. Taheri, V. Choutas, M. J. Black, and D. Tzionas GOAL: Generating 4D whole-body motion for hand-object grasping. In CVPR, Cited by: §1.
Tevet et al. (2023) G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-or, and A. H. Bermano Human motion diffusion model. In ICLR, Cited by: Appendix B.
Tripathi et al. (2023) S. Tripathi, A. Chatterjee, J. Passy, H. Yi, D. Tzionas, and M. J. Black Deco: dense estimation of 3d human-scene contact in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8001–8013. Cited by: §2.
Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. NeurIPS. Cited by: §3.2.
Wang et al. (2022a) J. Wang, Y. Rong, J. Liu, S. Yan, D. Lin, and B. Dai Towards diverse and natural scene-aware 3d human motion synthesis. In CVPR, Cited by: §2.
Wang et al. (2023) Y. Wang, J. Lin, A. Zeng, Z. Luo, J. Zhang, and L. Zhang PhysHOI: physics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393. Cited by: §2.
Wang et al. (2022b) Z. Wang, Y. Chen, T. Liu, Y. Zhu, W. Liang, and S. Huang Humanise: language-conditioned human motion generation in 3d scenes. NeurIPS. Cited by: §2.
Wu et al. (2022) Y. Wu, J. Wang, Y. Zhang, S. Zhang, O. Hilliges, F. Yu, and S. Tang SAGA: stochastic whole-body grasping with contact. In ECCV, Cited by: §1, §1, §2, §3.1.
Xiao et al. (2025) L. Xiao, S. Lu, H. Pi, K. Fan, L. Pan, Y. Zhou, Z. Feng, X. Zhou, S. Peng, and J. Wang MotionStreamer: streaming motion generation via diffusion-based autoregressive model in causal latent space. arXiv preprint arXiv:2503.15451. Cited by: §2.
Xu et al. (2025) S. Xu, D. Li, Y. Zhang, X. Xu, Q. Long, Z. Wang, Y. Lu, S. Dong, H. Jiang, A. Gupta, Y. Wang, and L. Gui InterAct: advancing large-scale versatile 3d human-object interaction generation. In CVPR, Cited by: §2, §4.1.
Xu et al. (2023) S. Xu, Z. Li, Y. Wang, and L. Gui InterDiff: generating 3d human-object interactions with physics-informed diffusion. In ICCV, Cited by: §1, §2.
Xu et al. (2024) S. Xu, Z. Wang, Y. Wang, and L. Gui InterDreamer: zero-shot text to 3d dynamic human-object interaction. arXiv preprint arXiv:2403.19652. Cited by: §2.
Xue et al. (2025) M. Xue, Y. Liu, L. Guo, S. Huang, and C. Ding Guiding human-object interactions with rich geometry and relations. arXiv preprint arXiv:2503.20172. Cited by: Appendix A, Appendix E, §1, §1, §2, §3.2, §4.2, Table 3, Table 5.
Yang et al. (2024) Y. Yang, W. Zhai, H. Luo, Y. Cao, and Z. Zha Lemon: learning 3d human-object interaction relation from 2d images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16284–16295. Cited by: §2.
Zhang et al. (2021) H. Zhang, Y. Ye, T. Shiratori, and T. Komura ManipNet: neural manipulation synthesis with a hand-object spatial representation. ACM ToG 40 (4). Cited by: §1, §2, §3.2.
Zhang et al. (2025) W. Zhang, R. Dabral, V. Golyanik, V. Choutas, E. Alvarado, T. Beeler, M. Habermann, and C. Theobalt BimArt: a unified approach for the synthesis of 3d bimanual interaction with articulated objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Cited by: §1, §1, §2, §2, §2, §2, §3.1, §3.1.
Zhang et al. (2024) X. Zhang, S. Starke, V. Guzov, Z. Zhang, E. P. Pellitero, and G. Pons-Moll SCENIC: scene-aware semantic navigation with instruction-guided control. arXiv preprint arXiv:2412.15664. Cited by: §2.
Zhao et al. (2023) K. Zhao, Y. Zhang, S. Wang, T. Beeler, and S. Tang Synthesizing diverse human motions in 3d indoor scenes. arXiv. Cited by: §2.
Zhou et al. (2019) Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5745–5753. Cited by: §2.

\begin{overpic}[figures/user_study.pdf] \put(58.0,5.0){\small{Prefer Ours}} \put(125.0,5.0){\small{Prefer Others}} \put(200.0,5.0){\small{Cannot tell}} \put(50.0,118.0){Interaction with Multiple or Articulated Objects} \put(70.0,60.0){Interaction with a Single Rigid Object} \end{overpic}

Figure 5. User perceptual study results. The percentage of times our approach is preferred over (a) Text Alignment, (b) Interaction Quality.

\begin{overpic}[figures/ablation_obj_rep.pdf] \put(2.0,110.0){\small{9D rot matrix + trans}} \put(82.0,110.0){\small{6D rot + trans}} \put(155.0,110.0){\small{Keypoints}} \put(62.0,8.0){\small{Reference Axis}} \put(155.0,8.0){\small{Generated Axis}} \end{overpic}

Figure 6. Point-based Representation vs. Object Rotation + Translation. Our surface keypoint representation produces more realistic and accurate object motions, leading to more plausible object part motion in articulated scenarios.

\begin{overpic}[figures/comparative_results_v3_w_text.pdf] \par\end{overpic}

Figure 7. Qualitative comparisons on the test sets of two datasets (OMOMO: top; ParaHome: bottom). For clarity, we visualize only keyframes from top to bottom for each interaction sequence. Compared to baseline methods, our approach generates more realistic and coherent human–object interactions, featuring more accurate contact and fewer artifacts.

We present more information in the supplementary material, organized as follows:

•

A: Implementation Details of Baseline Methods.
•

B: Details of Metrics and Evaluation.
•

C: Flow Matching Preliminary.
•

D: Implementation Details of Our Method.
•

E: Additional Results.
•

F: Details of User Study.
•

G: Failure Cases.
•

H: Supplemental Video.

Appendix A Implementation Details of Baseline Methods

HIMO-Gen (28): HIMO is the most similar work to ours. We train separate models for interactions with two and three objects, respectively, since it does not support a variable number of objects. We retrain it using its original input representation, which includes full-body global joints and SMPL-X parameters.

HOI-Diff (33): We use the original human pose representation from HOI-Diff in the HumanML3D (8) format, which contains 263 dimensions without hand motion modeling. We retrain the model using this original representation and convert the generated outputs to global joint coordinates and then obtain the marker motion from the recovery SMPL-X mesh vertices under our proposed metrics.

CHOIS (19): CHOIS was originally conditioned on both text and waypoint trajectories. To enable a fair comparison under text-only settings, we remove the waypoint input and adapt the input dimensions accordingly.

ROG (55): ROG is a recent open-source baseline for single-object HOI generation. It represents object geometry using boundary-focused sparse keypoints and constructs an interactive distance field to model human-object relations. We retrain their model and evaluate it on the OMOMO benchmark.

HOIDiNi (39): HOIDiNi is a recent text-driven HOI generation method that optimizes the diffusion noise space to improve contact accuracy and motion plausibility. We retrain their model and evaluate it on the OMOMO benchmark.

Appendix B Details of Metrics and Evaluation

For detailed information regarding metrics employed in human motion generation, including FID, R-Precision, and Diversity, we refer readers to (44; 8) for a comprehensive understanding. We first build an evaluator by following the architecture of the widely used motion–text evaluator (8), which consists of a convolutional movement encoder, a GRU (3)-based motion encoder, and a GRU-based text encoder using GloVe (34) embeddings. The evaluator is trained on our selected marker-based motion representation, which captures the key dynamics that are most relevant to HOI generation.

Jerk_obj. It quantifies object motion smoothness, defined as the rate of change of acceleration. Given a sequence of object keypoints trajectory $\mathbf{O}_{1:T}$ with $T$ frames, we compute:

(7)

\mathrm{Jerk}=\frac{1}{T-3}\sum_{t=1}^{T-3}\left\|\mathbf{a}_{t+1}-\mathbf{a}_{t}\right\|_{2},

where $\mathbf{a}_{t}$ denotes the per-frame acceleration of the object keypoints obtained by applying finite differences to $\mathbf{O}_{1:T}$ , and $\|\cdot\|_{2}$ is taken over the concatenated keypoint coordinates (in centimeters). Lower jerk scores indicate smoother motion.

Temporal Contact Accuracy $C_{acc}^{tem}$ and Body Contact Accuracy $C_{acc}^{body}$ .

Contact definition.

For each sequence with $N_{\mathrm{obj}}$ active rigid components ( $N_{\mathrm{obj}}\leq N$ ), we evaluate contact per component and average over components. Let $\mathbf{H}_{t}\in\mathbb{R}^{M\times 3}$ be the full set of $M$ human markers and $\mathbf{S}^{o}_{t}\in\mathbb{R}^{Q\times 3}$ the surface points of component $o$ at time $t$ . For each component we compute the per-marker minimum distance to that component’s surface,

(8)

\mathbf{d}^{o}_{t}\;=\;\min_{i\in\{1,\ldots,Q\}}\mathtt{dist}\!\big(\mathbf{H}_{t},\,\mathbf{S}^{o}_{t}[i]\big)\;\in\;\mathbb{R}^{M},

and a binary contact label using a distance threshold $\tau=2\,\text{cm}$ ,

(9)

\mathbf{g}^{o}_{t}\;=\;\mathbb{I}\!\left[\mathbf{d}^{o}_{t}<\tau\right]\;\in\;\{0,1\}^{M}.

We apply the same procedure to the ground-truth and predicted trajectories to obtain $\mathbf{g}^{o,\text{gt}}_{t}$ and $\mathbf{g}^{o,\text{pred}}_{t}$ . Contact accuracy is then computed per component from these labels and averaged over the $N_{\mathrm{obj}}$ components (and over sequences). We report a body-level score over all $(t,m)$ pairs and a temporal (frame-level) score in which a frame is in contact if any marker is, i.e. $\bigvee_{m}\mathbf{g}^{o}_{t}[m]$ .

(A) Temporal Contact Accuracy.

We first define a frame-level contact indicator that is active if any marker is in contact:

(10)

G_{t}\;=\;\mathbb{I}\!\left[\mathbf{1}^{\top}\mathbf{g}_{t}>0\right].

Let $G_{t}^{\text{gt}}$ and $G_{t}^{\text{pred}}$ denote ground-truth and predicted frame-level contact labels. We compute the confusion counts:

(11)	$\displaystyle\mathrm{TP}$	$\displaystyle=\sum_{t=1}^{T}\mathbb{I}\!\left[G_{t}^{\text{gt}}=1\text{ }\&\text{ }G_{t}^{\text{pred}}=1\right],$
(12)	$\displaystyle\mathrm{FP}$	$\displaystyle=\sum_{t=1}^{T}\mathbb{I}\!\left[G_{t}^{\text{gt}}=0\text{ }\&\text{ }G_{t}^{\text{pred}}=1\right],$
(13)	$\displaystyle\mathrm{TN}$	$\displaystyle=\sum_{t=1}^{T}\mathbb{I}\!\left[G_{t}^{\text{gt}}=0\text{ }\&\text{ }G_{t}^{\text{pred}}=0\right],$
(14)	$\displaystyle\mathrm{FN}$	$\displaystyle=\sum_{t=1}^{T}\mathbb{I}\!\left[G_{t}^{\text{gt}}=1\text{ }\&\text{ }G_{t}^{\text{pred}}=0\right].$

Then the temporal contact accuracy is:

(15)

\displaystyle C_{acc}^{tem}

\displaystyle=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}.

(B) Body Contact Accuracy.

We treat each marker-time pair $(t,m)$ as an independent binary classification target. Let $g_{t,m}^{\text{gt}}$ and $g_{t,m}^{\text{pred}}$ denote ground-truth and predicted marker-level contact labels (i.e., the $m$ -th entries of $\mathbf{g}_{t}^{\text{gt}}$ and $\mathbf{g}_{t}^{\text{pred}}$ ). We compute the confusion counts over all $(t,m)$ :

(16)	$\displaystyle\mathrm{TP}_{m}$	$\displaystyle=\sum_{t=1}^{T}\sum_{m=1}^{M}\mathbb{I}\!\left[g_{t,m}^{\text{gt}}=1\text{ }\&\text{ }g_{t,m}^{\text{pred}}=1\right],$
(17)	$\displaystyle\mathrm{FP}_{m}$	$\displaystyle=\sum_{t=1}^{T}\sum_{m=1}^{M}\mathbb{I}\!\left[g_{t,m}^{\text{gt}}=0\text{ }\&\text{ }g_{t,m}^{\text{pred}}=1\right],$
(18)	$\displaystyle\mathrm{TN}_{m}$	$\displaystyle=\sum_{t=1}^{T}\sum_{m=1}^{M}\mathbb{I}\!\left[g_{t,m}^{\text{gt}}=0\text{ }\&\text{ }g_{t,m}^{\text{pred}}=0\right],$
(19)	$\displaystyle\mathrm{FN}_{m}$	$\displaystyle=\sum_{t=1}^{T}\sum_{m=1}^{M}\mathbb{I}\!\left[g_{t,m}^{\text{gt}}=1\text{ }\&\text{ }g_{t,m}^{\text{pred}}=0\right].$

The body contact accuracy is then defined as:

(20)

\displaystyle C_{acc}^{body}

\displaystyle=\frac{\mathrm{TP}_{m}+\mathrm{TN}_{m}}{\mathrm{TP}_{m}+\mathrm{FP}_{m}+\mathrm{TN}_{m}+\mathrm{FN}_{m}}.

Penetration Score. Following 19, we quantify body-object interpenetration by querying the precomputed signed distance field at each marker location. At time step $t$ , $\mathrm{SDF}_{t}(\cdot)$ returns the signed distance to the object surface, with negative values indicating penetration. When multiple objects are present, $\mathrm{SDF}_{t}(\cdot)$ takes the minimum signed distance over all active objects. We compute:

(21)

\text{Pene}\;=\;\frac{1}{TM}\sum_{t=1}^{T}\sum_{m=1}^{M}\max\bigl(-\mathrm{SDF}_{t}\!\left(\mathbf{H}_{t}(m)\right),\,0\bigr),

measured in centimeters.

Appendix C Flow Matching Preliminaries.

In each stage, we use a diffusion-based generative model, implemented via rectified flow (24) and ODE sampling. Given a condition $c$ (e.g., text and geometry features), we model the generative process as a continuous-time flow defined by an ODE:

(22)

\frac{d\mathbf{x}(t)}{dt}=\mathbf{v}_{\theta}(\mathbf{x}(t),t,c),\quad t\in[0,1],

where $\mathbf{v}_{\theta}$ is a neural velocity field.

Training objective. We construct an interpolation path between data $\mathbf{x}_{0}\sim p_{\text{data}}$ and noise $\mathbf{x}_{1}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ :

(23)

\mathbf{x}_{t}=(1-t)\mathbf{x}_{0}+t\mathbf{x}_{1}.

The corresponding target velocity is

(24)

\mathbf{v}^{\ast}(\mathbf{x}_{t},t)=\frac{d\mathbf{x}_{t}}{dt}=\mathbf{x}_{1}-\mathbf{x}_{0}.

We train $\mathbf{v}_{\theta}$ via flow matching:

(25)

\mathcal{L}_{\text{FM}}=\mathbb{E}_{\mathbf{x}_{0},\mathbf{x}_{1},t}\Big[\big\|\mathbf{v}_{\theta}(\mathbf{x}_{t},t,c)-\mathbf{v}^{\ast}(\mathbf{x}_{t},t)\big\|_{2}^{2}\Big].

Sampling. At inference time, we sample $\mathbf{x}(1)\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and solve the ODE in Eq. equation 22 backward from $t{=}1$ to $t{=}0$ using an ODE solver (e.g., Euler or Heun):

(26)

\mathbf{x}(t-\Delta t)=\mathbf{x}(t)-\Delta t\cdot\mathbf{v}_{\theta}(\mathbf{x}(t),t,c).

The final sample $\mathbf{x}(0)$ is the generated sequence.

Figure 1. Architecture of the Causal AutoEncoder.

Appendix D Implementation Details of Our Method.

Human Marker Representation. As illustrated in Figure 2, we select a total of 138 surface markers covering the full human body. We place denser markers on the hands, since fine-grained contacts often occur on the palms and fingers. Specifically, we use 38 markers on the main body, 40 markers on each hand, 6 markers on each foot, 5 markers on the head, and 3 markers on the hip/buttocks region.

Object Representation. For keypoint selection, we sample $K$ local points from the vertex set using Farthest Point Sampling (FPS). For the surface representation used in Stages II and III, we sample $Q=384$ surface points via Poisson-disk sampling to obtain approximately uniform coverage of the object surface, which also reduces the computational cost of distance-field computation.

Main Model Architecture Details. We use a CLIP text encoder to obtain prompt embeddings and an MLP to encode object geometry represented by BPS features or sparse object waypoints if available. Across all three stages, the input sequences, including object motion, contact distance fields, and human marker motion, are compressed into a 64-dimensional latent space using task-specific autoencoders with the same architecture as shown in the Fig. 1. The AutoEncoder is a 3-layer ResNet-based encoder-decoder architecture with a total downsampling rate of 4. For each diffusion backbone, we utilize an 8-layer AdaLN-Zero transformer (32) encoder with a hidden dimension of 1024 and 4 heads. We also adopt several recent architectural advances for better sequence modeling: Rotary Positional Embeddings (RoPE) (42) and QK Normalization (9) are employed within attention layers, and SwiGLU activations (40) are used in the feed-forward networks (FFNs).

Causal Autoencoder All three stages operate in a compact latent space produced by a causal temporal autoencoder (Fig. 1). Rather than one joint encoder, we use three structurally identical but independently trained branches—one each for object motion, the contact distance field, and human marker motion—so every modality has a dedicated codec while sharing the same design. Each branch maps a length- $T$ per-frame feature sequence to a latent of length $T/4$ with $d{=}64$ channels; the Stage I–III models predict flow-matching velocities directly on these latents.

Causal temporal convolutions.

Every convolution in the encoder and decoder is left-padded only, so the latent at temporal block $\tau$ summarizes exclusively input frames $t\leq 4\tau$ and never accesses the future. This makes the encoding of the clean initial state ( $\mathbf{H}_{\text{init}},\mathbf{O}_{\text{init}}$ ) and of the generated continuation consistent under the same operator, which is what our initial-state conditioning (Sec. 3.2) relies on.

Architecture.

The encoder lifts the input to a hidden width of $512$ with a $1$ D convolution ( $k{=}3$ ), then applies two downsampling stages; each stage is a strided causal convolution ( $k{=}4$ , stride $2$ ) that halves the temporal length, followed by three residual blocks. The two stages give the total temporal compression of $4\times$ . Each residual block is a pre-activation unit with SiLU (swish) gating, a dilated causal convolution ( $k{=}3$ ; dilations $9,3,1$ across the three blocks to enlarge the temporal receptive field), a $1{\times}1$ convolution, and dropout $0.2$ . A final convolution maps the hidden state to the $d{=}64$ latent. The decoder mirrors this: it expands the latent to width $512$ , applies two upsampling stages (nearest-neighbor $\times 2$ followed by a causal convolution), each preceded by three residual blocks, and two output convolutions map back to the input dimension.

Per-modality inputs and latents.

Human. The $M{=}138$ surface markers form a $3M{=}414$ -d per-frame vector, encoded into one latent stream $\mathbf{z}^{H}\!\in\!\mathbb{R}^{(T/4)\times d}$ . Object. Each rigid component is described by its $K{=}3$ keypoints ( $3K{=}9$ -d per frame) and is encoded independently by the shared object branch, so an interaction with $N$ components yields $N$ latent streams $\mathbf{z}^{O}\!\in\!\mathbb{R}^{N\times(T/4)\times d}$ ; weight sharing lets a single model handle a variable number of objects. Contact. The contact distance field $\mathbf{D}\!\in\!\mathbb{R}^{T\times M_{c}\times Q}$ (per component) is compressed per marker: for each of the $M_{c}{=}47$ selected markers, its length- $T$ , $Q{=}384$ -d proximity trajectory is encoded independently, giving $\mathbf{z}^{D}\!\in\!\mathbb{R}^{N\times M_{c}\times(T/4)\times d}$ . Encoding each marker independently preserves the sparse, high-frequency structure of the field (best reconstruction and contact recall) and keeps the codec agnostic to the number of markers and components.

Training objective.

Each branch is trained with a reconstruction loss over valid frames only (padded frames excluded). Human and object motion use the Smooth- $L_{1}$ loss, with the object loss additionally masked to the real components. Since $\mathbf{D}$ is dominated by a near-zero background, a uniform loss under-weights the rare in-contact entries; we therefore up-weight cells with $\mathbf{D}>0.5$ by $(1+\lambda_{c})$ , which substantially improves contact recall. The three branches have disjoint parameters and are optimized independently, so the contact codec can be trained separately and combined with the motion codec.

Latent standardization.

Before the Stage II contact diffusion, we standardize the contact latent to zero mean and unit variance using statistics precomputed on the training set, and invert this after sampling; this stabilizes flow matching on the contact latent, whose raw scale differs from the motion latents.

Optimization.

Each autoencoder is trained on ParaHome and OMOMO with $64$ -frame clips, batch size $128$ , and AdamW ( $\beta_{1}{=}0.9,\beta_{2}{=}0.99$ ) at learning rate $2\times 10^{-4}$ .

Conditioning. We condition the model on the initial human state and initial object state by temporally concatenating them with the input human sequence and object sequence, respectively. The BPS embedding is repeated across $T+L_{init}$ time steps and fused with the object motion via feature-wise concatenation. When sparse object waypoints is available, we pad it to length $T+L_{init}$ , and project it to the latent feature dimension, and then fuse with the object motion through feature-wise concatenation. When full object trajectories are provided as additional conditions (Stage III), we incorporate them by temporally concatenating the trajectories with the input human motion.

Training. The model is implemented in PyTorch (30) and trained on a single NVIDIA A6000 GPU. Our training setting involves 200k steps and utilize a batch size of 128 and employ the AdamW optimizer (26) with a learning rate set at $2\times 10^{-4}$ .

During training, we use the AdamW optimizer with $\beta_{1}{=}0.9$ and $\beta_{2}{=}0.99$ . Following prior work, we train the autoencoders on ParaHome and OMOMO with a batch size of 128, where each sample contains 64 frames. We then train the diffusion models for 200K steps with a batch size of 64 and a maximum sequence length of 124 frames. The learning rate is set to $2\times 10^{-4}$ .

For additional experiments on the ARCTIC and HIMO benchmarks, we follow their task settings using our model implementation.

\begin{overpic}[figures/ablation_componets_vis.pdf] \put(6.0,120.0){\small{w/o Contact}} \put(6.0,110.0){\small{Optimization}} \put(6.0,210.0){\small{w/o Separate}} \put(15.0,200.0){\small{Stage}} \put(6.0,30.0){\small{Ours (Full)}} \end{overpic}

Figure 3. Visual Results of Ablation Study.

\begin{overpic}[figures/results_gallery_omomo.pdf] \end{overpic}

Figure 4. Results Gallery. We provide additional results for single-object interaction on OMOMO dataset

		2 objects/components								3 objects/components
Dataset	Method	Motion					Interaction			Motion					Interaction
		FID $\downarrow$	$R_{prec}$ $\uparrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$	FID $\downarrow$	$R_{prec}$ $\uparrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$
ParaHome	Real motion (reference)	0.00	0.727	7.78	0.0039	0.15	-	-	-	0.00	0.679	7.47	0.0031	0.05	-	-	-
	w/o Separate Stage	4.52	0.673	7.69	0.0027	1.14	0.544	0.861	0.550	6.60	0.580	7.30	0.0017	2.34	0.646	0.898	0.773
	w/o Contact Optimization	4.55	0.691	7.85	0.0064	0.72	0.613	0.883	0.619	6.14	0.581	7.90	0.0022	0.40	0.657	0.911	0.763
	Ours	4.49	0.707	8.38	0.0035	0.72	0.669	0.896	0.536	6.09	0.598	7.57	0.0017	0.40	0.680	0.906	0.776
HIMO	Real motion (reference)	0.000	0.729	11.905	0.0007	0.08	-	-	-	0.267	0.713	9.755	0.0004	0.10	-	-	-
	w/o Separate Stage	7.821	0.579	10.963	0.0017	0.98	0.674	0.829	0.535	3.571	0.547	10.212	0.0026	1.44	0.693	0.812	0.638
	w/o Contact Optimization	6.893	0.608	10.358	0.0012	0.17	0.706	0.897	0.572	2.345	0.613	9.907	0.0022	0.19	0.737	0.879	0.603
	Ours	6.572	0.612	11.021	0.0011	0.17	0.731	0.912	0.519	2.273	0.628	9.812	0.0015	0.19	0.751	0.883	0.621

Table 1. Ablation results on ParaHome and HIMO for multi-object and articulated HOI generation. We compare against our full method and its variants. Best results are highlighted in bold.

Contact Optimization Detail. During inference in Stage III, we perform noise-space optimization with an ODE solver using 5 denoising steps and 200 noise optimization iterations. We use a cosine-decayed learning rate with an initial value of $0.05$ . The optimization objective is a weighted sum of losses, with $\lambda_{\text{pene}}=0.001$ .

Marker-to-SMPLX Fitting Model. For rendering and evaluation, we train a lightweight 10-layer ResNet for 100 epochs on ParaHome and OMOMO to regress SMPL-X parameters from the input human markers. We use the neutral-gender SMPL-X model and predict the shape coefficients $\bm{\beta}$ , SMPL-X pose $\bm{\theta}$ and global translation. During inference, we further perform a 200-step test-time optimization to refine the fitted parameters and better align the recovered mesh with the marker observations. Specifically, given a set of target markers $\mathbf{H}\in\mathbb{R}^{T\times M\times 3}$ , we only optimize $\bm{\theta}$ and shape coefficients $\bm{\beta}$ to minimize the L1 reconstruction error between SMPL-X markers and target markers. We optimize body poses, hands poses, and shape coefficients ( $\bm{\beta}$ ) with separate Adam optimizers at a learning rate of $10^{-3}$ . We run 400 optimization steps and compute the marker loss:

(27)

\mathcal{L}_{\text{marker}}=\left\|\mathbf{H}-\tilde{\mathbf{H}}(\bm{\theta},\bm{\beta})\right\|_{1},

where $\tilde{\mathbf{H}}(\cdot)$ denotes the SMPL-X markers extracted from the reconstructed mesh vertices. During fitting, facial and eye pose parameters are fixed to zero. After optimization, we use the fitted SMPL-X parameters to recover the full human mesh for visualization and metric computation.

Table 5 reports the fitting error before and after this optimization: the feed-forward regressor provides a coarse initialization, which the optimization refines to a marker error of 21.88 mm, confirming that the recovered mesh closely matches the input markers and does not introduce a significant bottleneck in the reported human-motion metrics.

Appendix E Additional Results

\begin{overpic}[figures/comparative_results_himo.pdf] \put(10.0,30.0){\small{Ours}} \put(5.0,120.0){\small{HIMO-Gen}} \put(10.0,240.0){\small{Ours}} \put(5.0,320.0){\small{HIMO-Gen}} \put(10.0,440.0){\small{Ours}} \put(5.0,520.0){\small{HIMO-Gen}} \par\end{overpic}

Figure 5. Qualitative comparisons on the test set of HIMO dataset. Compared to baseline method, our approach generates more realistic and coherent human–object interactions, featuring more accurate contact and fewer artifacts.

Additional results of ablation study. We compare our full model with several variants to evaluate the contribution of each component. Without the three-stage design (Separate Stage), the generated object motion becomes unstable, leading to physically implausible interactions. Without contact optimization, the object motion can still exhibit plausible patterns, such as opening or closing the laptop, but the human motion often fails to establish accurate contact, especially in complex multi-object scenarios. This suggests that contact optimization plays an important role in refining fine-grained human-object interactions after the initial motion generation.

Furthermore, we provide quantitative results for the ablation study in Table 1. The results show that removing key components consistently degrades performance, while our full model achieves better motion quality and interaction accuracy across the evaluated metrics. These findings demonstrate the effectiveness of the proposed staged generation pipeline and the contact-aware refinement strategy.

Additional visual results on OMOMO. We also provide additional qualitative results on the OMOMO dataset for single-object interactions, as shown in Figure 4. These examples demonstrate that our method also work well on the single-object setting, producing natural whole-body motions with accurate spatial alignment and physically plausible contacts.

Additional visual results on HIMO benchmark. We present additional qualitative results in Figure 5. These examples further illustrate the ability of our method to generate coherent human-object interactions across diverse scenarios and varying object configurations.

Qualitative results on ARCTIC benchmark. We present qualitative results in Figure 6, which further demonstrate the effectiveness of our method in generating realistic and coherent interactions for articulated objects.

\begin{overpic}[figures/comparative_results_arctic_2rows.pdf] \put(10.0,30.0){\small{Ours}} \put(5.0,110.0){\small{HIMO-Gen}} \put(10.0,190.0){\small{CoDA}} \put(10.0,310.0){\small{Ours}} \put(5.0,390.0){\small{HIMO-Gen}} \put(10.0,470.0){\small{CoDA}} \end{overpic}

Figure 6. Qualitative comparisons on the test set of ARCTIC dataset. Compared to baseline methods, our approach generates more realistic and coherent human–object interactions, featuring more accurate contact and fewer artifacts. For HIMO-Gen, We modify its input and output layer to suit this articulated object motion.

Quantitative results on OMOMO benchmark using GT object motion. We report quantitative comparisons on the OMOMO dataset, where all methods are conditioned on the GT object motion. As a reference upper bound, we include GT-OMOMO (20), which utilizes ground-truth object motion as input.

We evaluate human motion quality using FID, R-Precision (R_prec), and Foot Sliding (FS), and measure the deviation from ground-truth motion using GT Difference metrics, including MPJPE (mean per-joint position error) and root translation error (T_root). These metrics quantify how closely the generated motion matches the ground-truth human motion in both joint positions and global translation.

As shown in Table 2, our method achieves the best overall performance across most metrics. In particular, it yields the lowest FID and GT Difference errors (MPJPE and T_root), indicating improved motion realism and closer alignment with ground-truth motion. Compared to the variant without noise optimization, our full model further reduces reconstruction errors, demonstrating the effectiveness of the proposed optimization strategy.

	OMOMO Dataset
Method	Human Motion			GT Difference
	FID $\downarrow$	$R_{prec}$ $\uparrow$	FS $\downarrow$	MPJPE $\downarrow$	$T_{root}$ $\downarrow$
Real	0.00	0.566	0.0002	-	-
GT-OMOMO (20)	6.79	0.492	0.0025	14.37	21.55
w $/$ o Noise Optimization	2.85	0.524	0.0003	12.78	20.65
Ours	2.78	0.531	0.0003	11.59	19.57

Table 2. Quantitative results on the OMOMO dataset for conditioning on the GT object motion. We compare our full method with existing baseline. The best results are highlighted in bold, and the second-best results are underlined.

Quantitative results on the ParaHome for unseen multi-object composition. Test sequences contain object combinations never seen together during training, while every individual object is observed in training. We report results over 118 two-object and 35 three-object held-out sequences as shown in Table 9. $R_{prec}$ is omitted because ParaHome uses templated captions, causing heavy caption collisions under this split; interaction metrics are undefined for ground truth. Under unseen compositions HIMO-Gen’s object motion degrades sharply—foot sliding (FS) and object jerk (Jerk_obj) are an order of magnitude higher—while our staged design keeps object motion smooth (low FS/Jerk_obj), attains the lowest FID, and yields markedly higher contact accuracy; the only regression is three-object penetration.

			Revolute
Method	Prismatic	Screw	ParaHome	ARCTIC
HIMO-Gen	0.0041	0.3532	0.1589	0.2173
CoDA	N/A	N/A	$0.0000^{\dagger}$	$0.0000^{\dagger}$
Ours	0.0014	0.2493	0.0641	0.0157
Ours + Correction	0.0000	0.0307	0.0216	0.0011

$\dagger$ Trivially zero: CoDA assumes a fixed, hardcoded hinge axis, so its revolute residual is zero by construction and not directly comparable.

Table 3. Hinge compliance of generated articulated objects: deviation of generated parts from their hinge axes (lower is better). We outperform HIMO-Gen on every joint type. CoDA attains

0

on revolute joints only because it hardcodes a single hinge axis (

\dagger

), which also prevents it from representing prismatic or screw joints (N/A); our surface-keypoint representation supports all three. “Ours + Correction” projects the generated motion onto the constrained trajectory given a predefined pivot/axis and joint type (no retraining), leaving only a small residual while keeping generation joint-type-agnostic.

Hinge compliance of generated articulated objects. We measure hinge compliance as the deviation of generated articulated parts from their hinge axes (lower is better; Table 3). Our method outperforms HIMO-Gen across all joint types. CoDA attains zero deviation on revolute joints only because it hardcodes a single hinge axis, which also prevents it from representing prismatic or screw joints at all; in contrast, our surface-keypoint representation supports all three joint types without any per-mechanism assumption. Finally, “Ours + Correction” takes the generated motion together with a predefined pivot/axis and articulation type and projects it onto the corresponding constrained trajectory (no retraining), substantially reducing the residual while keeping the generator joint-type-agnostic. This is a favorable trade-off: CoDA achieves exact compliance for a single mechanism, whereas we support all three with only a small, correctable residual.

Component-slot ordering and permutation sensitivity. Our model represents an interaction as a set of per-component token streams (Sec. 3.2), so the assignment of physical components to input slots is arbitrary. During preprocessing on ParaHome we randomize this ordering, so the model is exposed to varied slot assignments throughout training, and the object tokens are processed jointly by the Transformer rather than in any fixed per-slot channel. To verify that generation is insensitive to this choice, we re-evaluate the two-object ParaHome split under the original component order and three random slot permutations. As reported in Table 10, FID and both contact-accuracy metrics are essentially unchanged—FID varies by only $0.02$ and the contact accuracies by $0.002$ across permutations—confirming that our results do not depend on how components are assigned to slots.

Modality	Metric	Value
Human motion	MPJPE (mm) $\downarrow$	27.52
Object motion	Vertex error (mm) $\downarrow$	16.28
Contact field	MAE $\downarrow$	0.0073
	RMSE $\downarrow$	0.0205
	MAE on contact cells ( $>\!0.5$ ) $\downarrow$	0.0665
	Precision / Recall / F1 $\uparrow$	0.928 / 0.927 / 0.927

Table 4. Reconstruction quality of the frozen Causal Autoencoder on held-out ParaHome/OMOMO sequences. Because all three stages generate in this latent space (Sec. D), these numbers upper-bound the fidelity attainable by the diffusion models. Motion errors are per-marker/per-vertex

L_{2}

in millimeters; contact-field errors are on the

[0,1]

distance field, and precision/recall/F1 use a

0.5

threshold.

Autoencoder reconstruction quality. Because every stage generates in the latent space of the frozen autoencoder (Sec. D), its reconstruction quality upper-bounds the fidelity of the full pipeline. Table 4 reports encode–decode error on held-out sequences. Human motion is recovered with a mean per-marker error of $27.5$ mm and object keypoints with $16.3$ mm, both small relative to body and object scale. The contact distance field is reconstructed almost exactly (MAE $0.007$ over the $[0,1]$ field); although the field is dominated by a near-zero background, the per-marker contact codec recovers the actual contacts with well-balanced precision and recall ( $0.928/0.927$ ; F1 $0.927$ at a $0.5$ threshold), confirming that the latent space preserves the fine-grained, sparse contact structure the downstream stages depend on.

Metric (mm) $\downarrow$	Regression only	+ Test-time Optimization
Marker error	101.83	21.88
Joint MPJPE	99.96	33.72
Trans error	60.32	18.93

Table 5. Marker-to-SMPL-X fitting error (mm) on held-out sequences, before and after the 400-step test-time optimization.

Quantitative kinematic-validity measurements. Although articulated components are generated without explicit joint constraints, we quantify how often the outputs are kinematically invalid by thresholding per-frame residuals, evaluated on $282$ articulated part–base pairs across $222$ ParaHome sequences (Table 6). Joint axis measures the deviation of the per-frame relative-rotation axis between a part and its base from the fitted hinge axis; it extends our hinge-compliance residual by thresholding the same per-frame residual at $10^{\circ}$ to yield a violation rate rather than a mean deviation. Joint limits flag frames whose recovered joint value exceeds the per-category range measured from ground truth by more than $10\%$ , verifying that the motion magnitude stays physical. Assembly connectivity fits a single shared base-to-part pivot by least squares and reports its residual (connection drift), flagging frames whose drift exceeds $2$ cm; this verifies that the two parts remain assembled around one consistent pivot. Joint limits and connectivity are complementary—the former checks motion magnitude, the latter structural consistency. Across all three aspects the violation rates are low (frame-level $\leq\!5.2\%$ , with a mean pivot drift of only $0.43$ cm and a median of $0$ ), showing that the generated articulated motions are kinematically valid the large majority of the time even without hard constraints. The small residual violations are precisely what the optional projection step removes.

Robustness of Kabsch pose recovery. We recover each component’s per-frame $SE(3)$ pose from its predicted keypoints using the Kabsch algorithm (SVD with reflection handling), which returns an exact $SO(3)$ rotation for any non-collinear keypoint set. The canonical keypoints are selected by farthest-point sampling, which maximizes their spread, so the triplets are well conditioned across all 87 objects (median anisotropy $\sigma_{2}/\sigma_{1}{=}0.65$ , mean minimum interior angle $42^{\circ}$ ). Recovery stays stable while $\sigma_{2}/\sigma_{1}{>}0.2$ and degrades only as the keypoints approach collinearity ( $\sigma_{2}/\sigma_{1}{<}0.1$ ), a regime reached only by thin, elongated objects (e.g., knife, pen), which we state explicitly as a limitation. Under injected keypoint noise the rotation error grows roughly linearly ( $\approx\!1.3^{\circ}/$ mm for $k{=}3$ ; Table 7). Because our formulation and solver accept any number of keypoints, an over-determined set ( $k>3$ ) both improves noise robustness and eliminates the collinearity failure mode, and we therefore adopt $k>3$ for elongated objects.

Validity aspect	Threshold	Frames $\downarrow$
Joint axis	$>\!10^{\circ}$ axis deviation	1.46%
Joint limits	$>$ GT range $+\,10\%$	0.02%
Assembly connectivity	$>\!2$ cm pivot drift	5.20%

Table 6. Kinematic-validity violation rates on ParaHome (282 articulated part–base pairs across 222 sequences). Components are generated without explicit joint constraints; we threshold per-frame residuals to measure how often the outputs are kinematically invalid. A frame is a violation if its residual exceeds the threshold; Assembly connectivity is reported at the frame level, with mean/median pivot drift of

0.43

0.00

cm. All rates are low.

Keypoint noise	$k{=}3$	$k{=}6$	$k{=}10$
5 mm	$6.3^{\circ}$	$3.8^{\circ}$	$2.9^{\circ}$
10 mm	$13.8^{\circ}$	$6.5^{\circ}$	$4.6^{\circ}$

Table 7. Robustness of Kabsch pose recovery. We apply a known

SE(3)

to the canonical keypoints, perturb them with isotropic Gaussian noise, and recover the pose by Kabsch (SVD with reflection handling); we report rotation error (degrees). An over-determined keypoint set (

k>3

) markedly improves robustness and removes the near-collinear failure mode. This i.i.d. setting is a worst case—the model’s per-frame keypoints are temporally correlated and smoother, and translation recovery is considerably more stable.

Effect of the number of object keypoints.

Num. of	Motion					Interaction
Keypoints	FID $\downarrow$	$R_{prec}$ $\uparrow$	Div $\rightarrow$	FS $\downarrow$	Jerk ${}_{obj}\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}\uparrow$	Pene $\downarrow$
3	6.09	0.598	7.57	0.0017	0.40	0.680	0.906	0.776
6	6.21	0.576	7.81	0.0021	0.42	0.673	0.894	0.721
9	6.18	0.582	7.74	0.0020	0.38	0.668	0.890	0.716

Table 8. Quantitative results on ParaHome (3 objects) with different numbers of keypoints

K

. Best results are highlighted in bold.

Table 8 studies how the number of sampled object keypoints $K$ affects generation quality and interaction plausibility. Overall, we observe a clear trade-off between motion fidelity and interaction accuracy. Using fewer keypoints ( $K{=}3$ ) yields the best overall performance, achieving the lowest FID, R_prec, and foot sliding score (FS), which indicates smoother and consistent motions. It also provides the strongest interaction quality, with the highest contact accuracies. In contrast, increasing the number of keypoints does not further improve interaction plausibility and instead degrades object human motion quality (e.g., R_prec), and contact accuracy, suggesting that overly dense object representations may overfit local geometry and introduce unnecessary constraints during generation. Based on these results, we set $K{=}3$ as a good trade-off between motion realism and physically plausible interactions.

	ParaHome — Unseen Compositions (2 Objects)							ParaHome — Unseen Compositions (3 Objects)
Method	Motion				Interaction			Motion				Interaction
	FID $\downarrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$	FID $\downarrow$	Div $\rightarrow$	FS $\downarrow$	Jerk_obj $\downarrow$	$C_{acc}^{tem}\uparrow$	$C_{acc}^{body}$ $\uparrow$	Pene $\downarrow$
	Unseen Multi-Object Compositions
Real	0.00	6.18	0.0059	0.14	-	-	-	0.00	5.00	0.0091	0.14	-	-	-
HIMO-Gen	12.37	7.63	0.2791	4.89	0.511	0.623	0.622	10.87	4.52	0.3271	6.28	0.543	0.627	0.578
Ours	7.81	6.96	0.0132	0.59	0.692	0.863	0.516	8.32	5.82	0.0571	0.48	0.675	0.821	0.663

Table 9. Ground-truth (Real) statistics on the ParaHome unseen multi-object composition split. Test sequences contain object combinations never seen together during training (e.g. {book, bookshelf, desk}, {cutting board, pan}), while every individual object is observed in training. We report GT reference statistics over 118 (2-object) and 35 (3-object) held-out sequences.

R_{prec}

/ Matching Score are omitted here: ParaHome uses templated captions, so restricting to a few held-out compositions causes heavy caption collisions (e.g. 41 sequences share one caption), which makes batch-wise text-to-motion retrieval degenerate; interaction metrics are undefined for ground truth.

Effect of contact representations. We compare three types of intermediate contact supervision: our distance field, a binary contact label, and the raw Euclidean distance. For evaluation, we directly threshold the predicted contact signals to obtain binary contact labels and compare them against the ground-truth labels using the contact metrics described in Sec. B.

As shown in Table 5, the proposed distance field achieves the best overall results across all metrics, indicating that it provides a more informative and learnable interaction cue.

In contrast, the binary label leads to a clear performance drop. We attribute this to its extreme sparsity: most marker–object pairs are non-contact and thus take zero values, resulting in weak gradients and making it difficult for the model to learn fine-grained correspondence and contact switching over time. Using raw Euclidean distances, as in ROG (55), also degrades performance, as they are absolute and unnormalized measures with a large dynamic range. This increases the learning complexity and makes training sensitive to scale variations across different objects and motions.

Some works utilize Signed Distance Fields (SDF) to encode object geometry, which represents signed inside/outside information. While SDF is a powerful representation for geometry modeling, we find that it is not necessary for our interaction-centric objective. Our goal is to model interaction-aware proximity between human and object surfaces over time. Overall, our distance-field formulation strikes a better balance between informativeness and numerical stability, leading to more accurate and physically plausible HOI generation.

Inference speed. Table 11 report the inference time of each module in our pipeline. All timings are measured on a single NVIDIA A6000 GPU for generating a 124-frame sequence. The overall runtime is primarily dominated by the contact optimization in Stage III, which involves iterative ODE-based sampling and gradient-based updates. Nevertheless, our method remains significantly more efficient than CoDA (36). In particular, CoDA does not support batch-level optimization and requires approximately 14 minutes per sample, whereas our approach enables more efficient batched processing.

Permutation	FID $\downarrow$	$C^{tem}_{acc}\uparrow$	$C^{body}_{acc}\uparrow$
Original order	4.49	0.669	0.896
Random perm. #1	4.52	0.665	0.893
Random perm. #2	4.47	0.671	0.898
Random perm. #3	4.51	0.667	0.895
Mean $\pm$ std	$4.50\pm 0.02$	$0.668\pm 0.002$	$0.895\pm 0.002$

Table 10. Sensitivity to component-slot ordering on the ParaHome two-object split. We re-run generation with the input components assigned to different slot orders. FID and contact accuracy are stable across permutations (std

\leq\!0.02

), showing our model is effectively invariant to component ordering.

Module	Time
Object Motion Generation	0.22 secs
Contact Distance Field Prediction	0.59 secs
Body Motion Synthesis w/ Contact Optimization	4.21 mins

Table 11. Inference time.

Appendix F Details of User Study

Figure 7 illustrates the questionnaire interface. For each trial, participants are shown the text description and a set of anonymized generated animations. For single rigid object interactions, we run a three-way comparison among our method, CHOIS, and HOI-Diff. For multiple/articulated objects interactions, we run a pairwise comparison between our method and HIMO. Participants provide two judgments per trial: text alignment (which animation best matches the prompt) and interaction quality (which looks more natural and physically plausible), with an optional “Can’t tell” choice when differences are unclear. To minimize presentation bias, we randomize the placement of results across trials. For fairness, we keep the camera consistent across methods.

Appendix G Failure Cases

\begin{overpic}[figures/failure_cases.pdf] \put(38.0,9.0){\small{foot floating}} \put(158.0,9.0){\small{contact missing}} \par\end{overpic}

Figure 8. Representative failure cases. Left: errors in body fitting lead to incorrect body scale and foot floating. Right: temporally inconsistent distance field predictions result in missed contacts.

In Figure 8, we present several failure cases of our method. First, the fitting model is not always accurate; in some instances, it recovers a human mesh with a smaller height, leading to foot floating, as shown on the left. Additionally, contact may be missed when the predicted distance field is not sufficiently accurate in the temporal dimension.

Appendix H Supplemental Video.

We provide a supplemental video to qualitatively demonstrate the effectiveness of our method. The video includes: (i) background and motivation; (ii) additional comparisons with baseline methods on ParaHome and OMOMO; (iii) diverse examples of multi-object manipulation and articulated object interactions; (iv) ablation results illustrating the contributions of key components; (v) predicted contact distance field; and (vi) failure cases and more our visual results. We recommend viewing the video in full screen to better inspect contact events and interaction details.

Textual prompt: Open the cabinet.

$t=4$	$t=43$	$t=83$	$t=123$