logo GeneOH Diffusion

Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion

Xueyi Liu1,3 Li Yi1,2,3
1Tsinghua University, 2Shanghai Artificial Intelligence Laboratory, 3Shanghai Qi Zhi Institute
ICLR 2024

GeneOH Diffusion cleans out-of-domain erroneous HOI tracks with new objects, motions, and novel noise distributions into natural sequences by only training on limited data.


In this work, we tackle the challenging problem of denoising hand-object interactions (HOI). Given an erroneous interaction sequence, the objective is to refine the incorrect hand trajectory to remove interaction artifacts for a perceptually realistic sequence. This challenge involves intricate interaction noise, including unnatural hand poses and incorrect hand-object relations, alongside the necessity for robust generalization to new interactions and diverse noise patterns. We tackle those challenges through a novel approach, GeneOH Diffusion , incorporating two key designs: an innovative contact-centric HOI representation named GeneOH and a new domain-generalizable denoising scheme. The contact-centric representation GeneOH informatively parameterizes the HOI process, facilitating enhanced generalization across various HOI scenarios. The new denoising scheme consists of a canonical denoising model trained to project noisy data samples from a whitened noise space to a clean data manifold and a "denoising via diffusion" strategy which can handle input trajectories with various noise patterns by first diffusing them to align with the whitened noise space and cleaning via the canonical denoiser. Extensive experiments on four benchmarks with significant domain variations demonstrate the superior effectiveness of our method. GeneOH Diffusion also shows promise for various downstream applications.


GRAB test set

Challenges: novel interactions with new objects.

More results

GRAB (Beta) test set

Challenges: 1) novel interactions with new objects, 2) unseen synthetic noise patterns.

More results

HOI4D dataset

Challenges: 1) difficult and novel interactions with new objects and 2) unseen real noise patterns.

More results

ARCTIC dataset

Challenges: challenging interactions with changing contacts.

Comparisons on sequences of standard length

More results

Results on long sequences with bimanual manipulations

Stochastic Denoising Results

More results


More results


Please contact us at xymeow7@gmail.com if you have any question.


      title={GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion},
      author={Liu, Xueyi and Yi, Li},
      booktitle={The Twelfth International Conference on Learning Representations},