From Perception to Simulation: The Emergence of World Models in Multi-modal Reasoning

Abstract: This opening remark sets the stage for the tutorial by highlighting the shift from perception-oriented vision systems toward world models that can predict, simulate, and support action. We briefly discuss the key capabilities required for useful world models, including dynamics, causality, physical consistency, counterfactual reasoning, planning, and evaluation. The session will then introduce the tutorial structure and invited speakers, whose talks provide complementary perspectives on building and understanding multimodal world models.

Abstract: Multimodal foundation models are evolving from isolated perception systems into unified architectures that understand and generate content across images, video, audio, and language. However, unifying modalities and tasks does not automatically make model outputs actionable. A model may understand a design request and generate a visually compelling interface, while still returning only flat pixels: objects cannot be selected, assets cannot be independently edited, and the result cannot be directly executed or verified by an agent. In this talk, we present a three-stage progression from perception to action. First, modality unification establishes a shared perceptual substrate while preserving modality-specific expertise through cross-modal routing and collaborative training. Second, task unification connects understanding with generation, enabling capabilities such as generative segmentation, fine-grained editing, and multimodal content creation. Third, we introduce an actionable interface stack that extends unified models into real digital workflows: Generative UI supports visual exploration, editable visual structures expose semantic RGBA layers and compositional relationships, and Design-grounded Coding translates visual intent, assets, and design constraints into executable interfaces. Using the Ming series as connected system examples, we discuss how intent can be preserved from multimodal perception to visual proposals, structured editing, and code execution. We conclude with evaluation challenges, failure propagation across the pipeline, and open questions for multimodal systems that move beyond understanding and generation toward creating, editing, and acting within digital worlds.

Abstract: Multimodal world models are moving beyond passive perception toward systems that can reason about, simulate, and interact with dynamic environments. However, as models increasingly rely on visual evidence, temporal structure, prior knowledge, and multi-step inference, reliability becomes a central challenge. This talk presents a reliability-oriented perspective on multimodal reasoning and world models. I will discuss several key questions: whether the model perceives the required evidence, grounds its reasoning in the correct visual cues, preserves structural and temporal consistency, maintains appropriate knowledge as the environment evolves, and verifies its conclusions before acting. Recent advances in grounded reasoning, continual adaptation, knowledge control, and multimodal unlearning will be discussed, together with examples from our recent work. The broader goal is to move from models that merely generate plausible predictions to systems that maintain coherent, controllable, and trustworthy internal representations of the world.

Abstract: Generative modeling has rapidly evolved from a tool for creating synthetic data to a foundational paradigm for solving diverse real-world problems. In this talk, I will present a series of works where generative priors are designed not merely for pose generation, but as robust, geometry-aware motion models that enable reconstruction, recovery, and reasoning under uncertainty. Starting from pose generation via Neural Riemannian Distance Fields (NRDF), I will show how articulated pose distributions (of humans or hands) can be faithfully captured on product manifolds of rotations, yielding powerful priors for inverse kinematics, image-based pose estimation, and denoising. I will then transition to motion models such as HuMoR, which introduces an expressive, dynamics-aware generative model of human motion via conditional VAEs, enabling robust 3D motion recovery even under noise, occlusions, or missing observations. Finally, Neural Riemannian Motion Fields (NRMF) extend these ideas to higher-order motion dynamics, explicitly modeling pose, velocity, and acceleration on motion manifolds through neural distance fields. Together, these models demonstrate how generative formulations serve as flexible priors that power tasks as diverse as denoising, temporal interpolation, spatiotemporal infilling, and recovery of motion from highly incomplete or corrupted data, as evidenced by Dyn-HaMR. By embedding geometry, dynamics, and stochasticity into generative frameworks, the talk will move towards a unified view where generation, perception, and recovery are all facets of the same modeling principle.

Abstract: Video is beyond a sequence of images. The unique spatio-temporal pattern makes video modeling and generation different from representing a single image. Instead of treating videos as a 3D volume of pixels, we propose to learn disentangled representation that separates the appearance, motion, geometry, and semantic information of a video. Utilizing these disentangled representations in an integrated manner can enhance the quality, realism, and interpretability of generated videos and improve the robustness of video modeling. In this talk, we present methods that explicitly utilize disentangled representations, including appearance, motion, geometry, and semantics, to improve performance in video generation and modeling tasks.

Abstract: World models - internal representations that let a robot predict, plan, and act - are central to progress across robotics, yet turning general-purpose video-generation and vision-language foundation models into something a controller can use remains an open problem. This tutorial presents cases of world models for robots across different robot embodiments, showing how imagined futures and generalist visual-language understanding can be converted into structured, action-ready representations. Through case studies and open discussion, I hope attendees will leave with practical thoughts on building world models across robot embodiments.