mr fitting simulation and clothes modeling for wheelchair
TRANSCRIPT
![Page 1: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/1.jpg)
National Institute of Advanced Industrial Science and Technology
MR Fitting Simulation and Clothes Modeling for Wheelchair Users
○Ryosuke Ichikari, Masaki Onishi, and Takeshi Kurata AIST, Japan
2014/4/18-20 KJMR2014@Jeonju (KIST Jeonbuk Branch), Korea
![Page 2: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/2.jpg)
National Institute of Advanced Industrial Science and Technology
Backgrounds • Demands for improving quality of life (QoL) of
people with disabilities • Wheelchair users’ problems for purchasing clothes
– Difficulties of trial fitting – Lack of items specially designed for the users
Clothes fabrication is normally based on standing position
⇒Research development for supporting clothes fabrication for wheelchair users is required
2 Fashion show showing clothes for the wheelchair users
Images from http://www.rehab.go.jp/rehanews/japanese/webnews/201302/news_201302_6.html
![Page 3: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/3.jpg)
National Institute of Advanced Industrial Science and Technology
Research Project System
3
Project members consist of AIST, National Rehabilitation Center for Persons with Disabilities, and Bunka Fashion College (all in Japan)
Non-contactMeasurement by RGB-D Camera
Modeling/Capturing Clothes
MR Fitting Simulation
Contact-typeMeasurement
Pattern Makingby Apparel CAD
Design (Photo) ofthe Dedicated DummyMeasurement
of the Wheel-chair User
Virtual FittingActual ClothesFabrication
Visualizingthe measuredresults Utilizing data from
fitting results (size, personal preference, etc.)
Utilizing the measured data for the clothes fabrication
Feedback of data obtained duringclothes fabrication
![Page 4: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/4.jpg)
National Institute of Advanced Industrial Science and Technology
MR(Mixed Reality) Fitting Simulation
4
• Needs for MR fitting simulation from the field – Virtual fitting without physically-taxing operations – Visualizing the fitting result from various viewpoints – Simple configuration at a low price
• Computer-generated clothes are superimposed onto the users body by MR technology.
• Camera position is not restricted by the real positing, it can be changed virtually in the 3D-CG space.
![Page 5: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/5.jpg)
National Institute of Advanced Industrial Science and Technology
Related Works • MR-based virtual fitting systems in the industry
– EON Reality Eon Interective Mirror Visualizing virtual fitting results with simple cloth simulation in real time on a display which represents mirror
– DNP Fitting system demonstrated in UNIQLO. This shows color variations of the clothes by changing color of the clothes in the display.
• Researches related to virtual fitting or clothes fabrication – Stefan et.al. (ISMAR2011):
Fitting simulation based pre-captured clothes with various poses – Igarashi et.al.(UIST2002):
Interactive simulation of sewing the parts of clothes and fitting on CG dolls
5
No work have focused on the fitting simulation with sitting positions
![Page 6: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/6.jpg)
National Institute of Advanced Industrial Science and Technology
System Requirements and Prototype To be beneficial for both users and system owners Prototype Measuring users’ body for actual fabrications ✔ Trial fitting without forcing users to get off from the wheelchair ✔ Improve the quality of the experience of trial fitting ✔
6
Requirements for system configurations Simple configuration at a moderate price ✔
Adopting contact-less measurement for the users ✔
Requirements for MR functions Allowing to check their appearance from various view-points ✔ Composition of the clothes without visual paradox ✔ Accurate registration between CG clothes and real body ✔ Tracking the user motion and deformation of the clothes On going Practical method for preparing clothes models On going
![Page 7: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/7.jpg)
National Institute of Advanced Industrial Science and Technology
MR System Based on 3D Reconstruction of Objects in the Real World
• Visualization of output from the non- contact measuring method
• Inputting the sequence from a fixed RGB-D camera – Library:OpenNI(RGB-D camera)
OpenGL(Rendering CG space) • MR composition in the 3D-CG space where
real objects in the world are reconstructed in – Enabling virtual camera movement
• CG Clothes are prepared by capturing real clothes and superimposed onto the image of the user. – Utilizing multi-path rendering for rigorous compositions without
visual paradox
• . 7
RGB-D camera: Asus Xtion
![Page 8: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/8.jpg)
National Institute of Advanced Industrial Science and Technology
How to Render the MR Space • Real-Objects: Importing RGB-D sequence and rendered as
point cloud in the 3D-CG space (with OpenNI, OpenGL) • Clothes: Rendering mesh of the clothes generated by RGB-D
sequence in the same space file format of meshes: Collada(.dae)
8
CG model of the clothes Rendered RGB-D
sequence
Composition result
![Page 9: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/9.jpg)
National Institute of Advanced Industrial Science and Technology
Rigorous Composition without Visual Paradox • Geometric consistency is important
for MR-based visualization e.g. ) If inner clothes under superimposed outer clothes is visible, it may cause visual paradox.
• Rendering Method – Keeping the silhouette of the clothes – If the silhouette of the body is unnaturally protruded from the
clothes, these area should be deleted – Necessary parts (face, hand, foot) are preserved in the “clean-
up” process – The deleted area and occluded area of RGB-D camera are filled
with the pre-captured background image or inpainting method ⇒Composition process is implemented using multipath-rendering
9
Example of bad case
![Page 10: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/10.jpg)
National Institute of Advanced Industrial Science and Technology
Fulfilling the holes or the deleted areas • The holes and the deleted areas are fulfilled with
pre-captured background images or inpainting method*
10 *A. Telea, "An image inpainting technique based on the fast marching method," J. Graphics Tools, 2004
Input RGB-D image (rendered)
Background RGB-D Image(rendered)
After composition
![Page 11: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/11.jpg)
National Institute of Advanced Industrial Science and Technology
Topics about Preparing Clothes Model and Deformation
11
• How to prepare CG model of clothes – Simulation based: Using output of 3D sewing simulation from
apparel CAD
– Capturing based:Scanning actual clothes by 3D scanning.
• Deformation of the clothes model according to the shape of body and movement of joint angles
• Consideration of expansion and contraction(stretching) of the material
• Measuring the shape and pose of the users and their tracking.
• How to utilize the result of the simulation for the actual fabrication – Compatibility with the apparel CAD
![Page 12: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/12.jpg)
National Institute of Advanced Industrial Science and Technology
Pattern→3D clothes model
(Sewing Simulation)
Conceptual-Flow
Clothes model→Fitting results
(Fitting Simulation)
Pattern data
Torso data
3D clothes model
Shape data
Sitting Pose data Fitting result
Rigorous Composition
of the CG clothes
MR composition result
Contact-type measurement
Contact-less body capture
Real scene capture
Prototype
3D scanning of real clothes by RGB-D camera
Mesh Deformation of scanned mesh by
skeleton and measured data
![Page 13: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/13.jpg)
National Institute of Advanced Industrial Science and Technology
3D Scanning with RGB-D Camera • ReconstructMe: A 3D scanning software based on the
SLAM with RGB-D camera – Outputs: Triangle mesh with vertex color
13
![Page 14: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/14.jpg)
National Institute of Advanced Industrial Science and Technology
Pipeline of the Sawing Emulation • Preparing the CG model of the clothes
– Adding pattern information
• Captured mesh is separated into parts which imitates pattern of the clothes – No UV coordinate and texture at this time
• Example of the operating pipeline with CG software (Autodesk Maya)
– Meth separation – UV space definition by planner/cylindrical projection (Mesh Lab.) – Baking(Projecting) vertex color onto texture
14
![Page 15: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/15.jpg)
National Institute of Advanced Industrial Science and Technology
Mesh Separation and UV Space Definition • Imitating pattern of the real clothes by UV space
definition – (e.g.) Separating into four parts (front, back, left, right) – Capability for accessing/modifying each parts
separately
15
![Page 16: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/16.jpg)
National Institute of Advanced Industrial Science and Technology
Approximate Fitting Simulation • Deformation to deal with variation of the shape of body
– Shape deformation: Changing vertex position along normal direction
– Estimating size of real body and clothes for the deformation by ellipse approximation
• Deformation to deal with variation of the joint angles – Rigorous clothes simulation is not employed in the prototype – Employing skeleton based mesh deformation
• Pros: Calculate deformation of the clothes in real-time • Cons:
– Quality of the result depends on the mesh weight – Not physically aculeate (gravity, volume preservation, stretch
property, etc.)
16
![Page 17: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/17.jpg)
National Institute of Advanced Industrial Science and Technology
Example of the Shape Deformation • Shape deformation is done by moving vertex position
along the normal while keeping the UV coordinate
17
Normal of a Mesh
Mask for the deformation (target: waist)
![Page 18: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/18.jpg)
National Institute of Advanced Industrial Science and Technology
Example of Skinning and Deformation of Captured Mesh in Maya
• Definition of joints (skeleton) • Definition of mesh weight • Skeleton based mesh deformation
18
![Page 19: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/19.jpg)
National Institute of Advanced Industrial Science and Technology
Demo
19
![Page 20: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/20.jpg)
National Institute of Advanced Industrial Science and Technology
Conclusions
20
• Proposed MR fitting simulation system which aims for supporting wheelchair users to purchase clothes
• Utilized RGB-D camera for changing virtual viewpoints in the MR space
• Rigorous rendering pipeline keeps geometric consistency
• Proposed methods for preparing clothe model and deformation of the model – Clothe model creation using RGB-D camera based
scanning method – Mesh deformation to deal with the variety of the
shapes and the joint angles
![Page 21: MR Fitting Simulation and Clothes Modeling for Wheelchair](https://reader031.vdocument.in/reader031/viewer/2022011903/61d6b0b680fc5024880ec38b/html5/thumbnails/21.jpg)
National Institute of Advanced Industrial Science and Technology
Future Works
• Integration with contact-type measuring method for measuring occluded parts
• Capturing from back-side of the users • Automatic tracking of the skeleton from RGB-D
sequence • Measuring or representing material stretching • Feasibility study of the service with the proposed
system
21