Abstract
This project explores the challenges and advancements in multi-view camera systems, 2D pose estimation, and 3D reconstruction for capturing and reconstructing live performances. It conducts a comparative analysis of methodologies at each stage of the pipeline, identifying strengths, weaknesses, and effective techniques. The study addresses the robustness of existing techniques in diverse scenarios and aims to integrate these fields into a unified framework for high-quality reconstructions. It contributes to the development of advanced multi-camera systems applicable across domains and serves as a valuable resource for future research in the field of performance capture and analysis.
Reference
Pratik Singh Bisht, Philippe Colantoni, Damien Muselet, Alain Trémeau, « MultiView Markerless MoCap – MultiView Performance Capture, 3D Pose Motion Reconstruction and Comparison » in 2022 16th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Bangkok, Thailand, 2023.
Objective
Methodology
Setup
Data Acquisition: Calibration
- As mentioned earlier, calibration is a 2-step approach for intrinsic and extrinsic parameters.
- Calibrate individual cameras using ChAruco board to get intrinsic matrix.
- Need to make sure that board covers different areas of field of view, rotations, etc. as shown.
- Next is common calibration for all cameras to get extrinsic matrices relative to the board placed on floor.
Data Acquisition: 2D Pose
- A simple and naive approach to synchronize the pose estimation input video.
- External Flash for synchronizing multiple GoPro cameras.
- Clipping at exact frame based on intensity peak caused by flash.
3D Triangulation
ThreeJS Visualization
Visualization Results
Pose Examples
Different types of poses categorized as Easy, Medium and Hard based on the complexity of pose, plus the amount of motion involved. Some examples shown are normal standing, T-Pose, lunges/knee bend, leap/jump, running (causing motion blur).
IXR Platform 