Eye Tracking using Deflectometric Information
Exploiting deflectometric information for high precision eye tracking
Although the task has been studied for several decades, robust, accurate, and fast eye tracking remains a challenging problem. To estimate the gazing direction of the human eye, current approaches utilize 2D features detected from eye images, or exploit sparse reflections of a few point light sources at the eye surface (“corneal/scleral reflections”). Our approach significantly increases the information content provided from corneal or scleral reflections by using teachings from single-shot Deflectometry to acquire a dense and precise 3D model of the eye surface. To estimate the gazing direction, we exploit the retrieved surface normals and dense 3D features extracted from the measurement. Additionally, we utilize learning-based and an inverse rendering-based approaches to estimate the gazing direction from the retrieved deflectometric information.
This project is current work in progress. More information will be available soon.
Applications: The concept of deflectometric eye tracking can be used in tabletop or mobile setups, as well as in VR/AR/MR headsets and data glasses. Applications can be found in Aerospace technology, Human-Computer-Interaction, Medicine, Behavioral Science, or Automotive. In particular it became evident in recent years that precise and fast eye tracking can solve many of the current problems in VR/AR/MR headsets, such as foveated rendering.
Related Talk Recordings
|
F. Willomitzer
Utilizing deflectometric information for single-shot eye-tracking with high accuracy
OPTICA Imaging Congress, 3D Image Acquisition and Display: Technology, Perception and Applications, Boston, 2023
[Full Video]
| 
|
F. Willomitzer The Computational 3D Imaging and Measurement Lab - From Fast and Precise 3D Sensors to Looking Around Corners Industrial Affiliates Fall Meeting, University of Arizona, 2023 [Full Video] | 
|
F. Willomitzer
Utilizing Nature’s Limits for Better Computational 3D Imaging
Center For Robotics and Biosystems Seminar, Northwestern University, USA, 2022
[Full Video] , [Jump to Project (29:40)] |
Publications
Accurate Eye Tracking from Dense 3D Surface Reconstructions using Single-Shot Deflectometry
J. Wang, T. Wang, B. Xu, O. Cossairt, F. Willomitzer.
ArXiv Preprint, 2023
3D Imaging of Complex Specular Surfaces by Fusing Polarimetric and Deflectometric Information
J. Wang, O. Cossairt, F. Willomitzer
ArXiv Preprint, 2024
Differentiable Deflectometric Eye Tracking
T. Wang, J. Wang, O. Cossairt, F. Willomitzer
IEEE Transactions on Computational Imaging, 2024
Accurate Eye-Tracking from Deflectometric Information using Deep Learning
J. Choi, J. Wang, T. Wang, F. Willomitzer
Optica Open, 2024
Accurate and Fast VR Eye-Tracking using Deflectometric Information
J. Wang, T. Wang, B. Xu, O. Cossairt, F. Willomitzer.
Optica Computational Optical Sensing and Imaging, 2023
Optimization-Based Eye Tracking using Deflectometric Information
T. Wang, J. Wang, O. Cossairt, F. Willomitzer.
Proceedings of the 124th DGaO Conference, Berlin, B7, 2023
Precise VR Eye Tracking using Single-Shot Deflectometry
J. Wang, T. Wang, B. Xu, O. Cossairt, F. Willomitzer.
Proceedings of the 124th DGaO Conference, Berlin, A31, 2023
VR Eye-Tracking using Deflectometry
J. Wang*, B. Xu* (* joint first authors), T. Wang, W. Lee, M. Walton, N. Matsuda, O. Cossairt, F. Willomitzer.
OSA Computational Optical Sensing and Imaging, 2021
Easy and Flexible Calibration Approach for Deflectometry-based VR Eye-Tracking systems
J. Wang, B. Xu, O. Cossairt, F. Willomitzer
OSA Computational Optical Sensing and Imaging, 2022
Publications on related topics:
Hand-Guided Qualitative Deflectometry with a Mobile Device
F. Willomitzer, C-K. Yeh, V. Gupta, W. Spies, F. Schiffers, A. Katsaggelos, M. Walton, O. Cossairt.
Optics Express 28(7), 9027-9038, 2020
Low-budget 3D scanning and material estimation using PyTorch3D
O. Cossairt, F. Willomitzer, C.-K. Yeh, M. Walton.
54th IEEE Asilomar Conference on Signals, Systems and Computers, 2020
A Mitsuba-based Study on Trade-offs Between Projectionand Reflection Based Systems in Structured-Light 3D Imaging
T. Wang, F. Schiffers, F. Willomitzer, O. Cossairt.
OSA Computational Optical Sensing and Imaging, 2021
