VR Eye Tracking using Deflectometry

VR Eye Tracking using Deflectometry

Exploiting deflectometric information for high precision eye tracking

Although the task has been studied for several decades, robust, accurate, and fast eye tracking remains a challenging problem. To estimate the gazing direction of the human eye, current approaches utilize 2D features detected from eye images, or exploit sparse reflections of a few point light sources at the eye surface (“corneal/scleral reflections”). Our approach significantly increases the information content provided from corneal or scleral reflections by using teachings from single-shot Deflectometry to acquire a dense and precise 3D model of the eye surface. To estimate the gazing direction, we exploit the retrieved surface normals and dense 3D features extracted from the measurement. Additionally, we utilize learning-based and an inverse rendering-based approaches to estimate the gazing direction from the retrieved deflectometric information.

This project is current work in progress. More information will be available soon.

Applications: The concept of deflectometric eye tracking can be used in tabletop or mobile setups, as well as in VR/AR/MR headsets and data glasses. Applications can be found in Aerospace technology, Human-Computer-Interaction, Medicine, Behavioral Science, or Automotive. In particular it became evident in recent years that precise and fast eye tracking can solve many of the current problems in VR/AR/MR headsets, such as foveated rendering.

Related Talk Recordings

F. Willomitzer
Utilizing Nature’s Limits for Better Computational 3D Imaging
Center For Robotics and Biosystems Seminar, Northwestern University, USA, 2022
[Full Video] , [Jump to Project (29:40)]


VR Eye-Tracking using Deflectometry
J. Wang*, B. Xu* (* joint first authors), T. Wang, W. Lee, M. Walton, N. Matsuda, O. Cossairt, F. Willomitzer.
OSA Computational Optical Sensing and Imaging, 2021

Optimization-Based Eye Tracking using Deflectometric Information
T. Wang, J. Wang, O. Cossairt, F. Willomitzer.
ArXiv Preprint, 2023

Easy and Flexible Calibration Approach for Deflectometry-based VR Eye-Tracking systems
J. Wang, B. Xu, O. Cossairt, F. Willomitzer
OSA Computational Optical Sensing and Imaging, 2022

Publications on related topics:

Hand-Guided Qualitative Deflectometry with a Mobile Device
F. Willomitzer, C-K. Yeh, V. Gupta, W. Spies, F. Schiffers, A. Katsaggelos, M. Walton, O. Cossairt.
Optics Express 28(7), 9027-9038, 2020

Low-budget 3D scanning and material estimation using PyTorch3D
O. Cossairt, F. Willomitzer, C.-K. Yeh, M. Walton.
54th IEEE Asilomar Conference on Signals, Systems and Computers, 2020

A Mitsuba-based Study on Trade-offs Between Projectionand Reflection Based Systems in Structured-Light 3D Imaging
T. Wang, F. Schiffers, F. Willomitzer, O. Cossairt.
OSA Computational Optical Sensing and Imaging, 2021

Florian Willomitzer
Associate Professor