The analysis of video see-through challenges in mixed reality

doi:10.12267/j.issn.2096-5931.2024.06.004

Abstract

Abstract:

As the demand for the level of immersion in extended reality (which includes virtual reality and augmented reality) continues to grow, video see-through (VST) has become a more promising mixed-reality technique than optical see-through (OST) due to its advantages in environment brightness adaptability, field of view and occlusion implementation. However, a few products equipped with VST have showcased multiple problems as the technique is still in an early phase. Therefore, the analysis has been conducted in the aspects of temporal, spacial and image quality, focusing on problems of image latency, point of view accommodation, oculus resolution, perspective distortion and dynamic range. Hopefully, this paper can provide an insight into VST status quo and facilitate its future development.

Key words: mixed reality, video see-through, latency analysis, point of view accommodation, image quality

CLC Number:

R318
TP391

XIANG Xiao, ZHAO Lei. The analysis of video see-through challenges in mixed reality[J]. Information and Communications Technology and Policy, 2024, 50(6): 23-29.

Add to citation manager EndNote|Ris|BibTeX

URL:

http://ictp.caict.ac.cn/EN/10.12267/j.issn.2096-5931.2024.06.004

http://ictp.caict.ac.cn/EN/Y2024/V50/I6/23

Figures/Tables 4

References 23

[1]	MILGRAM P, KISHINO F. A taxonomy of mixed reality visual displays[J]. IEICE Transactions on Information and Systems, 1994, 77(12): 1321-1329.
[2]	Work Portfolio Blog. What’s the difference between AR,VR and MR?[EB/OL]. (2024-02-14)[2024-05-20]. https://forwork.meta.com/blog/difference-between-vr-ar-and-mr/.
[3]	JOHN C. Latency mitigation strategies[EB/OL]. (2013-02-22)[2024-05-20]. https://web.archive.org/web/20130225013015/ http://www.altdevblogaday.com/2013/02/22/latency-mitigation-strategies/.
[4]	ADELSTEIN B D, LEE T G, ELLIS S R. Head tracking latency in virtual environments: psychophysics and a model[C]// Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Los Angeles: Sage Publications, 2003, 47(20): 2083-2087.
[5]	MANIA K, ADELSTEIN B D, ELLIS S R, et al. Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity[C]// Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization. Los Angeles: ACM, 2004: 39-47.
[6]	ISHIHARA A, AGA H, ISHIHARA Y, et al. Integrating both parallax and latency compensation into video see-through head-mounted display[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(5): 2826-2836.
[7]	VAN WAVEREN J M P. The asynchronous time warp for virtual reality on consumer hardware[C]// Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. New York: ACM, 2016: 37-46.
[8]	Nvidia Corporation. Nvidia CloudXR Suite[EB/OL]. 2024[2024-05-20]. https://www.nvidia.com/en-us/design-visualization/solutions/cloud-xr/.
[9]	中国信息通信研究院, 华为技术有限公司, 京东方科技集团股份有限公司. 虚拟(增强)现实白皮书[R/OL]. (2021-03-30)[2024-05-20]. http://www.caict.ac.cn/kxyj/qwfb/bps/202103/P020210331549691466973.pdf.
[10]	Qualcomm Inc. Snapdragon XR2 Gen 2 platform[EB/OL]. 2024 [2024-05-20]. https://www.qualcomm.com/products/mobile/snapdragon/xr-vr-ar/snapdragon-xr2-gen-2-platform#Overview.
[11]	Qualcomm Inc. Snapdragon XR2+Gen 2 platform[EB/OL]. 2024[2024-05-20]. https://www.qualcomm.com/products/mobile/snapdragon/xr-vr-ar/snapdragon-xr2-plus-gen-2-platform.
[12]	BJÖRK T. Apple vision pro benchmark test 1: see-through latency, photon-to-photon[EB/OL]. (2024-02-14)[2024-05-20]. https://www.optofidelity.com/insights/blogs/apple-vision-pro-benchmark-test-1-see-through-latency-photon-to-photon.
[13]	CHAURASIA G, NIEUWOUDT A, ICHIM A E, et al. Passthrough+ real-time stereoscopic view synthesis for mobile mixed reality[J]. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2020, 3(1): 1-17.
[14]	XIAO L, NOURI S, HEGLAND J, et al. Neuralpassthrough: learned real-time view synthesis for vr[C]// ACM Siggraph 2022 Conference Proceedings. Vancouver: ACM, 2022: 1-9.
[15]	ROLLAND J P, FUCHS H. Optical versus video see-through head-mounted displays in medical visualization[J]. Presence, 2000, 9(3): 287-309.
[16]	NDJIKI-NYA P, KOPPEL M, DOSHKOV D, et al. Depth image-based rendering with advanced texture synthesis for 3-D video[J]. IEEE Transactions on Multimedia, 2011, 13(3): 453-465.
[17]	JANTET V, GUILLEMOT C, MORIN L. Object-based layered depth images for improved virtual view synthesis in rate-constrained context[C]// 2011 18th IEEE International Conference on Image Processing. Brussels: IEEE Press, 2011: 125-128.
[18]	HEDMAN P, ALSISAN S, SZELISKI R, et al. Casual 3D photography[J]. ACM Transactions on Graphics (TOG), 2017, 36(6): 1-15.
[19]	FLYNN J, BROXTON M, DEBEVEC P, et al. Deepview: view synthesis with learned gradient descent[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE Press, 2019: 2367-2376.
[20]	GAO C, SARAF A, KOPF J, et al. Dynamic view synthesis from dynamic monocular video[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE Press, 2021: 5712-5721.
[21]	GUTTAG K. Meta quest pro (part1) - - unbelievably bad AR passthrough[EB/OL]. (2023-01-03)[2024-05-20]. https://kguttag.com/2023/01/03/meta-quest-pro-part-1-unbelievably-bad-ar-passthrough/.
[22]	BEZMALINOVIC T. Passthrough on quest 3 will continue to improve, says meta CTO[EB/OL]. (2023-10-24)[2024-05-20]. https://mixed-news.com/en/meta-quest-3-passthrough-quality-will-improve/.
[23]	PATEL N. Apple vision pro review: magic, until it’s not[EB/OL]. (2024-01-30)[2024-05-20]. https://www.theverge.com/24054862/apple-vision-pro-review-vr-ar-headset-features-price.