基于眼动跟踪的人机交互应用

doi:10.6040/j.issn.1672-3961.0.2020.346

摘要/Abstract

摘要：

使用价格低廉的网络摄像头作为视频源, 研究基于眼动识别和视线跟踪在计算机上精确交互的方法。提出一种基于外观人眼视线方向估计的人机交互方法: 使用一个简单快速的卷积神经网络模型粗略估计人眼在屏幕上的注视点, 进而将眼动识别和不精确视线跟踪结果用于实现在计算机上的精确人机交互。在此基础上开发了眼动鼠标与眼动打字的功能以替代鼠标和键盘硬件。试验结果表明: 本研究让使用者仅用一个普通的单目摄像头, 就可以实现眼动跟踪, 并利用眼动控制完成计算机上大部分精确的交互。

关键词: 视线追踪, 人机交互, 眼动鼠标, 眼动打字

Abstract:

To actualize the simple and low-cost eye-tracking based human-computer interaction, an exact interaction method based on the visual directions estimation and eye tracking with webcam videos was proposed. A simple and fast convolution neural network model was used to roughly estimate the user′s viewpoints on the screen. And then an accurate human-computer interaction method was proposed on the basis of the eye movements recognition and sight line tracking results. To verify the effectiveness of the method, the key operations of eye mouse and eye typing were developed. The test results show that the proposed method enabled users to achieve eye tracking and to actualize most precise human-computer interactions with only one common monocular camera, which was expected to completely replace the mouse and keyboard hardwares.

Key words: eye tracking, human computer interaction, eye mouse, eye typing

中图分类号:

TP319

贺辉,黄君浩. 基于眼动跟踪的人机交互应用[J]. 山东大学学报 (工学版), 2021, 51(2): 1-8.

Hui HE,Junhao HUANG. Eye tracking in human-computer interaction control[J]. Journal of Shandong University(Engineering Science), 2021, 51(2): 1-8.

图/表 21

图1

图2

图3

图4

表1

图5

图6

图7

图8

图9

图10

图11

图12

图13

图14

图15

表2

表3

图16

表4

图17

参考文献 21

1	王向军, 蔡方方, 刘峰, 等. 非接触动态实时视线跟踪技术[J]. 计算机科学与探索, 2015, 9 (3): 266- 278.
	WANG Xiangjun , CAI Fangfang , LIU Feng , et al. Non-contact dynamic real-time eye tracking technology[J]. Journal of Frontiers of Computer Science and Technology, 2015, 9 (3): 266- 278.
2	许洁, 王豪龙. 阅读行为眼动跟踪研究综述[J]. 出版科学, 2020, (2): 52- 66. doi: 10.3969/j.issn.1009-5853.2020.02.009
	XU Jie , WANG Haolong . A review of studies on behavioral eye tracking in reading[J]. Publishing Journal, 2020, (2): 52- 66. doi: 10.3969/j.issn.1009-5853.2020.02.009
3	尤洋, 王以宁, 张海. 教学视频终端显示空间对学习者影响的眼动研究[J]. 中国电化教育, 2019, (8): 123- 129. doi: 10.3969/j.issn.1006-9860.2019.08.018
	YOU Yang , WANG Yining , ZHANG Hai . An eye movement study on the influence of teaching video terminal display on learners[J]. China Educational Technology, 2019, (8): 123- 129. doi: 10.3969/j.issn.1006-9860.2019.08.018
4	靳慧斌, 刘亚威, 穆小萌. 基于眼动追踪的航空公司官网可用性评估[J]. 包装工程, 2018, 39 (10): 173- 176.
	JIN Huibin , LIU Yawei , MU Xiaomeng . Evaluation of the usability of airline official website based on eye tracking[J]. Packaging Engineering, 2018, 39 (10): 173- 176.
5	胡晓红, 王红, 任衍具. 基于眼动技术的互联网广告效果研究[J]. 计算机应用研究, 2018, 35 (5): 1345- 1349. doi: 10.3969/j.issn.1001-3695.2018.05.014
	HU Xiaohong , WANG Hong , REN Yanju . Research on network advertisement effect based on eye tracking technology[J]. Application Research of Computers, 2018, 35 (5): 1345- 1349. doi: 10.3969/j.issn.1001-3695.2018.05.014
6	WANLUK N, VISITSATTAPONGSE S, JUHONG A, et al. Smart wheelchair based on eye tracking[C]//2016 9th Biomedical Engineering International Conference (BMEi-CON). Laung Prabang, Laos: IEEE, 2016: 1-4.
7	王增磊, 张树生, 白晓亮. 面向协同装配的混合现实手眼交互技术研究[J]. 计算机应用与软件, 2019, 36 (1): 257- 264.
	WANG Zenglei , ZHANG Shusheng , BAI Xiaoliang . Mixed reality hand-eye interaction technology for collaborative assembly[J]. Computer Applications and Software, 2019, 36 (1): 257- 264.
8	PFEUFFER K, VIDAL M, TURNER J, et al. Pursuit calibration: Making gaze calibration less tedious and more flexible[C]//Proceedings of the 26th annual ACM symposium on User interface software and technology(UIST). Andrews Scotland, the United Kingdom: ACM, 2013: 3147-3150.
9	HOSSAIN Z, SHUVO M M H, SARKER P. Hardware and software implementation of real time electro-oculogram (EOG) acquisition system to control computer cursor with eyeball movement[C]//2017 4th International Conference on Advances in Electrical Engineering (ICAEE). Dhaka, Bangladesh: IEEE, 2017: 132-137.
10	CHAKRABORTY T, SARCAR S, SAMANTA D. Design and evaluation of a dwell-free eye typing technique[C]//Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems. Toronto, Canada: ACM, 2014: 1573-1578.
11	程时伟, 沈哓权, 孙凌云, 等. 多用户眼动跟踪数据的可视化共享与协同交互[J]. 软件学报, 2019, 30 (10): 3037- 3053.
	CHENG Shiwei , SHEN Xiaoquan , SUN Lingyun , et al. Shared visualization and collaborative interaction based on multiple user eye tracking data[J]. Journal of Software, 2019, 30 (10): 3037- 3053.
12	朱琳, 王圣凯, 袁伟舜, 等. 眼动控制的交互式地图设计[J]. 武汉大学学报(信息科学版), 2020, 45 (5): 736- 743.
	ZHU Lin , WANG Shengkai , YUAN Weishun , et al. An interactive map based on gaze control[J]. Geomatics and Information Science of Wuhan University, 2020, 45 (5): 736- 743.
13	朱麒文, 闫隆鑫, 张若兰, 等. 基于视线追踪的眼控鼠标设计[J]. 电子器件, 2016, 39 (2): 235- 241.
	ZHU Qiwen , YAN Longxin , ZHANG Ruolan , et al. Design of eye-controlled mouse based on eye tracking technology[J]. Chinese Journal of Electron Devices, 2016, 39 (2): 235- 241.
14	郜东瑞, 甘玉龙, 李鹏霄, 等. 基于眼电的智能输入系统研究[J]. 中国生物医学工程学报, 2015, 34 (6): 662- 669. doi: 10.3969/j.issn.0258-8021.2015.06.004
	GAO Dongrui , GAN Yulong , LI Pengxiao , et al. Research of intelligent speller system based on EOG[J]. Chinese Journal of Biomedical Engineering, 2015, 34 (6): 662- 669. doi: 10.3969/j.issn.0258-8021.2015.06.004
15	CAZZATO D , DOMINIO F , MANDUCHI R , et al. Real-time gaze estimation via pupil center tracking[J]. Paladyn Journal of Behavioral Robotics, 2018, 9 (1): 6- 18. doi: 10.1515/pjbr-2018-0002
16	ZHENG C, USAGAWA T. A Rapid Webcam-Based Eye Tracking Method for Human Computer Interaction[C]//2018 International Conference on Control, Automation and Information Sciences (ICCAIS). Hangzhou, China: IEEE, 2018: 133-136.
17	TONSEN M, STEIL J, SUGANO Y, et al. Invisi-bleeye: Mobile eye trac-king using multiple low-resolution cameras and learning-based gaze estimation[C]//Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Hawaii, USA: ACM, 2017, 1(3): 1-21.
18	ZHANG X , SUGANO Y , FRITZ M , et al. Mpiigaze: Real-world dataset and deep appearance-based gaze estimation[J]. IEEE PAMI, 2017, 41 (1): 162- 175.
19	KRAFKA K, KHOSLA A, KELLNHOFER P, et al. Eye tracking for everyone[C]//Proceedings of the IEEE conference on computer vision and pattern recognition(CPVR). Las Vegas, USA: IEEE, 2016: 2176-2184.
20	LIU Y, LEE B S, SLUZEK A, et al. Feasibility analysis of eye typing with a standard webcam[C]//European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016: 254-268.
21	黄君浩, 贺辉. 基于LSTM的眼动行为识别及人机交互应用[J]. 计算机系统应用, 2020, 29 (3): 206- 212.
	HUANG Junhao , HE Hui . Eye movement recognition and its human-computer interaction application based on LSTM[J]. Computer Systems & Applications, 2020, 29 (3): 206- 212.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

功能	眼动行为描述
左单击	左眼单眨眼且闭眼时间超过0.5 s。
右单击	右眼单眨眼且闭眼时间超过0.5 s。
左双击	左眼单眨眼且闭眼时间超过1.0 s。
拖动	左眨眼1.5 s以上, 鼠标为按下状态, 随后鼠标可以移动。再次左眨眼1.5 s以上, 鼠标为松开状态。
滚动	向上眼动时, 触发向上滚动, 眨眼向上眼动可对应滚轮向下滚动。

编号	时间/s	出错次数	速度/(s·次^-1)
1	304	4	12.1
2	196	5	7.8
3	251	5	10.0

方块大小/ cm×cm	时间/s	总点击次数	出错次数	正确率/ %	速度/ (s·次^-1)
2×2	228	45	0	100.0	5.06
2×1	325	60	0	100.0	5.41
2×0.5	301	40	1	97.5	7.52
1×0.5	313	40	0	100.0	7.80
0.5×0.5	721	80	9	88.7	9.01
0.25×0.25	869	80	35	56.2	10.86

输入对象	总测试时间/s	输入字符总数	出错次数	正确率/ %	速度/ (s·字符^-1)
英文字母	1541	193	35	81.8	7.9
中文拼音	948	103	13	89.1	7.9

[1]	邹启杰,李昊宇,张汝波,裴腾达,刘艳. 自主驾驶的人机交互控制[J]. 山东大学学报 (工学版), 2019, 49(2): 23-33.
[2]	张冕,黄颖,梅海艺,郭毓. 基于Kinect的配电作业机器人智能人机交互方法[J]. 山东大学学报 (工学版), 2018, 48(5): 103-108.