Journal of Shandong University(Engineering Science) ›› 2021, Vol. 51 ›› Issue (2): 1-8.doi: 10.6040/j.issn.1672-3961.0.2020.346

• Machine Learning & Data Mining •     Next Articles

Eye tracking in human-computer interaction control

Hui HE1,2(),Junhao HUANG1   

  1. 1. Advanced Institute of Natural Sciences, Beijing Normal University at Zhuhai, Zhuhai 519087, Guangdong, China
    2. Intelligent Engineering and Educational Application Research Center, Beijing Normal University, Zhuhai 519087, Guangdong, China
  • Received:2020-08-24 Online:2021-04-20 Published:2021-04-16

Abstract:

To actualize the simple and low-cost eye-tracking based human-computer interaction, an exact interaction method based on the visual directions estimation and eye tracking with webcam videos was proposed. A simple and fast convolution neural network model was used to roughly estimate the user′s viewpoints on the screen. And then an accurate human-computer interaction method was proposed on the basis of the eye movements recognition and sight line tracking results. To verify the effectiveness of the method, the key operations of eye mouse and eye typing were developed. The test results show that the proposed method enabled users to achieve eye tracking and to actualize most precise human-computer interactions with only one common monocular camera, which was expected to completely replace the mouse and keyboard hardwares.

Key words: eye tracking, human computer interaction, eye mouse, eye typing

CLC Number: 

  • TP319

Fig.1

Hardware equipment"

Fig.2

Monocular images after processed"

Fig.3

The process of eye gesture recognition"

Fig.4

Structure diagram of the CNN eye tracking model"

Table 1

Function of eye cursor"

功能 眼动行为描述
左单击 左眼单眨眼且闭眼时间超过0.5 s。
右单击 右眼单眨眼且闭眼时间超过0.5 s。
左双击 左眼单眨眼且闭眼时间超过1.0 s。
拖动 左眨眼1.5 s以上, 鼠标为按下状态, 随后鼠标可以移动。再次左眨眼1.5 s以上, 鼠标为松开状态。
滚动 向上眼动时, 触发向上滚动, 眨眼向上眼动可对应滚轮向下滚动。

Fig.5

Staring motion diagram"

Fig.6

Diagram of relative center movement"

Fig.7

Remote-controlled movement schematics"

Fig.8

Coding based on eye tracking"

Fig.9

Valid input area of code elements"

Fig.10

Input process of single key bit coding"

Fig.11

Encoding process of alphabetic key bit code"

Fig.12

Key coding of common keyboard"

Fig.13

Trajectory reference for line of sight calibration"

Fig.14

Results of eye tracking after calibration"

Fig.15

Test results of eye movement mouse"

Table 2

Test results of mouse click on 2cm×0.5cm square"

编号 时间/s 出错次数 速度/(s·次-1)
1 304 4 12.1
2 196 5 7.8
3 251 5 10.0

Table 3

Accuracy test results of mouse click on different size blocks"

方块大小/
cm×cm
时间/s 总点击
次数
出错
次数
正确率/
%
速度/
(s·次-1)
2×2 228 45 0 100.0 5.06
2×1 325 60 0 100.0 5.41
2×0.5 301 40 1 97.5 7.52
1×0.5 313 40 0 100.0 7.80
0.5×0.5 721 80 9 88.7 9.01
0.25×0.25 869 80 35 56.2 10.86

Fig.16

Eye typing test"

Table 4

Test results of typing efficiency"

输入对象 总测试
时间/s
输入字符
总数
出错
次数
正确率/
%
速度/
(s·字符-1)
英文字母 1541 193 35 81.8 7.9
中文拼音 948 103 13 89.1 7.9

Fig.17

Test of web browsing"

1 王向军, 蔡方方, 刘峰, 等. 非接触动态实时视线跟踪技术[J]. 计算机科学与探索, 2015, 9 (3): 266- 278.
WANG Xiangjun , CAI Fangfang , LIU Feng , et al. Non-contact dynamic real-time eye tracking technology[J]. Journal of Frontiers of Computer Science and Technology, 2015, 9 (3): 266- 278.
2 许洁, 王豪龙. 阅读行为眼动跟踪研究综述[J]. 出版科学, 2020, (2): 52- 66.
doi: 10.3969/j.issn.1009-5853.2020.02.009
XU Jie , WANG Haolong . A review of studies on behavioral eye tracking in reading[J]. Publishing Journal, 2020, (2): 52- 66.
doi: 10.3969/j.issn.1009-5853.2020.02.009
3 尤洋, 王以宁, 张海. 教学视频终端显示空间对学习者影响的眼动研究[J]. 中国电化教育, 2019, (8): 123- 129.
doi: 10.3969/j.issn.1006-9860.2019.08.018
YOU Yang , WANG Yining , ZHANG Hai . An eye movement study on the influence of teaching video terminal display on learners[J]. China Educational Technology, 2019, (8): 123- 129.
doi: 10.3969/j.issn.1006-9860.2019.08.018
4 靳慧斌, 刘亚威, 穆小萌. 基于眼动追踪的航空公司官网可用性评估[J]. 包装工程, 2018, 39 (10): 173- 176.
JIN Huibin , LIU Yawei , MU Xiaomeng . Evaluation of the usability of airline official website based on eye tracking[J]. Packaging Engineering, 2018, 39 (10): 173- 176.
5 胡晓红, 王红, 任衍具. 基于眼动技术的互联网广告效果研究[J]. 计算机应用研究, 2018, 35 (5): 1345- 1349.
doi: 10.3969/j.issn.1001-3695.2018.05.014
HU Xiaohong , WANG Hong , REN Yanju . Research on network advertisement effect based on eye tracking technology[J]. Application Research of Computers, 2018, 35 (5): 1345- 1349.
doi: 10.3969/j.issn.1001-3695.2018.05.014
6 WANLUK N, VISITSATTAPONGSE S, JUHONG A, et al. Smart wheelchair based on eye tracking[C]//2016 9th Biomedical Engineering International Conference (BMEi-CON). Laung Prabang, Laos: IEEE, 2016: 1-4.
7 王增磊, 张树生, 白晓亮. 面向协同装配的混合现实手眼交互技术研究[J]. 计算机应用与软件, 2019, 36 (1): 257- 264.
WANG Zenglei , ZHANG Shusheng , BAI Xiaoliang . Mixed reality hand-eye interaction technology for collaborative assembly[J]. Computer Applications and Software, 2019, 36 (1): 257- 264.
8 PFEUFFER K, VIDAL M, TURNER J, et al. Pursuit calibration: Making gaze calibration less tedious and more flexible[C]//Proceedings of the 26th annual ACM symposium on User interface software and technology(UIST). Andrews Scotland, the United Kingdom: ACM, 2013: 3147-3150.
9 HOSSAIN Z, SHUVO M M H, SARKER P. Hardware and software implementation of real time electro-oculogram (EOG) acquisition system to control computer cursor with eyeball movement[C]//2017 4th International Conference on Advances in Electrical Engineering (ICAEE). Dhaka, Bangladesh: IEEE, 2017: 132-137.
10 CHAKRABORTY T, SARCAR S, SAMANTA D. Design and evaluation of a dwell-free eye typing technique[C]//Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems. Toronto, Canada: ACM, 2014: 1573-1578.
11 程时伟, 沈哓权, 孙凌云, 等. 多用户眼动跟踪数据的可视化共享与协同交互[J]. 软件学报, 2019, 30 (10): 3037- 3053.
CHENG Shiwei , SHEN Xiaoquan , SUN Lingyun , et al. Shared visualization and collaborative interaction based on multiple user eye tracking data[J]. Journal of Software, 2019, 30 (10): 3037- 3053.
12 朱琳, 王圣凯, 袁伟舜, 等. 眼动控制的交互式地图设计[J]. 武汉大学学报(信息科学版), 2020, 45 (5): 736- 743.
ZHU Lin , WANG Shengkai , YUAN Weishun , et al. An interactive map based on gaze control[J]. Geomatics and Information Science of Wuhan University, 2020, 45 (5): 736- 743.
13 朱麒文, 闫隆鑫, 张若兰, 等. 基于视线追踪的眼控鼠标设计[J]. 电子器件, 2016, 39 (2): 235- 241.
ZHU Qiwen , YAN Longxin , ZHANG Ruolan , et al. Design of eye-controlled mouse based on eye tracking technology[J]. Chinese Journal of Electron Devices, 2016, 39 (2): 235- 241.
14 郜东瑞, 甘玉龙, 李鹏霄, 等. 基于眼电的智能输入系统研究[J]. 中国生物医学工程学报, 2015, 34 (6): 662- 669.
doi: 10.3969/j.issn.0258-8021.2015.06.004
GAO Dongrui , GAN Yulong , LI Pengxiao , et al. Research of intelligent speller system based on EOG[J]. Chinese Journal of Biomedical Engineering, 2015, 34 (6): 662- 669.
doi: 10.3969/j.issn.0258-8021.2015.06.004
15 CAZZATO D , DOMINIO F , MANDUCHI R , et al. Real-time gaze estimation via pupil center tracking[J]. Paladyn Journal of Behavioral Robotics, 2018, 9 (1): 6- 18.
doi: 10.1515/pjbr-2018-0002
16 ZHENG C, USAGAWA T. A Rapid Webcam-Based Eye Tracking Method for Human Computer Interaction[C]//2018 International Conference on Control, Automation and Information Sciences (ICCAIS). Hangzhou, China: IEEE, 2018: 133-136.
17 TONSEN M, STEIL J, SUGANO Y, et al. Invisi-bleeye: Mobile eye trac-king using multiple low-resolution cameras and learning-based gaze estimation[C]//Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Hawaii, USA: ACM, 2017, 1(3): 1-21.
18 ZHANG X , SUGANO Y , FRITZ M , et al. Mpiigaze: Real-world dataset and deep appearance-based gaze estimation[J]. IEEE PAMI, 2017, 41 (1): 162- 175.
19 KRAFKA K, KHOSLA A, KELLNHOFER P, et al. Eye tracking for everyone[C]//Proceedings of the IEEE conference on computer vision and pattern recognition(CPVR). Las Vegas, USA: IEEE, 2016: 2176-2184.
20 LIU Y, LEE B S, SLUZEK A, et al. Feasibility analysis of eye typing with a standard webcam[C]//European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016: 254-268.
21 黄君浩, 贺辉. 基于LSTM的眼动行为识别及人机交互应用[J]. 计算机系统应用, 2020, 29 (3): 206- 212.
HUANG Junhao , HE Hui . Eye movement recognition and its human-computer interaction application based on LSTM[J]. Computer Systems & Applications, 2020, 29 (3): 206- 212.
[1] An ZHU,Chu XU. Method for super-resolution using parallel interlaced sampling [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 10-16,26.
[2] Yanghao ZHOU,Yifan LIU,Li LI. An automatic reading method for pointer meter [J]. Journal of Shandong University(Engineering Science), 2019, 49(4): 1-7.
[3] Yan WANG,Xiaojing SHEN. Research and application of marine metadata based on life cycle model [J]. Journal of Shandong University(Engineering Science), 2019, 49(3): 15-21.
[4] Mian ZHANG,Ying HUANG,Haiyi MEI,Yu GUO. Intelligent interaction method for power distribution robot based on Kinect [J]. Journal of Shandong University(Engineering Science), 2018, 48(5): 103-108.
[5] MOU Chunqian, TANG Yan, HU Jinge. A new 3D model retrieval method based on manifold ranking [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(4): 19-24.
[6] MOU Chunqian, TANG Yan. A novel 3D model retrieval method fusing global and local information [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(6): 48-53.
[7] CHEN Ya-fei,LEI Hong,CHEN Yin-xing,FENG Jin-jun . Application of PSPICE CAD in designing a highvoltage high power switching power supply [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(1): 119-124 .
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] LI Ke,LIU Chang-chun,LI Tong-lei . Medical registration approach using improved maximization of mutual information[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 107 -110 .
[2] LIU Zhongguo,ZHANG Xiaojing,LIU Boqiang,LIU Changchun, . The development of ultrasonic characterization of the biological tissue elasticity[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(3): 34 -38 .
[3] LI Kan . Empolder and implement of the embedded weld control system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 37 -41 .
[4] SUN Dianzhu, ZHU Changzhi, LI Yanrui. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 84 -86 .
[5] QU Yan-peng,CHEN Song-ying,LI Chun-feng,WANG Xiao-peng,TENG Shu-ge . [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(4): 16 -20 .
[6] YUE Yuan-Zheng. Relaxation in glasses far from equilibrium[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 1 -20 .
[7] XIA Bin,ZHANG Lian-jun . Energy comparison-based TOA estimation algorithm for the DS-CDMA UWB system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(1): 70 -73 .
[8] XU Xiaodan, DUAN Zhengjie, CHEN Zhongyu. The sentiment mining method based on extended sentiment dictionary and integrated features[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2014, 44(6): 15 -18 .
[9] SHI Lai-shun,WAN Zhong-yi . Synthesis and performance evaluation of a novel betaine-type asphalt emulsifier[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 112 -115 .
[10] HU Tian-liang,LI Peng,ZHANG Cheng-rui,ZUO Yi . Design of a QEP decode counter based on VHDL[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(3): 10 -13 .