Journal of Shandong University(Engineering Science) ›› 2021, Vol. 51 ›› Issue (2): 1-8.doi: 10.6040/j.issn.1672-3961.0.2020.346

• Machine Learning & Data Mining •     Next Articles

Eye tracking in human-computer interaction control

Hui HE1,2(),Junhao HUANG1   

  1. 1. Advanced Institute of Natural Sciences, Beijing Normal University at Zhuhai, Zhuhai 519087, Guangdong, China
    2. Intelligent Engineering and Educational Application Research Center, Beijing Normal University, Zhuhai 519087, Guangdong, China
  • Received:2020-08-24 Online:2021-04-20 Published:2021-04-16

RichHTML

117

PDF (PC)

3262

Abstract:

To actualize the simple and low-cost eye-tracking based human-computer interaction, an exact interaction method based on the visual directions estimation and eye tracking with webcam videos was proposed. A simple and fast convolution neural network model was used to roughly estimate the user′s viewpoints on the screen. And then an accurate human-computer interaction method was proposed on the basis of the eye movements recognition and sight line tracking results. To verify the effectiveness of the method, the key operations of eye mouse and eye typing were developed. The test results show that the proposed method enabled users to achieve eye tracking and to actualize most precise human-computer interactions with only one common monocular camera, which was expected to completely replace the mouse and keyboard hardwares.

Key words: eye tracking, human computer interaction, eye mouse, eye typing

CLC Number: 

  • TP319

Fig.1

Hardware equipment"

Fig.2

Monocular images after processed"

Fig.3

The process of eye gesture recognition"

Fig.4

Structure diagram of the CNN eye tracking model"

Table 1

Function of eye cursor"

功能 眼动行为描述
左单击 左眼单眨眼且闭眼时间超过0.5 s。
右单击 右眼单眨眼且闭眼时间超过0.5 s。
左双击 左眼单眨眼且闭眼时间超过1.0 s。
拖动 左眨眼1.5 s以上, 鼠标为按下状态, 随后鼠标可以移动。再次左眨眼1.5 s以上, 鼠标为松开状态。
滚动 向上眼动时, 触发向上滚动, 眨眼向上眼动可对应滚轮向下滚动。

Fig.5

Staring motion diagram"

Fig.6

Diagram of relative center movement"

Fig.7

Remote-controlled movement schematics"

Fig.8

Coding based on eye tracking"

Fig.9

Valid input area of code elements"

Fig.10

Input process of single key bit coding"

Fig.11

Encoding process of alphabetic key bit code"

Fig.12

Key coding of common keyboard"

Fig.13

Trajectory reference for line of sight calibration"

Fig.14

Results of eye tracking after calibration"

Fig.15

Test results of eye movement mouse"

Table 2

Test results of mouse click on 2cm×0.5cm square"

编号 时间/s 出错次数 速度/(s·次-1)
1 304 4 12.1
2 196 5 7.8
3 251 5 10.0

Table 3

Accuracy test results of mouse click on different size blocks"

方块大小/
cm×cm		 时间/s 总点击
次数		 出错
次数		 正确率/
%		 速度/
(s·次-1)
2×2 228 45 0 100.0 5.06
2×1 325 60 0 100.0 5.41
2×0.5 301 40 1 97.5 7.52
1×0.5 313 40 0 100.0 7.80
0.5×0.5 721 80 9 88.7 9.01
0.25×0.25 869 80 35 56.2 10.86

Fig.16

Eye typing test"

Table 4

Test results of typing efficiency"

输入对象 总测试
时间/s		 输入字符
总数		 出错
次数		 正确率/
%		 速度/
(s·字符-1)
英文字母 1541 193 35 81.8 7.9
中文拼音 948 103 13 89.1 7.9

Fig.17

Test of web browsing"

1 王向军, 蔡方方, 刘峰, 等. 非接触动态实时视线跟踪技术[J]. 计算机科学与探索, 2015, 9 (3): 266- 278.
WANG Xiangjun , CAI Fangfang , LIU Feng , et al. Non-contact dynamic real-time eye tracking technology[J]. Journal of Frontiers of Computer Science and Technology, 2015, 9 (3): 266- 278.
2 许洁, 王豪龙. 阅读行为眼动跟踪研究综述[J]. 出版科学, 2020, (2): 52- 66.
doi: 10.3969/j.issn.1009-5853.2020.02.009
XU Jie , WANG Haolong . A review of studies on behavioral eye tracking in reading[J]. Publishing Journal, 2020, (2): 52- 66.
doi: 10.3969/j.issn.1009-5853.2020.02.009
3 尤洋, 王以宁, 张海. 教学视频终端显示空间对学习者影响的眼动研究[J]. 中国电化教育, 2019, (8): 123- 129.
doi: 10.3969/j.issn.1006-9860.2019.08.018
YOU Yang , WANG Yining , ZHANG Hai . An eye movement study on the influence of teaching video terminal display on learners[J]. China Educational Technology, 2019, (8): 123- 129.
doi: 10.3969/j.issn.1006-9860.2019.08.018
4 靳慧斌, 刘亚威, 穆小萌. 基于眼动追踪的航空公司官网可用性评估[J]. 包装工程, 2018, 39 (10): 173- 176.
JIN Huibin , LIU Yawei , MU Xiaomeng . Evaluation of the usability of airline official website based on eye tracking[J]. Packaging Engineering, 2018, 39 (10): 173- 176.
5 胡晓红, 王红, 任衍具. 基于眼动技术的互联网广告效果研究[J]. 计算机应用研究, 2018, 35 (5): 1345- 1349.
doi: 10.3969/j.issn.1001-3695.2018.05.014
HU Xiaohong , WANG Hong , REN Yanju . Research on network advertisement effect based on eye tracking technology[J]. Application Research of Computers, 2018, 35 (5): 1345- 1349.
doi: 10.3969/j.issn.1001-3695.2018.05.014
6 WANLUK N, VISITSATTAPONGSE S, JUHONG A, et al. Smart wheelchair based on eye tracking[C]//2016 9th Biomedical Engineering International Conference (BMEi-CON). Laung Prabang, Laos: IEEE, 2016: 1-4.
7 王增磊, 张树生, 白晓亮. 面向协同装配的混合现实手眼交互技术研究[J]. 计算机应用与软件, 2019, 36 (1): 257- 264.
WANG Zenglei , ZHANG Shusheng , BAI Xiaoliang . Mixed reality hand-eye interaction technology for collaborative assembly[J]. Computer Applications and Software, 2019, 36 (1): 257- 264.
8 PFEUFFER K, VIDAL M, TURNER J, et al. Pursuit calibration: Making gaze calibration less tedious and more flexible[C]//Proceedings of the 26th annual ACM symposium on User interface software and technology(UIST). Andrews Scotland, the United Kingdom: ACM, 2013: 3147-3150.
9 HOSSAIN Z, SHUVO M M H, SARKER P. Hardware and software implementation of real time electro-oculogram (EOG) acquisition system to control computer cursor with eyeball movement[C]//2017 4th International Conference on Advances in Electrical Engineering (ICAEE). Dhaka, Bangladesh: IEEE, 2017: 132-137.
10 CHAKRABORTY T, SARCAR S, SAMANTA D. Design and evaluation of a dwell-free eye typing technique[C]//Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems. Toronto, Canada: ACM, 2014: 1573-1578.
11 程时伟, 沈哓权, 孙凌云, 等. 多用户眼动跟踪数据的可视化共享与协同交互[J]. 软件学报, 2019, 30 (10): 3037- 3053.
CHENG Shiwei , SHEN Xiaoquan , SUN Lingyun , et al. Shared visualization and collaborative interaction based on multiple user eye tracking data[J]. Journal of Software, 2019, 30 (10): 3037- 3053.
12 朱琳, 王圣凯, 袁伟舜, 等. 眼动控制的交互式地图设计[J]. 武汉大学学报(信息科学版), 2020, 45 (5): 736- 743.
ZHU Lin , WANG Shengkai , YUAN Weishun , et al. An interactive map based on gaze control[J]. Geomatics and Information Science of Wuhan University, 2020, 45 (5): 736- 743.
13 朱麒文, 闫隆鑫, 张若兰, 等. 基于视线追踪的眼控鼠标设计[J]. 电子器件, 2016, 39 (2): 235- 241.
ZHU Qiwen , YAN Longxin , ZHANG Ruolan , et al. Design of eye-controlled mouse based on eye tracking technology[J]. Chinese Journal of Electron Devices, 2016, 39 (2): 235- 241.
14 郜东瑞, 甘玉龙, 李鹏霄, 等. 基于眼电的智能输入系统研究[J]. 中国生物医学工程学报, 2015, 34 (6): 662- 669.
doi: 10.3969/j.issn.0258-8021.2015.06.004
GAO Dongrui , GAN Yulong , LI Pengxiao , et al. Research of intelligent speller system based on EOG[J]. Chinese Journal of Biomedical Engineering, 2015, 34 (6): 662- 669.
doi: 10.3969/j.issn.0258-8021.2015.06.004
15 CAZZATO D , DOMINIO F , MANDUCHI R , et al. Real-time gaze estimation via pupil center tracking[J]. Paladyn Journal of Behavioral Robotics, 2018, 9 (1): 6- 18.
doi: 10.1515/pjbr-2018-0002
16 ZHENG C, USAGAWA T. A Rapid Webcam-Based Eye Tracking Method for Human Computer Interaction[C]//2018 International Conference on Control, Automation and Information Sciences (ICCAIS). Hangzhou, China: IEEE, 2018: 133-136.
17 TONSEN M, STEIL J, SUGANO Y, et al. Invisi-bleeye: Mobile eye trac-king using multiple low-resolution cameras and learning-based gaze estimation[C]//Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Hawaii, USA: ACM, 2017, 1(3): 1-21.
18 ZHANG X , SUGANO Y , FRITZ M , et al. Mpiigaze: Real-world dataset and deep appearance-based gaze estimation[J]. IEEE PAMI, 2017, 41 (1): 162- 175.
19 KRAFKA K, KHOSLA A, KELLNHOFER P, et al. Eye tracking for everyone[C]//Proceedings of the IEEE conference on computer vision and pattern recognition(CPVR). Las Vegas, USA: IEEE, 2016: 2176-2184.
20 LIU Y, LEE B S, SLUZEK A, et al. Feasibility analysis of eye typing with a standard webcam[C]//European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016: 254-268.
21 黄君浩, 贺辉. 基于LSTM的眼动行为识别及人机交互应用[J]. 计算机系统应用, 2020, 29 (3): 206- 212.
HUANG Junhao , HE Hui . Eye movement recognition and its human-computer interaction application based on LSTM[J]. Computer Systems & Applications, 2020, 29 (3): 206- 212.
[1] An ZHU, Chu XU. Method for super-resolution using parallel interlaced sampling [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 10-16.
[2] Yanghao ZHOU,Yifan LIU,Li LI. An automatic reading method for pointer meter [J]. Journal of Shandong University(Engineering Science), 2019, 49(4): 1-7.
[3] Yan WANG,Xiaojing SHEN. Research and application of marine metadata based on life cycle model [J]. Journal of Shandong University(Engineering Science), 2019, 49(3): 15-21.
[4] Mian ZHANG,Ying HUANG,Haiyi MEI,Yu GUO. Intelligent interaction method for power distribution robot based on Kinect [J]. Journal of Shandong University(Engineering Science), 2018, 48(5): 103-108.
[5] MOU Chunqian, TANG Yan, HU Jinge. A new 3D model retrieval method based on manifold ranking [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(4): 19-24.
[6] MOU Chunqian, TANG Yan. A novel 3D model retrieval method fusing global and local information [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(6): 48-53.
[7] CHEN Ya-fei,LEI Hong,CHEN Yin-xing,FENG Jin-jun . Application of PSPICE CAD in designing a highvoltage high power switching power supply [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(1): 119-124 .
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] ZHANG Yong-hua,WANG An-ling,LIU Fu-ping . The reflected phase angle of low frequent inhomogeneous[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 22 -25 .
[2] LI Kan . Empolder and implement of the embedded weld control system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 37 -41 .
[3] SHI Lai-shun,WAN Zhong-yi . Synthesis and performance evaluation of a novel betaine-type asphalt emulsifier[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 112 -115 .
[4] KONG Xiang-zhen,LIU Yan-jun,WANG Yong,ZHAO Xiu-hua . Compensation and simulation for the deadband of the pneumatic proportional valve[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 99 -102 .
[5] LAI Xiang . The global domain of attraction for a kind of MKdV equations[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 87 -92 .
[6] YU Jia yuan1, TIAN Jin ting1, ZHU Qiang zhong2. Computational intelligence and its application in psychology[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 1 -5 .
[7] CHEN Rui, LI Hongwei, TIAN Jing. The relationship between the number of magnetic poles and the bearing capacity of radial magnetic bearing[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(2): 81 -85 .
[8] WANG Bo,WANG Ning-sheng . Automatic generation and combinatory optimization of disassembly sequence for mechanical-electric assembly[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 52 -57 .
[9] JI Tao,GAO Xu/sup>,SUN Tong-jing,XUE Yong-duan/sup>,XU Bing-yin/sup> . Characteristic analysis of fault generated traveling waves in 10 Kv automatic blocking and continuous power transmission lines[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 111 -116 .
[10] QIN Tong, SUN Fengrong*, WANG Limei, WANG Qinghao, LI Xincai. 3D surface reconstruction using the shape based interpolation guided by maximal discs[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(3): 1 -5 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd