Journal of Shandong University(Engineering Science) ›› 2021, Vol. 51 ›› Issue (1): 1-10.doi: 10.6040/j.issn.1672-3961.0.2020.382

• Control Science & Engineering •     Next Articles

Overview of multi-motion vision odometer

Fengyu ZHOU(),Panlong GU*(),Fang WAN,Lei YIN,Jiakai HE   

  1. School of Control Science and Engineering, Shandong University, Jinan 250061, Shandong, China
  • Received:2020-09-15 Online:2021-02-20 Published:2021-03-01
  • Contact: Panlong GU E-mail:zhoufengyu@sdu.edu.cn;gupanlongly@163.com

RichHTML

397

PDF (PC)

618

Abstract:

Multi-motion visual odometry (MVO) was an algorithm for estimating the pose change of dynamic objects in dynamic scenes. It was of great theoretical significance and practical value in autonomous things (AuT). The development process and the latest research progress of multi-motion visual odometer in robot field were reviewed. The important research results of multi-motion visual odometer with the fusion of semantic and geometric features were introduced. Based on the same evaluation criteria and datasets, this research compared several common methods, and prospected the future development direction of multi motion visual odometer.

Key words: visual odometer, MVO, visual simultaneous localization and mapping, dynamic culling, interframe estimation

CLC Number: 

  • TP391.4

Fig.1

MVO computing framework"

Fig.2

Image segmentation effect optimized"

Fig.3

Projection relationship from 3D box to 2D box"

Fig.4

The dataset used in this paper"

Table 1

Positioning accuracy of TUM data set, absolute position error, rmse cm"

子数据集名称 算法名称
ORB-SLAM2 DynaSLAM 文献[12]算法 Dynamic-SLAM MaskFusion
fr2_desk_ps 4.971 0.8 1.873
fr3_sit_half 1.992 1.62 1.461 5.2
fr3_sit_rpy 5.811 3.448
fr3_sit_xyz 0.836 1.3 0.9 0.601 3.1
fr3_walk_half 3.714 2.1 1.99 2.139 10.6
fr3_walk_rpy 10.528 2.999 6.025
fr3_walk_xyz 1.431 1.4 1.39 1.324 10.4

Table 2

Positioning accuracy of TUM data set, relative position error, rmse cm"

子数据集名称 算法名称
ORB-SLAM2 DVO 文献[12]算法 Dynamic-SLAM BaMVO MaskFusion
fr2_desk_ps 2.584 2.96 0.051 1.958 2.96
fr3_sit_half 1.694 10.05 1.57 1.451 5.89 4.1
fr3_sit_rpy 6.495 17.35 4.303 18.72
fr3_sit_xyz 0.948 4.53 1.37 0.998 4.82 4.6
fr3_walk_half 3.795 26.28 2.192 17.38 9.3
fr3_walk_rpy 6.776 40.38 4.48 5.605 35.84
fr3_walk_xyz 1.772 54.01 2.02 1.796 23.26 9.7

Table 3

Positioning accuracy of TUM data set, relative rotation error, rmse cm"

子数据集名称 算法名称
ORB-SLAM2 DVO 文献[12]算法 Dynamic-SLAM BaMVO MaskFusion
fr2_desk_ps 0.891 1.392 0 0.356 1 0.833 1.116 7
fr3_sit_half 0.581 4.464 9 0.921 0 0.551 2.880 4 2.07
fr3_sit_rpy 1.368 6.016 4 0.991 5.983 4
fr3_sit_xyz 0.557 1.498 0 0.509 1 0.613 1.388 5 1.25
fr3_walk_half 0.763 5.217 9 0.666 4.286 3 3.35
fr3_walk_rpy 1.278 7.066 2 0.954 3 1.149 6.339 8
fr3_walk_xyz 1.825 7.666 9 0.541 5 0.598 4.391 1 2.0

Table 4

Test results on KITTI dataset"

KITTI序列 ORB-SLAM2 DynaSLAM Cluster SLAM Cluster Vo 文献[37]
ATE/ cm RRE/ cm RPE/ (°) ATE/ cm RRE/ cm RPE/ (°) ATE/ cm RRE/ cm RPE/ (°) ATE/ cm RRE/ cm RPE/ (°) ATE/ cm
0926_0009 0.91 0.01 1.89 7.51 0.06 2.17 0.92 0.03 2.34 0.79 0.03 2.98 1.14
0926_0013 0.30 0.01 0.94 1.97 0.04 1.41 2.12 0.07 5.50 0.26 0.01 1.16 0.35
0926_0014 0.56 0.01 1.15 5.98 0.09 2.73 0.81 0.03 2.24 0.48 0.01 1.04 0.51
0926_0051 0.37 0.00 1.10 10.95 0.10 1.65 1.19 0.03 1.44 0.81 0.02 2.74 0.76
0926_0101 3.42 0.03 14.27 10.24 0.13 12.29 4.02 0.02 12.43 3.18 0.02 12.78 5.30
0929_0004 0.44 0.01 1.22 2.59 0.02 2.03 1.12 0.02 2.78 0.40 0.02 1.77 0.40
1003_0047 18.87 0.05 28.32 9.31 0.05 6.58 10.21 0.06 8.94 4.79 0.05 6.54 1.03

Fig.5

Accuracy difference between MVO and cluster VO"

1 丁文东, 徐德, 刘希龙, 等. 移动机器人视觉里程计综述[J]. 自动化学报, 2018, 44 (3): 385- 400.
DING Wendong , XU De , LIU Xilong , et al. Overview of mobile robot visual odometer[J]. Acta Automatica Sinica, 2018, 44 (3): 385- 400.
2 CADENA Caser , CARLONE Luca , CARRILLO Henry , et al. Past, present, and future of simultaneous locali-zation and mapping: toward the robust-perception age[J]. IEEE Transactions on Robotics, 2016, 32 (6): 1309- 1332.
doi: 10.1109/TRO.2016.2624754
3 LERREL Pinto, ABHINAV Gupta. Supersizing self-supervision: learning to grasp from 50k tries and 700 robot hours[C]//International Conference on Robotics and Automation (ICRA). Stockholm, Sweden: IEEE, 2016: 3406-3413.
4 张峻宁, 苏群星, 刘鹏远, 等. 一种自适应特征地图匹配的改进VSLAM算法[J]. 自动化学报, 2019, 45 (3): 553- 565.
ZHANG Junning , SU Qunxing , LIU Pengyuan , et al. An improved vSLAM algorithm for adaptive feature map matching[J]. Acta Automatica Sinica, 2019, 45 (3): 553- 565.
5 NAM Dinh Van , KIM Gon-Woo . Robust stereo visual inertial navigation system based on multi-stage outlier removal in dynamic environments[J]. Sensors, 2020, 20 (10): 2922- 2946.
doi: 10.3390/s20102922
6 MATTHIES Larry , STEVEN Shafer . Error mode-lling in stereo navigation[J]. IEEE Journal on Robotics and Automation, 1987, 3 (3): 239- 248.
doi: 10.1109/JRA.1987.1087097
7 BERTA Bescos , JOSE Facil , JAVIER Civera , et al. DynaSLAM: tracking, mapping and inpainting in dynamic scenes[J]. IEEE Robotics & Automation Letters, 2018, 32 (11): 1- 1.
8 RAúL Mur-Artal , TARDOS Juan . ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras[J]. IEEE Transactions on Robotics, 2016, 33 (5): 1255- 1262.
9 HE Kaiming, GEORGIA Gkioxari, DOLLAR Piotr, et al. Mask r-cnn[C]//International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 2961-2969.
10 GEIGER A , LENZ P , STILLER C , et al. Vision meets robotics: the kitti dataset[J]. The International Journal of Robotics Research, 2013, 32 (11): 1231- 1237.
doi: 10.1177/0278364913491297
11 STURM Jrgen, ENGELHARD Nikolas, ENDRES Felix, et al. A benchmark for the evaluation of RGB-D SLAM systems[C]//International Conference on Intelligent Robots and Systems. Vilamoura, Portual: IEEE, 2012: 573-580.
12 ZHAO Lili , LIU Zhili , CHEN Jianwen , et al. A compatible framework for RGB-D SLAM in dynamic scenes[J]. IEEE Access, 2019, 7 (1): 75604- 75614.
13 JOHN Canny . A computational approach to edge detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986, 8 (6): 679- 698.
14 BOUGUET J Y . Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm[J]. Intel Corporation, 2001, 5 (10): 4.
15 KIM Deok-Hwa , KIM John-Hwan . Effective background model-based RGB-D dense visual odometry in a dynamic environment[J]. IEEE Transactions on Robotics, 2016, 32 (6): 1565- 1573.
doi: 10.1109/TRO.2016.2609395
16 ELGAMMAL Ahmed Mahmoud, HARWOOD David, DAVIS L. Non-parametric model for background subtraction[C]//European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2000: 751-767.
17 KERL Christian, STURM Jurgen, CREMERS Daniel. Robust odometry estimation for RGB-D cameras[C]//International Conference on Robotics and Automation (ICRA). Karlsruhe, Germany: IEEE, 2013: 3748-3754.
18 XIAO Linhui , WANG Jinge , QIU Xiaoasong , et al. Dynamic-SLAM: semantic monocular visual localization and mapping based on deep learning in dynamic environment[J]. Robotics and Autonomous Systems, 2019, 117 (1): 1- 16.
19 LIU Wei, ANGUELOV Dragomir, ERHAN Dumitru, et al. SSD: single shot multibox detector[C]//European Conference on Computer Vision (ECCV). Amsterdam, Netherlands: Springer, 2016: 21-37.
20 JUDD Kevin M, GAMMELL Jonathan D, NEWMAN Paul. Multimotion visual odometry (mvo): simultaneous estimation of camera and third-party motions[C]//International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain: IEEE, 2018: 3949-3956.
21 JUDD Kevin Michael , GAMMELL Jonathan D . The Oxford multimotion dataset: multiple SE (3) motions with ground truth[J]. IEEE Robotics and Automation Letters, 2019, 4 (2): 800- 807.
doi: 10.1109/LRA.2019.2892656
22 HUANG Jiahui, YANG Sheng, ZHAO Zishuo, et al. ClusterSLAM: a slam backend for simultaneous rigid body clustering and motion estimation[C]// International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE, 2019: 5875-5884.
23 SOKAL Robert R , MICHENER Charles D . A statistical method for evaluating systematic relationships[J]. Univ Kansas (Sci Bull), 1958, 38 (1): 1409- 1438.
24 PRATIK Agarwal, GIAN Diego Tipaldi, LUCIANO Spinello, et al. Robust map optimization using dynamic covariance scaling[C]//International Conference on Robotics and Automation (ICRA). Karlsruhe, Germany: IEEE, 2013: 62-69.
25 SONG Shuran, YU Fisher, ZENG Andy, et al. Semantic scene completion from a single depth image[C]//Computer Vision and Pattern Recognition (CVPR). Honolulu, Hawaii, USA: IEEE, 2017: 1746-1754.
26 DOSOVITSKIY Alexey, ROS German, CODEVILLA Felipe, et al. CARLA: an open urban driving simulator[C]// Proceeding of 1st Annual Conference on Robot Learning, Hamilton, New Zealand, 2017: 1-16.
27 RVNZ Martin, BUFFIER Maud, AGAPITO Lourdes, et al. Maskfusion: real-time recognition, tracking and reconstruction of multiple moving objects[C]//International Symposium on Mixed and Augmented Reality (ISMAR). München, Germany: IEEE, 2018: 10-20.
28 THOMAS Whelan, STEFAN Leutenegger, RENATO F Salas-Moreno, et al. ElasticFusion: dense SLAM without a pose graph[C]// Robotics: Science and Systems. Rome, Italy: DBIP, 2015: 1-9.
29 RVNZ Martin, AGAPITO Lourdes. Co-fusion: real-time segmentation, tracking and fusion of multiple objects[C]// International Conference on Robotics and Automation (ICRA). Marina Bay Sands, Singapore: IEEE, 2017: 4471-4478.
30 KELLER Maik, DAMIEN Lefloch, MARTIN Lambers, et al. Real-time 3d reconstruction in dynamic scenes using point-based fusion[C]// International Conference on 3D Vision-3DV (3DIMPVT). Washington, USA: IEEE, 2013: 1-8.
31 RALUCA Scona, MARIANO Jaimez, YVAN Petillot, et al. Staticfusion: background reconstruction for dense rgb-d slam in dynamic environments[C]// International Conference on Robotics and Automation (ICRA). Brisbane, Australia: IEEE, 2018: 1-9.
32 HUANG Jiahui, YANG Sheng, MU Taijiang, et al. ClusterVO: clustering moving instances and estimating visual odometry for self and surroundings[C]//Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE, 2020: 2168-2177.
33 REDMON Joseph, ALI Farhadi. YOLO9000: better, faster, stronger[C]//Computer Vision and Pattern Recognition (CVPR). Honolulu, Hawaii: IEEE, 2017: 2168-2177.
34 KRHENBüHL Philipp, KOLTUN Vladlen. Efficient inference in fully connected crfs with gaussian edge potentials[C]//Advances in Neural Information Processing Systems (NIPS). Granada, Spain: Curran Associate, 2011: 109-117.
35 LI Peiliang, QIN Tong, SHEN Shaojie. Stereo vision-based semantic 3d object and ego-motion tracking for autonomous driving[C]// European Conference on Computer Vision (ECCV). Munich, Germany: Springer, 2018: 646-661.
36 HANDA Ankur, THOMAS Whelan, JOHN McDonald, et al. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM[C]// International Conference on Robotics and Automation (ICRA). Miami, USA: IEEE, 2014: 1524-1531.
37 MOUSAVIAN Arsalan, ANGUELOV Dragomir, FLYNN John, et al. 3d bounding box estimation using deep learning and geometry[C]// Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 7074-7082.
38 REN Shaoqing, HE Kaiming, GIRSHICK Ross, et al. Faster r-cnn: towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems (NIPS). Montreal, Canada: Curran Associate, 2015: 91-99.
39 LI Peiliang, CHEN Xiaozhi, SHEN Shaojie. Stereo R-CNN based 3D object detection for autonomous driving[C]//Computer Vision and Pattern Recognition (CVPR). California, USA: IEEE, 2019: 1902-09738.
40 BOLYA Daniel, ZHOU Chong, XIAO Fanyi, et al. YOLACT: real-time instance segmentation[C]// International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE, 2019: 5875-5884.
41 CHEN Hao, SUN Kunyang, TIAN Zhi, et al. Blendmask: top-down meets bottom-up for instance segmentation[C]//Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE, 2020: 2168-2177.
42 Ultralytics. github[EB/OL]. (2020-4-1)[2020-8-13]. https://github.com/ultralytics/yolov5.
43 MIN Erxue , GUO Xifeng , LIU Qiang , et al. A survey of clustering with deep learning: from the perspective of network architecture[J]. IEEE Access, 2018, 6 (1): 39501- 39514.
44 BOWMAN Sean L, NIKOLAY Atanasov, KOSTAS Daniilidis, et al. Probabilistic data association for semantic slam[C]//International Conference on Robo-tics and Automation (ICRA). Marina Bay Sands, Singapore: IEEE, 2017: 1722-1729.
45 POHLEN Tobias, HERMANS Alexander, MATHIAS Markus, et al. Full-resolution residual networks for semantic segmentation in street scenes[C]//Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE, 2017: 4151-4160.
46 WU Dazhong , ROSEN David , WANG Lihui , et al. Cloud-based design and manufacturing: a new paradigm in digital manufacturing and design innovation[J]. Computer-Aided Design, 2015, 59 (1): 1- 14.
47 DU Zhihui, YANG Weiqiang, CHEN Yinong, et al. Design of a robot cloud center[C]// International Symposium on Autonomous Decentralized Systems. Hiroshima, Japan: IEEE, 2011: 269-275.
[1] Baocheng LIU,Yan PIAO,Xuemei SONG. Adaptive fusion target tracking based on joint detection [J]. Journal of Shandong University(Engineering Science), 2020, 50(3): 51-57.
[2] Shengnan ZHANG,Lei WANG,Chunhong CHANG,Benli HAO. Image denoising based on 3D shearlet transform and BM4D [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 83-90.
[3] Yuenan ZHAO,Guiyou CHEN,Chen SUN,Ning LU,Liwei LIAO. Risk assessment method based on spatial hidden danger distribution and motion intention analysis [J]. Journal of Shandong University(Engineering Science), 2020, 50(1): 28-34.
[4] Yuanxi YAO. Analysis of wind power convergence trend quantitation based on sub-scene reconstruction [J]. Journal of Shandong University(Engineering Science), 2019, 49(6): 86-92.
[5] Jiachen WANG,Xianghong TANG,Jianguang LU. Research onfeature selection technology in bearing fault diagnosis [J]. Journal of Shandong University(Engineering Science), 2019, 49(2): 80-87, 95.
[6] Shiguang LIU,Hairong WANG,Jin LIU. Fast 4-points congruent sets for coarse registration of 3D point cloud [J]. Journal of Shandong University(Engineering Science), 2019, 49(2): 1-7.
[7] Jun FAN,Qiaolin YE,Ning YE. Face recognition based on improved prameter-free supervised localitypreserving projections [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 10-16.
[8] Guoxin WANG,Fengdong CHEN,Guodong LIU. Feature extraction method of color pseudo-random coded structured light [J]. Journal of Shandong University(Engineering Science), 2018, 48(5): 55-60.
[9] DOU Tingting, YAO Yuanxi, CHEN Peng, LU Deng. Arc modeling and practical simulation application based on ATP-EMTP [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(4): 102-108.
[10] ZHANG Xianhong, ZHANG Chunrui. Image enhancement algorithm based on six dimensional feedforward neural network model [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(4): 10-19.
[11] DING Xiaoling, ZHAO Qiang, LI Yibin, MA Xin. Modified target recognition algorithm based on template matching [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(2): 1-7.
[12] ZHAO Ye, HE Xiao, ZHOU Donghua. On least squares fault estimation with incorrect measurement noise statistics [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(5): 254-262.
[13] LIU Yan, LI Youjun, CHEN Meng. Research on the classification of resting state EEG signal between depression patients and normal controls by EMD and SVM methods [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(3): 21-26.
[14] JIN Peipei, SUN Fengrong, LIU Fanglei, YAO Guihua. Cardiac cycle estimation of echocardiography with speckle tracking [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(2): 94-99.
[15] WANG Bin, CHANG Faliang, LIU Chunsheng. Traffic sign classification based on multi-feature fusion [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(4): 34-40.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] WANG Su-yu,<\sup>,AI Xing<\sup>,ZHAO Jun<\sup>,LI Zuo-li<\sup>,LIU Zeng-wen<\sup> . Milling force prediction model for highspeed end milling 3Cr2Mo steel[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 1 -5 .
[2] ZHANG Yong-hua,WANG An-ling,LIU Fu-ping . The reflected phase angle of low frequent inhomogeneous[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 22 -25 .
[3] LI Kan . Empolder and implement of the embedded weld control system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 37 -41 .
[4] SHI Lai-shun,WAN Zhong-yi . Synthesis and performance evaluation of a novel betaine-type asphalt emulsifier[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 112 -115 .
[5] KONG Xiang-zhen,LIU Yan-jun,WANG Yong,ZHAO Xiu-hua . Compensation and simulation for the deadband of the pneumatic proportional valve[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 99 -102 .
[6] LAI Xiang . The global domain of attraction for a kind of MKdV equations[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 87 -92 .
[7] YU Jia yuan1, TIAN Jin ting1, ZHU Qiang zhong2. Computational intelligence and its application in psychology[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 1 -5 .
[8] LI Liang, LUO Qiming, CHEN Enhong. Graph-based ranking model for object-level search
[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 15 -21 .
[9] CHEN Rui, LI Hongwei, TIAN Jing. The relationship between the number of magnetic poles and the bearing capacity of radial magnetic bearing[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(2): 81 -85 .
[10] WANG Bo,WANG Ning-sheng . Automatic generation and combinatory optimization of disassembly sequence for mechanical-electric assembly[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 52 -57 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd