A novel vision-based multi-functional sensor for normality and position measurements in precise robotic manufacturing

Cobots play an essential role in the fourth industrial revolution and the automation of complex manufacturing processes. However, cobots still face challenges in achieving high precision, which obstructs their usage in precise applications such as the aerospace industry. Nonetheless, advances in perception systems unlock new cobot manufacturing capabilities. This paper presents a novel multi-functional sensor that combines visual and tactile feedback using a single optical sensor, featuring a moving gate mechanism. This work also marks the first integration of Vision-Based Tactile Sensing (VBTS) into a robotic machining end-effector. The sensor provides vision-based tactile perception capabilities for precise normality control and exteroceptive perception for robot localization and positioning. Its performance is experimentally demonstrated in a precise robotic deburring application, where the sensor achieves the high-precision requirements of the aerospace industry with a mean normality error of 0.13° and a mean positioning error of 0.2 mm. These results open a new paradigm for using vision-based sensing for precise robotic manufacturing, which surpasses conventional approaches in terms of precision, weight, size, and cost-effectiveness.

[1]
Welber I.
Factory of the future
IEEE Control Syst Mag, 7 (2) (1987), pp. 20-22, 10.1109/MCS.1987.1105295
[2]
Huang X., Halwani M., Muthusamy R., Ayyad A., Swart D., Seneviratne L., et al.
Real-time grasping strategies using event camera
J Intell Manuf, 33 (2) (2022), pp. 593-615, 10.1007/s10845-021-01887-9
[3]
Perez-Vidal C., Gracia L., de Paco J.M., Wirkus M., Azorin J.M., de Gea J.
Automation of product packaging for industrial applications
Int J Comput Integr Manuf, 31 (2) (2018), pp. 129-137, 10.1080/0951192X.2017.1369165
[4]
Oztemel E., Gursev S.
Literature review of industry 4.0 and related technologies
J Intell Manuf, 31 (1) (2020), pp. 127-182, 10.1007/s10845-018-1433-8
[5]
Verl A., Valente A., Melkote S., Brecher C., Ozturk E., Tunc L.T.
Robots in machining
CIRP Ann, 68 (2) (2019), pp. 799-822, 10.1016/j.cirp.2019.05.009
URL https://www.sciencedirect.com/science/article/pii/S0007850619301660
[6]
Eguti C.C.A., Trabasso L.G.
Design of a robotic orbital driller for assembling aircraft structures
Mechatronics, 24 (5) (2014), pp. 533-545, 10.1016/j.mechatronics.2014.06.007
URL https://www.sciencedirect.com/science/article/pii/S0957415814000968
[7]
Karim A., Verl A.
Challenges and obstacles in robot-machining
IEEE ISR 2013 (2013), 10.1109/ISR.2013.6695731
[8]
Chen Y., Dong F.
Robot machining: Recent development and future research issues
Int J Adv Manuf Technol, 66 (9) (2013), pp. 1489-1497, 10.1007/s00170-012-4433-4
[9]
Chen D., Yuan P., Wang T., Cai Y., Tang H.
A normal sensor calibration method based on an extended Kalman filter for robotic drilling
Sensors (Switzerland), 18 (10) (2018), 10.3390/s18103485
[10]
Yu L., Bi Q., Ji Y., Fan Y., Huang N., Wang Y.
Vision based in-process inspection for countersink in automated drilling and riveting
Precis Eng, 58 (2019), pp. 35-46, 10.1016/j.precisioneng.2019.05.002
URL https://www.sciencedirect.com/science/article/pii/S0141635919300613
[11]
Yuan P., Wang Q., Wang T., Wang C., Song B.
Surface normal measurement in the end effector of a drilling robot for aviation
Proceedings - IEEE international conference on robotics and automation, IEEE (2014), pp. 4481-4486, 10.1109/ICRA.2014.6907513
[12]
Frommknecht A., Kuehnle J., Effenberger I., Pidan S.
Multi-sensor measurement system for robotic drilling
Robot Comput-Integr Manuf, 47 (2017), 10.1016/j.rcim.2017.01.002
[13]
Santos K.R.D.S., De Carvalho G.M., Tricarico R.T., Ferreira L.F.L., Villani E., Suterio R.
Evaluation of perpendicularity methods for a robotic end effector from aircraft industry
2018 13th IEEE international conference on industry applications, INDUSCON 2018 - Proceedings (2019), pp. 1373-1380, 10.1109/INDUSCON.2018.8627218
[14]
Yu L., Zhang Y., Bi Q., Wang Y.
Research on surface normal measurement and adjustment in aircraft assembly
Precis Eng, 50 (2017), pp. 482-493, 10.1016/j.precisioneng.2017.07.004
[15]
Zhang Y., Bi Q., Yu L., Wang Y.
Online adaptive measurement and adjustment for flexible part during high precision drilling process
Int J Adv Manuf Technol, 89 (9) (2017), pp. 3579-3599, 10.1007/s00170-016-9274-0
[16]
Lin M., Yuan P., Tan H., Liu Y., Zhu Q., Li Y.
Improvements of robot positioning accuracy and drilling perpendicularity for autonomous drilling robot system
2015 IEEE international conference on robotics and biomimetics, IEEE-ROBIO 2015, IEEE (2015), pp. 1483-1488, 10.1109/ROBIO.2015.7418980
[17]
Yuan P., Lai T., Li Y., Han W., Lin M., Zhu Q., et al.
The attitude adjustment algorithm in drilling end-effector for aviation
Adv Mech Eng, 8 (1) (2016), pp. 1-9, 10.1177/1687814016629348
[18]
Shaomin L., Deyuan Z., yanqiang L., chunjian L., Hui T., guang M.
A self-adaption normal direction and active variable stiffness low-frequency vibration-assisted system for curved surface drilling
Precis Eng, 64 (2020), pp. 307-318, 10.1016/j.precisioneng.2020.04.017
URL https://www.sciencedirect.com/science/article/pii/S0141635920302233
[19]
Xiao R., Xu Y., Hou Z., Chen C., Chen S.
An automatic calibration algorithm for laser vision sensor in robotic autonomous welding system
J Intell Manuf, 33 (5) (2022), pp. 1419-1432, 10.1007/s10845-020-01726-3
[20]
Ayyad A., Halwani M., Swart D., Muthusamy R., Almaskari F., Zweiri Y.
Neuromorphic vision based control for the precise positioning of robotic drilling systems
Robot Comput-Integr Manuf, 79 (2023), Article 102419, 10.1016/j.rcim.2022.102419
URL https://www.sciencedirect.com/science/article/pii/S0736584522001041
[21]
Furtado L.F.F., Coracini G.K., Villani E., Trabasso L.G.
Comparative Study Between Two Methods for Perpendicularity Corrections in Robotic Manipulators
21st international congress of mechanical engineering - COBEM, vol. 5 (2011), pp. 1194-1200
Google Scholar
[22]
Tian W., Zhou W., Zhau W., Liao W., Zeng Y.
Auto-normalization algorithm for robotic precision drilling system in aircraft component assembly
Chin J Aeronaut, 26 (2) (2013), pp. 495-500, 10.1016/j.cja.2013.02.029
[23]
Gao Y., Wu D., Nan C., Chen K.
Normal direction measurement in robotic drilling and precision calculation
Int J Adv Manuf Technol, 76 (5–8) (2015), pp. 1311-1318, 10.1007/s00170-014-6320-7
[24]
Gao Y., Wu D., Dong Y., Ma X., Chen K.
The method of aiming towards the normal direction for robotic drilling
Int J Precis Eng Manuf, 18 (6) (2017), pp. 787-794, 10.1007/s12541-017-0094-4
[25]
Yu L., Zhang Y., Bi Q., Wang Y.
Research on surface normal measurement and adjustment in aircraft assembly
Precis Eng, 50 (2017), pp. 482-493, 10.1016/j.precisioneng.2017.07.004
URL https://www.sciencedirect.com/science/article/pii/S0141635917300491
[26]
Song T., Xi F., Guo S., Ming Z., Lin Y.
A comparison study of algorithms for surface normal determination based on point cloud data
Precis Eng, 39 (2015), pp. 47-55, 10.1016/j.precisioneng.2014.07.005
URL https://www.sciencedirect.com/science/article/pii/S014163591400124X
[27]
Rao G., Yang X., Yu H., Chen K., Xu J.
Fringe-projection-based normal direction measurement and adjustment for robotic drilling
IEEE Trans Ind Electron, 67 (11) (2020), pp. 9560-9570, 10.1109/TIE.2019.2952791
[28]
Olsson T., Haage M., Kihlman H., Johansson R., Nilsson K., Robertsson A., et al.
Cost-efficient drilling using industrial robots with high-bandwidth force feedback
Robot Comput-Integr Manuf, 26 (1) (2010), pp. 24-38, 10.1016/j.rcim.2009.01.002
[29]
Shah U.H., Muthusamy R., Gan D., Zweiri Y., Seneviratne L.
On the design and development of vision-based tactile sensors
J Intell Robot Syst, 102 (4) (2021), p. 82, 10.1007/s10846-021-01431-0
[30]
Lambeta M., Chou P.W., Tian S., Yang B., Maloon B., Most V.R., et al.
DIGIT: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation
IEEE Robot Autom Lett, 5 (3) (2020), pp. 3838-3845, 10.1109/LRA.2020.2977257
arXiv:2005.14679
[31]
hyun Choi S., Tahara K.
Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors
ROBOMECH J, 7 (1) (2020), 10.1186/s40648-020-00162-5
[32]
Kumagai K., Shimonomura K.
Event-based tactile image sensor for detecting spatio-temporal fast phenomena in contacts
2019 IEEE world haptics conference, Institute of Electrical and Electronics Engineers Inc. (2019), pp. 343-348, 10.1109/WHC.2019.8816132
[33]
Product manual syntouch biotac® SP tactile sensor. Tech. rep., 2020.
Google Scholar
[34]
Ito Y., Kim Y., Obinata G.
Robust slippage degree estimation based on reference update of vision-based tactile sensor
IEEE Sens J, 11 (9) (2011), pp. 2037-2047, 10.1109/JSEN.2010.2104316
[35]
Rigi A., Baghaei Naeini F., Makris D., Zweiri Y.
A novel event-based incipient slip detection using dynamic active-pixel vision sensor (DAVIS)
Sensors, 18 (2) (2018), 10.3390/s18020333
URL https://www.mdpi.com/1424-8220/18/2/333
[36]
Sun H., Kuchenbecker K.J., Martius G.
A soft thumb-sized vision-based sensor with accurate all-round force perception
Nat Mach Intell, 4 (2) (2022), pp. 135-145, 10.1038/s42256-021-00439-3
[37]
Kakani V., Cui X., Ma M., Kim H.
Vision-based tactile sensor mechanism for the estimation of contact position and force distribution using deep learning
Sensors, 21 (5) (2021), 10.3390/s21051920
URL https://www.mdpi.com/1424-8220/21/5/1920
[38]
Lepora N.F., Lloyd J.
Optimal deep learning for robot touch: Training accurate pose models of 3D surfaces and edges
IEEE Robot Autom Mag, 27 (2) (2020), pp. 66-77, 10.1109/MRA.2020.2979658
[39]
Sajwani H., Ayyad A., Alkendi Y., Halwani M., Abdulrahman Y., Abusafieh A., et al.
TactiGraph: An asynchronous graph neural network for contact angle prediction using neuromorphic vision-based tactile sensing
Sensors, 23 (14) (2023), 10.3390/s23146451
URL https://www.mdpi.com/1424-8220/23/14/6451
[40]
Ward-Cherrier B., Pestell N., Lepora N.F.
NeuroTac: A neuromorphic optical tactile sensor applied to texture recognition
Proceedings - IEEE international conference on robotics and automation (2020), pp. 2654-2660, 10.1109/ICRA40945.2020.9197046
arXiv:2003.00467
[41]
Kakani V., Cui X., Ma M., Kim H.
Vision-based tactile sensor mechanism for the estimation of contact position and force distribution using deep learning
Sensors, 21 (5) (2021), 10.3390/s21051920
URL https://www.mdpi.com/1424-8220/21/5/1920
[42]
Psomopoulou E., Pestell N., Papadopoulos F., Lloyd J., Doulgeri Z., Lepora N.F.
A robust controller for stable 3D pinching using tactile sensing
IEEE Robot Autom Lett, 6 (4) (2021), pp. 8150-8157, 10.1109/LRA.2021.3104057
[43]
Muthusamy R., Huang X., Zweiri Y., Seneviratne L., Gan D., Muthusamy R.
Neuromorphic event-based slip detection and suppression in robotic grasping and manipulation
IEEE Access, 8 (2020), pp. 153364-153384, 10.1109/ACCESS.2020.3017738
arXiv:2004.07386
[44]
Gupta A.K., Aitchison L., Lepora N.F.
Tactile image-to-image disentanglement of contact geometry from motion-induced shear
Proceedings of the 5th conference on robot learning, Proceedings of machine learning research, vol. 164, PMLR (2022), pp. 14-23
URL https://proceedings.mlr.press/v164/gupta22a.html
[45]
Mei B., Zhu W.
Accurate positioning of a drilling and riveting cell for aircraft assembly
Robot Comput-Integr Manuf, 69 (2021), Article 102112, 10.1016/j.rcim.2020.102112
URL https://www.sciencedirect.com/science/article/pii/S0736584520303227
[46]
Jiang T., Cui H., Cheng X., Tian W.
A measurement method for robot peg-in-hole prealignment based on combined two-level visual sensors
IEEE Trans Instrum Meas, 70 (2021), pp. 1-12, 10.1109/TIM.2020.3026802
[47]
AYYAD A. Robotic manipulator with visual guidance and tactile sensing (US 2023/0073681 A1. United States Patent and Trademark Office, March 2023).
[48]
Macdonald F.L.A., Lepora N.F., Conradt J., Ward-Cherrier B.
Neuromorphic tactile edge orientation classification in an unsupervised spiking neural network
Sensors, 22 (18) (2022), 10.3390/s22186998
URL https://www.mdpi.com/1424-8220/22/18/6998
[49]
Dornaika F., Horaud R.
Simultaneous robot-world and hand-eye calibration
IEEE Trans Robot Autom, 14 (4) (1998), pp. 617-622, 10.1109/70.704233
[50]
Ward-Cherrier B., Cramphorn L., Lepora N.F.
Exploiting sensor symmetry for generalized tactile perception in biomimetic touch
IEEE Robot Autom Lett, 2 (2) (2017), pp. 1218-1225, 10.1109/LRA.2017.2665692
[51]
Sferrazza C., D’Andrea R.
Design, motivation and evaluation of a full-resolution optical tactile sensor
Sensors (Switzerland), 19 (4) (2019), 10.3390/s19040928
[52]
Ward-Cherrier B., Cramphorn L., Lepora N.F.
Tactile manipulation with a TacThumb integrated on the open-hand M2 gripper
IEEE Robot Autom Lett, 1 (1) (2016), pp. 169-175, 10.1109/LRA.2016.2514420
[53]
Liu C., Wang K., Wang Y., Yuan X.
Learning deep multimanifold structure feature representation for quality prediction with an industrial application
IEEE Trans Ind Inf, 18 (9) (2022), pp. 5849-5858, 10.1109/TII.2021.3130411
[54]
Jiang Y., Yin S., Dong J., Kaynak O.
A review on soft sensors for monitoring, control, and optimization of industrial processes
IEEE Sens J, 21 (11) (2021), pp. 12868-12881, 10.1109/JSEN.2020.3033153
[55]
Sun Q., Ge Z.
Gated stacked target-related autoencoder: A novel deep feature extraction and layerwise ensemble method for industrial soft sensor application
IEEE Trans Cybern, 52 (5) (2022), pp. 3457-3468, 10.1109/TCYB.2020.3010331
[56]
Glorot X., Bengio Y.
Understanding the difficulty of training deep feedforward neural networks
Teh Y.W., Titterington M. (Eds.), Proceedings of the thirteenth international conference on artificial intelligence and statistics, Proceedings of machine learning research, vol. 9, PMLR, Chia Laguna Resort, Sardinia, Italy (2010), pp. 249-256
URL https://proceedings.mlr.press/v9/glorot10a.html
[57]
Zhu W., Mei B., Yan G., Ke Y.
Measurement error analysis and accuracy enhancement of 2D vision system for robotic drilling
Robot Comput-Integr Manuf, 30 (2) (2014), pp. 160-171, 10.1016/j.rcim.2013.09.014
URL https://www.sciencedirect.com/science/article/pii/S0736584513000732
[58]
Romero-Ramirez F.J., Muñoz-Salinas R., Medina-Carnicer R.
Speeded up detection of squared fiducial markers
Image Vis Comput, 76 (2018), pp. 38-47, 10.1016/j.imavis.2018.05.004
URL https://www.sciencedirect.com/science/article/pii/S0262885618300799
[59]
Akinlar C., Tonal C.
EDCircles: Real-time circle detection by edge drawing (ED)
2012 IEEE international conference on acoustics, speech and signal processing (2012), pp. 1309-1312, 10.1109/ICASSP.2012.6288130
[60]
Kuffner J., LaValle S.
RRT-connect: An efficient approach to single-query path planning
Proceedings 2000 iCRA. millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (cat. no.00CH37065) (2000), pp. 995-1001, 10.1109/ROBOT.2000.844730
[61]
Sucan I.A., Moll M., Kavraki L.E.
The open motion planning library
IEEE Robot Autom Mag, 19 (4) (2012), pp. 72-82, 10.1109/MRA.2012.2205651
[62]
Mei B., Zhu W., Yan G., Ke Y.
A new elliptic contour extraction method for reference hole detection in robotic drilling
Pattern Anal Appl, 18 (3) (2015), pp. 695-712, 10.1007/s10044-014-0394-6