Virtual prototyping of vision-based tactile sensors design for robotic-assisted precision machining

Vision-Based Tactile Sensors (VBTS) play a key role in enhancing the accuracy and efficiency of machining operations in robotic-assisted precision machining systems. Equipped with VBTS, these systems offer contact-based measurements, which are essential in machining accurate components for industries such as aerospace, automotive, medical devices, and electronics. This paper presents a novel approach to virtual prototyping of VBTS, specifically in perpendicularity measurements using Computer-Aided Design (CAD) generation of VBTS designs, Finite Element Analysis (FEA) simulations, and Sim2Real deep learning to achieve VBTS with high precision measurements. The virtual prototyping approach enables an understanding of the contact between VBTS with different designs and machined surfaces in terms of contact module shape, thickness, markers’ density. Additive manufacturing was employed to fabricate the molds of VBTS contact module, followed by experimental validation of the robotic arm to confirm the effectiveness of the optimized VBTS design. The results show that deviation from the hemispherical shape reduces the data quality captured by the camera, hence increasing the prediction errors. Additionally, reducing the thickness of the contact module enhances the precision of perpendicularity measurements. Importantly increasing markers’ distribution density significantly enhances the accuracy of up to 92 markers at which above it the rate of improvement becomes less pronounced. An VBTS with height of 20 mm, thickness of 2 mm, and 169 markers was found to be within the stringent perpendicularity standards of the aerospace manufacturing industry of 0.58 as a root mean square error, and 1.64 as a max absolute error around the roll and pitch axis of rotation. The established virtual prototyping methodology can be transferred to a wide variety of elastomer-based sensors.

[1] A. Verl, A. Valente, S. Melkote, C. Brecher, E. Ozturk, L. T. Tunc, Robots in machining, CIRP Annals
68 (2) (2019) 799–822.
[2] M. Ryalat, H. ElMoaqet, M. AlFaouri, Design of a smart factory based on cyber-physical systems and
internet of things towards industry 4.0, Applied Sciences 13 (4) (2023) 2156.
[3] V. B. Neto, C. Marques, A. Frizera-Neto, A. G. Leal-Junior, Fbg-based sensing system to improve
tactile sensitivity of robotic manipulators working in unstructured environments, Available at SSRN
4369590.
[4] M. Javaid, A. Haleem, R. P. Singh, R. Suman, Substantial capabilities of robotics in enhancing industry
4.0 implementation, Cognitive Robotics 1 (2021) 58–75.
[5] R. N. Albustanji, S. Elmanaseer, A. A. Alkhatib, Robotics: Five senses plus one—an overview, Robotics
12 (3) (2023) 68.
[6] J.-Y. Lin, X.-W. Lee, M.-C. Hsieh, C.-O. Chang, High-angular-sensitivity total-internal-reflection heterodyne
interferometry for small displacement measurements, Sensors and Actuators A: Physical 277
(2018) 163–168.
[7] W. Yuan, S. Dong, E. H. Adelson, Gelsight: High-resolution robot tactile sensors for estimating geometry
and force, Sensors 17 (12) (2017) 2762.
[8] S. Zhang, Z. Chen, Y. Gao, W. Wan, J. Shan, H. Xue, F. Sun, Y. Yang, B. Fang, Hardware technology
of vision-based tactile sensor: A review, IEEE Sensors Journal 0 (0) (2022) 0.
[9] N. Sunil, S. Wang, Y. She, E. Adelson, A. R. Garcia, Visuotactile affordances for cloth manipulation
with local control, in: Conference on Robot Learning, PMLR, 2023, pp. 1596–1606.
[10] W. Ruan, W. Zhu, K. Wang, Q. Lu, W. Yeh, L. Luo, C. Su, Q. Wang, Vision-tactile fusion based detection
of deformation and slippage of deformable objects during grasping, in: International Conference
on Cognitive Systems and Signal Processing, Springer, 2022, pp. 593–604.
[11] B. Fang, X. Long, Y. Zhang, G. Luo, F. Sun, H. Liu, Fabric defect detection using vision-based tactile
sensor, arXiv preprint arXiv:2003.00839 0 (0–10) (2020) 0.
[12] W.-C. Shi, J.-M. Zheng, Y. Li, X.-B. Li, Three-dimensional reconstruction method for machined surface
topography based on gray gradient constraints, Applied Sciences 9 (3) (2019) 591.
[13] C. Zhang, S. Cui, S. Wang, J. Hu, Y. Huangfu, B. Zhang, High-precision 3d reconstruction study with
emphasis on refractive calibration of gelstereo-type sensors, Sensors 23 (5) (2023) 2675.
[14] H. Ma, J. Ji, K.-M. Lee, Effects of refraction model on binocular visuotactile sensing of 3-d deformation,
IEEE Sensors Journal 22 (18) (2022) 17727–17736.
[15] B. T. Turkey, Image processing based tactile tactical sensor development and sensitivity determination
to extract the 3d surface topography of objects, Sensors and Actuators A: Physical 358 (2023) 114415.
[16] M. Aamir, A. Sharif, M. Z. Zahir, K. Giasin, M. Tolouei-Rad, Experimental assessment of hole quality
and tool condition in the machining of an aerospace alloy, Machines 11 (7) (2023) 726.
[17] Y. Fan, J. Huang, Y. Zhang, N. Huang, Q. Bi, Y. Wang, Improvement in hole-pose error for aerospace
drilling applications based on hermite surface reconstruction and manifold error similarity, Precision
Engineering 81 (2023) 22–35.
[18] L. Yu, Y. Zhang, Q. Bi, Y. Wang, Research on surface normal measurement and adjustment in aircraft
assembly, Precision Engineering 50 (2017) 482–493.
[19] Y. Zhang, Q. Bi, L. Yu, Y. Wang, Online adaptive measurement and adjustment for flexible part during
high precision drilling process, The International Journal of Advanced Manufacturing Technology 89
(2017) 3579–3599.
[20] L. Yu, Q. Bi, Y. Ji, Y. Fan, N. Huang, Y. Wang, Vision based in-process inspection for countersink in
automated drilling and riveting, Precision Engineering 58 (2019) 35–46.
[21] Y. Gao, D. Wu, Y. Dong, X. Ma, K. Chen, The method of aiming towards the normal direction for
robotic drilling, International Journal of Precision Engineering and Manufacturing 18 (2017) 787–794.
[22] W. Zhu, B. Mei, G. Yan, Y. Ke, Measurement error analysis and accuracy enhancement of 2d vision
system for robotic drilling, Robotics and Computer-Integrated Manufacturing 30 (2) (2014) 160–171.
[23] S. Zhang, Y. Sun, F. Sun, Y. Yang, B. Fang, Pfs 1.0: A development tool applied to vision-based tactile
sensor process formulation and fabrication, Sensors and Actuators A: Physical 367 (2024) 115090.
[24] C. Taesi, F. Aggogeri, N. Pellegrini, Cobot applications—recent advances and challenges, Robotics
12 (3) (2023) 79.
[25] I. M. Zaid, M. Halwani, A. Ayyad, A. Imam, F. Almaskari, H. Hassanin, Y. Zweiri, Elastomer-based
visuotactile sensor for normality of robotic manufacturing systems, Polymers 14 (23) (2022) 5097.
[26] H. Sajwani, A. Ayyad, Y. Alkendi, M. Halwani, Y. Abdulrahman, A. Abusafieh, Y. Zweiri, Tactigraph:
An asynchronous graph neural network for contact angle prediction using neuromorphic vision-based
tactile sensing, Sensors 23 (14) (2023) 6451.
[27] M. Halwani, A. Ayyad, L. AbuAssi, Y. Abdulrahman, F. Almaskari, H. Hassanin, A. Abusafieh,
Y. Zweiri, A novel vision-based multi-functional sensor for normality and position measurements in
precise robotic manufacturing, Available at SSRN 4360666 0 (0–10) (2023) 0.
[28] S. Tsuji, T. Kohama, Using a convolutional neural network to construct a pen-type tactile sensor system
for roughness recognition, Sensors and Actuators A: Physical 291 (2019) 7–12.
[29] V. Corsino, V. Ruiz-D´ıez, J. M. Gilp´erez, M. Ram´ırez-Palma, J. L. S´anchez-Rojas, Machine learning
techniques for the estimation of viscosity and density of aqueous solutions in piezo-actuated 3d-printed
cells, Sensors and Actuators A: Physical 363 (2023) 114694.
[30] S. Zhang, Y. Sun, J. Shan, Z. Chen, F. Sun, Y. Yang, B. Fang, Tirgel: A visuo-tactile sensor with
total internal reflection mechanism for external observation and contact detection, IEEE Robotics and
Automation Letters (2023).
[31] U. H. Shah, R. Muthusamy, D. Gan, Y. Zweiri, L. Seneviratne, On the design and development of
vision-based tactile sensors, Journal of Intelligent & Robotic Systems 102 (2021) 1–27.
[32] D. Zhao, F. Sun, An accurate positioning method for robotic manipulation based on vision and tactile
sensors, in: Cognitive Systems and Signal Processing: 5th International Conference, ICCSIP 2020,
Zhuhai, China, December 25–27, 2020, Revised Selected Papers 5, Springer, 2021, pp. 621–631.
[33] V. Kakani, X. Cui, M. Ma, H. Kim, Vision-based tactile sensor mechanism for the estimation of contact
position and force distribution using deep learning, Sensors 21 (5) (2021) 1920.
[34] M. Li, T. Li, Y. Jiang, Marker displacement method used in vision-based tactile sensors—from 2d to
3d-a review, IEEE Sensors Journal 0 (2023) 0.
[35] H. Can, P. Svec Jr, J. Bydzovsky, P. Svec Sr, H. S¨ozeri, U. Topal, Systematic optimization of the
sensing properties of ring-core fluxgate sensors with different core diameters and materials, Sensors and
Actuators A: Physical 255 (2017) 94–103.
[36] Z.-P. Wang, H. Yao, H. H. See, W. Yang, B. C. K. Tee, Z. Liu, Tactile sensory response prediction and
design using virtual tests, Sensors and Actuators A: Physical 360 (2023) 114571.
[37] D. J. Sut, P. Sethuramalingam, Soft manipulator for soft robotic applications: a review, Journal of
Intelligent & Robotic Systems 108 (1) (2023) 10.
[38] T.-D. Nguyen, J. S. Lee, Recent development of flexible tactile sensors and their applications, Sensors
22 (1) (2021) 50.
[39] Y. Chen, J. Lin, X. Du, B. Fang, F. Sun, S. Li, Non-destructive fruit firmness evaluation using visionbased
tactile information, in: 2022 International Conference on Robotics and Automation (ICRA),
IEEE, 2022, pp. 2303–2309.
[40] I. Andrussow, H. Sun, K. J. Kuchenbecker, G. Martius, Minsight: A fingertip-sized vision-based tactile
sensor for robotic manipulation, Advanced Intelligent Systems 5 (8) (2023) 2300042.
[41] H. Sun, K. J. Kuchenbecker, G. Martius, A soft thumb-sized vision-based sensor with accurate all-round
force perception, Nature Machine Intelligence 4 (2) (2022) 135–145.
[42] L. Marechal, P. Balland, L. Lindenroth, F. Petrou, C. Kontovounisios, F. Bello, Toward a common
framework and database of materials for soft robotics, Soft robotics 8 (3) (2021) 284–297.
[43] W. Small, M. A. Pearson, W. A. Jensen, Astm d395 short-term compression set of solid (nonporous)
siloxanes: Se 1700, sylgard 184, and” new” m9787, Tech. rep., Lawrence Livermore National
Lab.(LLNL), Livermore, CA (United States) (2015).
[44] I. Abubakar, P. Myler, E. Zhou, Constitutive modelling of elastomeric seal material under compressive
loading, Modeling and Numerical Simulation of Material Science 0 (2016) 28–40, (Accessed on
11/04/2023).
[45] Ecoflex, Ecoflex™ 00-30 product information — smooth-on, inc., (Accessed on 11/04/2023) (2023).
URL https://www.smooth-on.com/products/ecoflex-00-30/
[46] T. O’Malley, E. Bursztein, J. Long, F. Chollet, H. Jin, L. Invernizzi, et al., Kerastuner (2019).
URL https://github.com/keras-team/keras-tuner