"Touching to See" and "Seeing to Feel": Robotic Cross-modal Sensory Data Generation

3^rd March 2021, 11:30
Guanqun Cao

Abstract

The synergy of the visual and tactile sense is common in human daily experience, which enable us to achieve the information sources from different dimensions, multiplying our understanding of the environment in a multi-modal way. Sometimes one of a kind feeling is inaccessible under restricted conditions, e.g., it is difficult to observe in a dark environment with vision and using unimodal visual or tactile perception limits the perceivable dimensionality of a subject. In this paper, we propose a visual-tactile sensory data generation framework, which is based on generative adversarial network (GAN), to make up inaccessible data, by generating sensory outputs of one sense from the data of the other. Extensive experiments on the dataset of cloth textures show that the proposed method can produce photorealistic outputs. The structural similarity index and peak signal-to-noise ratio are implemented to evaluate similarity of the generated output and real data, and results demonstrate that realistic data have been generated. The proposed framework has potential to expand datasets for classification tasks, generate sensory outputs that are not easy to access, and also advance integrated visual-tactile perception.

Biography

I am a second year PhD student of the smARTLab at the Department of Computer Science, University of Liverpool. My research is mainly about sensory synergy of tactile sensing and vision for grasping, e.g., perception for manipulation, fusion of multimodal sensing.