This project introduces a multi-sensory search engine concept, exploring the potential of augmenting traditional visual search interfaces, such as Google Images, with haptic feedback. The core hypothesis is that integrating tactile sensations into search processes can enhance search efficiency by using the human brain’s ability to combine various sensory information. To investigate this, I created a multi-modal sensory dataset coupled with user questionnaires. These tools were used to identify critical sensory features that users rely on to distinguish materials and textures: compliance and roughness. To address these key features, I developed sensory feedback and recording systems, including two haptic feedback devices: DeepScreen, which provides compliance and stiffness feedback, and NaptX, offering roughness feedback via nail vibrotactile stimulation. For sensory recording, I developed an electronic skin (eSkin) for the recording of force and vibrotactile data.

This research could lead to two primary applications: tele-perception, where sensory recordings directly control sensory feedback devices, and automated sensory feedback generation based on the user’s virtual interaction with a selected material.

I am conducting evaluations to assess the realism and efficiency of the proposed feedback systems within the context of a multi-sensory search engine.