Inverse and transitivity of cross-modal correspondence in mulsemedia

Abstract

The human brain has evolved to learn, adapt and operate optimally in multisensory settings. Exploring the multisensory-phenomena in a human computer interaction (HCI) context could help design interfaces and displays that tap into users’ mental models. In this paper we explore the influence of crossmodal correspondences on the Quality of Experience (QoE) in a mulsemedia setup. A mulsemedia enhanced test-bed was developed to perform delivery of video enhanced with audio and haptic effects that were designed considering crossmodal congruencies principles. Accordingly, we performed an experiment on the correspondence from the visual features of selected videos into high/low pitch audio when they are experienced in conjunction with auto-generated haptic effects. Our results showed good insights into inverse and transitive cross-modal correspondences. In addition, the audio signal generated out of the angular shape (visual feature) and its corresponding haptic effect is found to have enhanced the viewers’ quality of experience.