Classification Models Inspired by Multisensory Integration

Abstract

This paper introduces two multisensory object classification models inspired by multisensory integration in the brain. The two models differ in the manner by which information from multimodal stimuli is combined. The feature-integrating model combines unisensory features which are classified using a multisensory classifier. In the decision-integrating model, the unisensory stimuli are classified independently and the classification results are combined and subsequently classified by a multisensory classifier. The models are implemented using multilayer perceptron classifiers. Through several sets of experiments involving the classification of auditory and visual representations of ten digits, it is shown that the multisensory classification systems yield significantly higher classification accuracies when compared with those of the unisensory classifiers. Furthermore, the flexibility offered by the generalized models makes it possible to simulate and evaluate various combinations of multi-modal stimuli and classifiers under varying uncertainty conditions.