Do multiple types of sensory stimuli relating to a similar, abstract concept (e.g. cold, hot, sweet) trigger a similar reaction in the brain? Does our brain show a common reaction when we think of the concept “cold”, hear the word “cold”, see a picture of ice, touch an ice cube or taste ice cream? If yes, are those reactions similar to different people? This seemed like an appropriate problem for semantic gap research – when we consider an abundance of representation used in contemporary human communication, we can see the necessity of implementing a similar ability to recognize abstract concepts into machines if they are to speak our language.

An important aspect of semantic gap research includes the possibility of teaching machines of artificial intelligence to associate sensory stimuli with meaning – much like humans do. This must, as we believe, be taken into account when programming speech-imitating software that recognizes multiple stimuli as representations of the same concept. If, additionally, a concept of meaning could be implemented into artificial brains, maybe we would begin to construct machines that and comprehend the same principles that human mind speaks. Tracking any links among reactions on different multimodal inputs may contribute to that enterprise. Either way, we regard our future work as justified.

We are looking for commonalities, or differences, in how people make sense of multimodal stimuli. Therefore, a randomly selected, representative group would have to be tested for trustful results. The test will address the question: “Does one concept or idea, when presented to a subject through different sensory channels, trigger a similar reaction in the brain?” The EEG headset’s software will be used for measuring brains’ responses to stimuli.

We expect different stimuli to show some similar brain activity just as long as they all relate to the same concept (e.g. hot, cold). The reaction to some types of stimulus (e.g. hearing and imagining) might more closely correlate than those of others (e.g. thinking and feeling).

See Brainiacs' presentation in this prezi