A Review of Multimodal Hallucinations

Categorization, Assessment, Theoretical Perspectives, and Clinical Recommendations


Schizophr Bull. 2021;47(1):237-248. 

In This Article

Theoretical Perspectives

Theories of hallucinations can be divided into those that consider these experiences to be attributable to modality-specific pathological processes and those that propose modality-general processes affecting multiple sensory modalities. An elegant discussion of these theories with regards to MMHs can be found in Fernyhough's paper.[43] The present review builds on this work and aims to synthesize how these theories stand up to scrutiny given the available evidence on MMHs.

A related question is to what extent the processes underlying MMHs are pathological. It could be that the pathology lies within one sensory system (visual in DLB or auditory in psychosis), and then the normal processes that ensure sensory consistency create MMHs from an initially unisensory experience (eg, you start to see something, priming you to later hear something consistent with it). This would be in line with an activation of modality-general representations, particularly of social agents, but, at present, we have little knowledge of how MMHs develop, both within a single episode and over time.

Modality-specific Processes

The fact that recent studies indicate that MMHs are more prevalent in psychiatric and organic disorders than previously assumed—albeit without a rigorous distinction between the exact type of MMHs—raises the question of how to best conceptualize these experiences across diagnostic categories and whether theories of unimodal hallucinations can account for such multimodality.

The presence of modality-specific processes of hallucinations is supported by evidence linking deficits in peripheral sensory systems to an increased likelihood of experiencing hallucinations in those domains. This is the case for VHs in eye diseases,[9] which are associated with increasing visual impairments and abnormal activity of visual pathways. Analogously, in the auditory domain, Linszen et al[44] found an association between hearing impairment and AHs, the latter worsening as a function of hearing loss severity. Capture studies in schizophrenia also showed that activity in the auditory cortex is linked to AHs.[45–48] Finally, while not much research is available in other modalities, case studies in PD patients found an association between impaired sense of smell and the experience of olfactory hallucinations.[49]

Few brain imaging studies have explored the role of sensory complexity on the neural networks identified in hallucinations. The functional patterns associated with the occurrence of AHs, VHs, or auditory–visual hallucinations were investigated in an fMRI capture study on medication-free adolescents experiencing first-episode psychosis[50] and confirmed the recruitment of physiological modality-specific pathways in these aberrant experiences. Furthermore, in schizophrenia,[51,52] audio–visual hallucinations are associated with distinct functional and structural dysconnectivity patterns compared to those associated with unimodal AHs.

Modality-general Processes

If both serial and simultaneous MMHs can be explained by the modality-specific processes discussed so far, one would expect to find sensory deficits across all sensory modalities in which patients experience hallucinations. To our knowledge, no empirical study has yet shown such a pattern of results, and the presence of MMHs across disorders cannot be fully explained by considering modality-specific processes alone, indicating the involvement of modality-general processes as well.

Misattribution Biases. One candidate process for explaining hallucinations across modalities is a general bias to misattribute internally generated representations (of any modality) to an externally generated source. This concept stems from the inner speech misattribution theory of AHs in schizophrenia,[53] which posited that such experiences are the outcome of misattributing one's inner speech to an external entity. As Fernyhough[43] suggested, the same mechanism could be translated to other modalities, with evidence for its analog in vision: PD patients with VHs have stronger visual imagery than those without hallucinations.[54] Furthermore, someone's tendency to be a visualizer or verbalizer relates to their proneness to MMHs.[55] Whether this stronger visual imagery is coupled with internal representations being unusually compelling—as is the case in the auditory domain—remains to be further investigated. It is unclear whether one can have internal smells or internal gustatory sensations equivalent to having internal speech. Further issues with this model's explanatory power for MMHs are that, in its current form, it cannot explain the different rates of hallucinations across modalities in several disorders, as it would predict that all sensory systems will be equally affected, resulting in similar rates of hallucinatory experiences across them. Nevertheless, this discrepancy could be due to the fact that some sensations might not be equally salient across senses—as suggested earlier—or because some senses might have a different role/weight in perception in distinct scenarios. Thus, MMHs pose interesting challenges to misattribution bias theories that warrant further attention.

Reality Monitoring Deficit. Could MMHs reflect deficits in reality monitoring? This is the ability to discern internally from externally generated memories and such monitoring of the origin of information might go awry in some disorders.

Research suggests that reality monitoring relies on the anterior paracingulate sulcus (PCS) in the medial prefrontal cortex[56,57] and evidence of its deficits in hallucinating patients is found across conditions (in schizophrenia;[58,59] in PD[60]). Work by Garrison et al[56] showed a potential link between reality monitoring and MMHs. They found that, in a group of patients with schizophrenia, reduced length of the PCS was linked to a 19.9% increased likelihood of experiencing hallucinations and, crucially, this relationship was independent of hallucinatory modality (but see[57,61] for work on PCS in nonclinical voice hearers). Such evidence, coupled with the previous findings, suggests that reality monitoring might be a possible modality-general mechanism for explaining the emergence of MMHs.

Nevertheless, this theory still suffers from the same shortcomings discussed in previous models and it needs to be reconciled with the evidence of separable modality-specific reality monitoring systems[62] that could be interacting with the modality-general ones. If so, it would be interesting to see if there are as many modality-specific reality monitoring systems as there are MMHs experienced by patients.

Social Agent Representations Theory. A third modality-general theory that considers the often-neglected social content of hallucinations is the "Aberrant Activation of Social Agent Representations" theory.[63] Rather than specific sensory stimuli (voices/images), it is the general representation of another agent that is mistakenly activated, triggering the experience of that entity in all modalities. The importance of this theory for MMHs lies in its suitability for experimental testing: (1) one could investigate the extent to which different modalities relate to social agents and to what extent the relatedness dimension discussed earlier might correspond to this and (2) test the prediction that simultaneous and related MMHs should be more commonly experienced than other types. Currently, these hypotheses have not been tested, and further research is needed to explore the explanatory power of this theory for MMHs.

Information Theory Frameworks: Predictive Coding and Circular Inference

The next set of theories cannot be easily characterized as either modality general or modality specific but have elements compatible with both. These theories share the view that hallucinations arise from a dysfunction in the interaction between top-down expectations and bottom-up information. They diverge in the specific way in which these sources of information are thought to interact.

The first theory is predictive coding (PC).[4,64] It considers the mind as a hierarchical structure engaged in message passing in which perception is a generative process.[65] The levels of this hierarchy are thought to mirror the hierarchical structure of the world at different levels of abstraction and what is passed along is a prediction error (PE),[66] ie, the difference between what the higher level predicted about the input from the lower level and what the signal actually was.[67] The brain aims to minimize PEs[68] so that what is expected and what is experienced are congruent. If PEs are falsely generated/assigned too much precision, the brain would mistakenly update its model of the world, potentially leading to altered perceptions (at the lower level, ie, hallucinations) and altered beliefs (at the higher level, ie, delusions). Research in schizophrenia and PD suggests that hallucinations might be linked to overreliance on top-down influences on perception,[69–72] possibly suggesting similar trans-diagnostic mechanisms of hallucinatory experiences.

Within this framework, one would predict that experiencing hallucinations in one modality might increase the prior expectation (top-down mechanism) of experiencing a congruent hallucination in other modalities. This is in line with Dudley's findings,[13] where 88% of their early psychosis patients experienced congruent MMHs. Nevertheless, current conceptualizations of PC do not explain the emergence of incongruent and/or unrelated serial or simultaneous MMHs, nor do they account for hallucinations that are not consonant with one's general model of the world—as is the case with unimodal or serial unrelated MMHs in which one experiences disembodied voices.[13] Furthermore, there is underspecification of whether PE and priors are modality specific or modality general or whether they can be a combination of the 2. More details regarding this would allow one to test specific empirical hypotheses regarding MMHs within this PC framework and thus pave the way for further research in the field.

A related theory in hallucinations research involves the concept of circular inference (CI).[73] The core idea is that hallucinations and delusions can be understood in the framework of information theory and, specifically, of message passing in the form of belief propagation in a hierarchical neural network. If not tightly controlled, information propagated in both bottom-up and top-down directions can be amplified and reverberated through "loops"—generating reverberation errors. Bottom-up sensory evidence could be erroneously taken as top-down expectations, while top-down information could be fed back up and be mistaken for sensory information, leading to "expecting what we see" or "seeing what we expect," respectively.[74] This system is thought to be under the control of the glutamatergic and GABAergic balance and, when it becomes dysfunctional, it can lead to hallucinations and delusions[75]—as found in schizophrenia.[76]

Evidence linking CI and MMHs comes from work on synesthesia, a perceptual phenomenon whereby stimulation of one modality leads to experiences in another modality[77] and which has been linked to disinhibited feedback from association cortex to sensory cortex.[78] Ongoing research using simulations of drug-induced synesthesia[79] has shown that reverberation of information descending the cortical hierarchy (descending loops) could induce audio–visual experiences in the absence of clear sensory input (cross-modal hallucinations) but also a contamination of one modality by another in the presence of a unimodal stimulus (synesthesia). These 2 features were retained as good candidates for modeling psychedelic-induced subjective experiences using probabilistic approaches. Both ascending and descending loops could account for variations in the phenomenology of psychotic (or pharmacologically induced) hallucinations: AHs with reduced illusions in the former and audio–visual hallucinations with synesthesia in the latter.

Functional Systems Approach: A Bridge Between Modality-specific and Modality-general Processes?

Finally, it is important to consider how MMHs can be conceptualized through dysconnectivity between elements of a functional system,[43,80] a framework that might allow one to understand how modality-general and modality-specific processes interact. Evidence of aberrant connectivity of neural networks has been found in relation to AHs in schizophrenia[81] and to VHs in PD.[82,83] There is some work on resting state and network connectivity in patients experiencing MMHs in schizophrenia, but no evidence is available for other disorders. One could speculate a hypothesis whereby the observed heterogeneity of unimodal and MMHs within and across groups could be attributable to a common modality-general process that gives rise to different clinical phenomenological manifestations depending on what part of the network it affects, alongside potential, although perhaps not sufficient and necessary, pathology in modality-specific networks.