Does the semantic content of verbal categories influence categorical perception? An ERP study

Abstract

Accumulating evidence suggests that visual perception and, in particular, visual discrimination, can be influenced by verbal category boundaries. One issue that still awaits systematic investigation is the specific influence of semantic contents of verbal categories on categorical perception (CP). We tackled this issue with a learning paradigm in which initially unfamiliar, yet realistic objects were associated with either bare labels lacking explicit semantic content or labels that were accompanied by enriched semantic information about the specific meaning of the label. Two to three days after learning, the EEG was recorded while participants performed a lateralized oddball task. Newly acquired verbal category boundaries modulated low-level aspects of visual perception as early as 100–150 ms after stimulus onset, suggesting a genuine influence of language on perception. Importantly, this effect was not further influenced by enriched semantic category information, suggesting that bare labels and the associated minimal and predominantly perceptual information are sufficient for CP. Distinct effects of semantic knowledge independent of category boundaries were found subsequently, starting at about 200 ms, possibly reflecting selective attention to semantically meaningful visual features.

Introduction

Categorization is an important mechanism of human cognition or, as Lakoff put it, “…there is nothing more basic than categorization to our thought, perception, action, and speech” (Lakoff, 1987, p. 5). By means of neural mechanisms such as various forms of predictive coding, categorizing an object can be achieved intriguingly fast (Delorme, Rousselet, Macé, & Fabre-Thorpe, 2004), sometimes even as fast as merely detecting an object (Grill-Spector & Kanwisher, 2005). Taking into account the functional architecture of the brain, dynamic interactions of top-down and bottom-up processing set a plausible frame for cognitive factors such as expectations, previous knowledge, or language structures to serve as modulators of perception (Gilbert and Li, 2013, Lupyan, 2012). Concerning the relation between language and cognition it has been suggested that linguistic categories affect how we perceive our physical environment, a view that has typically been referred to as the Sapir–Whorf hypothesis (see Gentner and Goldin-Meadow, 2003, Levelt, 2013 for reviews). The language and thought debate has recently gained impetus following studies on the categorical perception (CP) of colors, demonstrating that colors from different verbal categories (e.g., green vs. blue) are discriminated faster than colors from the same category (e.g., different shades of blue; Gilbert et al., 2006, Regier and Kay, 2009, Winawer et al., 2007). Specifically, using a visual search task Gilbert et al. (2006) found a color CP effect in reaction times (RT) for stimuli presented in the right visual field (RVF), but not in the left visual field (LVF), an effect that extends to other stimulus domains such as animals (Gilbert, Regier, Kay, & Ivry, 2008). Given the special role of the left hemisphere in language processing (e.g., Caplan, 1994) these findings were discussed as evidence for stronger influences of linguistic representations on perceptual discrimination in the left hemisphere (e.g., Gilbert et al., 2006). However, Witzel and Gegenfurtner (2011) pointed out that the stimuli used in these previous studies may not have been psychophysically equidistant, which complicates the distinction between a bottom-up perceptual basis for CP and top-down factors such as language. In a series of experiments that sought to replicate lateralized CP with psychophysically well-controlled stimuli they found behavioral evidence for color CP, but no lateralization to the RVF (see also Liu, Chen, Wang, Zhou, & Sun, 2008). Moreover, whereas Witzel and Gegenfurtner (2011) questioned the lateralization of color CP to the RVF, recent results from Brown, Lindsey, and Guckes (2011) challenged the very existence of color CP under balanced perceptual conditions. Thus, one of the aims of the present study was to investigate CP while fully controlling physical stimulus features by using a learning design (see below).

Evidence on the time course of color CP has been provided by recent studies using event-related potentials (ERPs). Unlike most behavioral studies, several electrophysiological studies employed oddball paradigms to investigate CP, bearing the advantage of a well-known succession of visual ERP-components in these tasks (Boutonnet et al., 2013, Clifford et al., 2012, Holmes et al., 2009, Mo et al., 2011, Thierry et al., 2009). For instance, Holmes et al. (2009) reported earlier onset latencies of the P1 and N1 components, associated with low and high level visual perception in extrastriate cortex (Di Russo, Martinez, Sereno, Pitzalis, & Hillyard, 2002) and spatial attention (Hillyard & Anllo-Vento, 1998), for deviants that crossed linguistic boundaries compared to deviants from the same category in a non-lateralized visual oddball task. Using a lateralized oddball paradigm and similar color stimuli as Gilbert et al. (2006), Mo et al. (2011), reported an effect of color term boundaries for stimuli presented in the right visual field on the visual mismatch negativity, a component taken to reflect preattentive change detection (cf. Thierry et al., 2009). These findings suggest that CP effects for colors are located at early stages before or during attentive visual perception, rather than later post-perceptual stages (but see Clifford et al., 2012).

Theoretically, CP can be explained in terms of feedback connections from areas associated with language processing and perceptual areas (e.g., Bar, 2004, Kveraga et al., 2007). The effects of linguistic categories on perception described above can be accounted for by assuming feedback from (predominantly left-hemispheric) areas associated with language processing to perceptual regions in the brain (Gilbert et al., 2006). In line with this account, Lupyan (2012) pointed out that in CP, language exerts an on-line top-down influence on perceptual processes. For instance, mental representations could become transiently more categorical when a verbal label is activated. Lupyan (2012) argued that, if CP merely consisted of a perceptual long-term warping of the underlying mental representations, it would seem paradoxical that color CP can be disrupted by verbal interference, but not by comparable nonverbal interference (Drivonikou et al., 2007, Gilbert et al., 2006, Winawer et al., 2007, Witthoft et al., 2003). Accordingly, the fact that CP appears to be both deep, i.e. affecting basic perceptual processes, and shallow (i.e. disruptable) is best explained by assuming linguistic top-down influences on perceptual processes in a highly dynamic neurocognitive system. Specifically, if mental representations may become transiently more categorical in the sense that objects that belong to the same category are perceived as more similar because shared features relevant for categorization are highlighted or “warped” (Goldstone et al., 2001, Lupyan, 2012), feature-based visual processing might play a major role in CP. We will test this idea with components of the event-related brain potential that are associated with low and high level aspects of visual feature processing and their integration (see below).

Several studies have shown that CP can be induced by perceptual training (e.g., Clifford et al., 2012, Goldstone, 1994, Notman et al., 2005, Özgen and Davies, 2002; see Goldstone & Hendrickson, 2010 for a review). For instance, in a study by Özgen and Davies (2002), participants acquired a new category-boundary in the green color space by sorting different shades of green into two categories. This led to a CP-effect comparable to the effect observed in the “natural” green–blue distinction (see also Clifford et al., 2012). Other findings on trained CP include the discrimination of geometric shapes (Goldstone, 1994), similarity ratings of faces (e.g., Stevenage, 1998) and discrimination of grating patterns, in the latter case with effects located in area V1 of the visual cortex (Notman et al., 2005).

These perceptual training studies show that CP can be acquired relatively fast and appears to rely at least partly on genuine perceptual mechanisms. However as new category boundaries were introduced via perceptual training, the specific role of language—independent of training—remains unclear. Thus, demonstrations of CP could be stronger evidence for a role of language if perceptual conditions were balanced, with category boundaries being introduced by an arbitrary assignment of verbal labels (Livingston, Andrews, & Harnad, 1998). Zhou et al. (2010) and Holmes and Wolff (2012) presented learning studies designed to test for CP based on new verbal categories with an equal amount of perceptual training in all conditions. In the study by Zhou et al. (2010), participants in the experimental group learned to label four color stimuli from the blue and green color spaces with pseudowords. Participants in the control group were exposed to the same color stimuli for the same amount of time, but without learning new categories or labels. Before learning, both groups showed stronger CP in the RVF for distinctions between stimuli crossing the preexisting green–blue linguistic boundary. In the visual search task after learning, the experimental group additionally exhibited stronger CP in the RVF for the new linguistic boundaries, whereas there was no change in the control group. The authors concluded that the acquisition of lateralized CP can thus be attributed to language.

In the study by Holmes and Wolff (2012), participants learned to sort silhouettes of four previously unfamiliar objects into two categories. The objects were either labeled with pseudowords or learned without labels. Subsequently, participants showed acquired CP lateralized to the RVF in a visual search task. Notably, this effect occurred in both learning conditions (i.e., with or without the acquisition of verbal labels). The authors concluded that lateralization of CP to the left hemisphere may not be based on language, but may instead reflect a general preference of the left hemisphere for categorical processing. This finding is in contrast to other studies demonstrating facilitated learning of new categories by arbitrary verbal labels in infants (Waxman & Markow, 1995) and adults (Lupyan, Rakison, & McClelland, 2007). Clearly, further research is needed to elucidate learning mechanisms, and the use of previously unfamiliar stimuli is a promising approach for investigating learned CP.

One factor that still awaits systematic investigation is the specific influence of semantic contents of verbal categories in CP. It is unclear whether and to which extent categorical perception is influenced by the contribution of semantic knowledge about verbal categories. Because virtually all existing linguistic categories entail a minimal amount of meaning (see below), this question can only be addressed in a learning paradigm with initially unfamiliar stimuli and verbal labels.

During language acquisition the speaker or hearer implicitly establishes a link between a verbal label and a number of objects that are referred to with this label in the respective language (Bloom, 2002). Initially, this link is made on the basis of perceivable properties, which are common to these objects. At a later stage of language acquisition a learner is able to individually apply a category label to a newly encountered object on the basis of this knowledge (Aitchinson, 2012, chap. 18). We suggest that this is the minimal amount of meaning that is represented by category labels (in the following also called bare labels). However, typically, additional semantic knowledge about the objects of a given category is acquired, e.g., semantic knowledge about the functional properties of the members of a certain object category. This knowledge goes beyond perceptual properties and has to be learned more explicitly. This is what shall be called semantically enriched verbal category labels within the present study.

To date it is unclear what the relative contributions of bare labels and semantically enriched verbal labels to CP are. Studies on CP, even those that include the learning of new category boundaries (e.g., Clifford et al., 2012), typically involve stimuli that are well-established in semantic memory, such as colors (Gilbert et al., 2006, Winawer et al., 2007), animal species (Gilbert et al., 2008) or verbal material (Lupyan, 2008). However, semantic knowledge has been shown to shape visual perception. For instance, Mitterer, Horschig, Müsseler, and Majid (2009) reported that declarative world knowledge influences the perceived color of objects. Furthermore, expert knowledge about specific object categories such as dogs and birds (Tanaka & Curran, 2001) and verbally transmitted object-related semantic information (Abdel Rahman and Sommer, 2008, Rabovsky et al., 2012) can influence early stages of visual perception, as indexed by the P1 and N1/N170 components. Moreover, Gauthier, James, Curby, and Tarr (2003) have observed faster responses in a sequential object matching task with novel objects that were associated with arbitrary semantic features (e.g., fast, friendly, heavy) when the objects had distinct compared to overlapping (shared) attributes, suggesting that semantic knowledge has a genuine influence on visual discrimination. Thus, visual perception and perceptual discrimination have been shown to be affected by linguistic categories and by semantic knowledge, but the relative contributions of the two factors and their potential interplay remain elusive.

Taking into account that concrete semantic contents of verbal categories do not only stress common conceptual attributes of the category members but may also highlight their shared perceptual features, a direct influence of semantic information on CP seems likely. Specifically, as it has been argued that verbal labels affect perception and categorization by selectively activating perceptual features that are diagnostic of the category (e.g., Lupyan, 2012), such selective enhancements of diagnostic perceptual features might be augmented by semantic information on visual object properties.

To distinguish CP induced by bare labels from semantically enriched verbal categories we employed a learning paradigm with initially unfamiliar yet realistic objects that were associated with distinct or shared novel verbal labels. Additionally, half of the verbal labels were associated with enriched semantic information about object functions that related to the visual appearance of the objects. As discussed above, semantic content may be a major source of verbal category effects, giving rise to or augmenting categorical perception. Alternatively, the effects of verbal labels and semantic knowledge associated with the labels may be independent and located at different processing stages.

The learning of initially unknown objects and categories additionally allowed us to investigate whether findings on CP hold for newly learned object categories. As suggested above, during language acquisition the link between a category label and certain perceptual properties that are representative for a category is made implicitly and depends on perceptual experience. In the present study the use of unfamiliar yet realistic objects about which participants had no previous knowledge allowed for a systematic investigation of such implicit category assignments. At the same time, the assignment of objects to conditions can be fully counterbalanced in a learning paradigm, thereby assuring a control of previous knowledge and low-level stimulus features. If implicit categorization processes for those new objects take place just like during natural language learning, effects of CP comparable to those reported for color or animal perception should be induced by either bare or semantically enriched labels or both.

As the lateralization of CP effects to the RVF has been an important element in the literature, visual perception of the newly learned objects was tested with a lateralized oddball task in which two objects were presented simultaneously, one in the right and one in the left hemifield. In the frequent standard trials, identical objects were shown, whereas in the rare deviant trials, a different object was presented in the left or right hemifield. Although in the literature behavioral CP effects have most reliably been observed in visual search paradigms, we expected to replicate a similar behavioral effect in the present oddball task. Specifically, we expected an interaction of category boundaries and visual field, because targets that belong to different verbal categories (the between-categories condition) should be detected faster when presented in the right compared to the left visual field. Furthermore, if the semantic content of verbal labels has an effect on CP, this interaction should be strengthened by semantic information.

We used event related potentials to gain insight into the time course and functional loci of CP effects. A genuine influence of verbal categories on perceptual processes should be reflected in components that are associated with low and high level visual processing, namely, the P1 and/or N1 components. The P1 peaks about 100–130 ms after stimulus onset and reflects processing of low-level visual object features. The neural generators have been localized in dorsal extrastriate cortex of the middle occipital gyrus (early phase) and in the ventral extrastriate cortex of the fusiform gyrus (late phase; Di Russo et al., 2002). The N1 component (peaking at about 150 and 200 ms) is taken to reflect higher-level processing of visual features and their integration during holistic processing of objects and faces. The generators of the N1 family—that may vary depending on the specific materials and the source localization procedure—have been located in bilateral occipito-temporal cortex and the fusiform gyrus (e.g. Bötzel et al., 1995, Rossion et al., 2003) or in the posterior superior temporal sulcus (Itier and Taylor, 2004, Watanabe et al., 2003; for a recent review, see Eimer, 2011).

Irrespective of the precise loci of the neural sources of the P1 and N1, both components are generated in areas subserving the visual processing of features and whole objects. Crucially, given the assumption that shared features may be highlighted or warped by shared verbal categories (Lupyan, 2012; see discussion above), low and/or high-level aspects of visual feature processing, as indexed by the P1 and N1 components, should be modulated. This effect may be enhanced by additional semantic information relating visual and functional object features. Furthermore, if lateralized CP is replicable, category boundaries should modulate early visual ERP components for target stimuli presented in the RVF, but not (or to a lesser degree) in the LVF. Again, we assumed that this interaction would be strengthened by semantically enriched verbal labels if the semantic content of the labels has an influence on CP. In contrast, if linguistic categories and semantic knowledge affect distinct perceptual or post-perceptual processes, different components should be modulated at different points in time.