The evolution of grounded spatial language

Front matter

The evolution of grounded spatial language

Michael Spranger

Computational Models of Language

Preface

This book contributes to our understanding of the origins of spatial language by carrying out language game experiments with artificial agents instantiated as humanoid robots. It tests the theory of language evolution by linguistic selection, which states that language emerges through a cultural process based on the recruitment of various cognitive capacities in the service of language. Agents generate possible paradigmatic choices in their language systems and explore different language strategies. Which ones survive and dominate depends on linguistic selection criteria, such as expressive adequacy with respect to the ecological challenges and conditions in the environment, minimization of cognitive effort, and communicative success.

To anchor this case study in empirical phenomena, the book reconstructs the syntax and semantics of German spatial language, in particular German locative phrases. Syntactic processing is organized using Fluid Construction Grammar (FCG), a computational formalism for representing linguistic knowledge. For the semantics the book focusses in particular on proximal, projective and absolute spatial categories as well as perspective, perspective reversal and frame of reference. The semantic investigations use the perspective of Embodied Cognitive Semantics. The spatial semantics is grounded in the sensory-motor experiences of the robot and made compositional by using the Incremental Recruitment Language (IRL) developed for this purpose. The complete reconstructed system allows humanoid robots to communicate successfully and efficiently using the German locative system and provides a performance base line. The reconstruction shows that the computational formalisms, i.e. FCG and IRL, are sufficient for tackling complex natural language phenomena. Moreover, the reconstruction efforts reveal the tight interaction of syntax and semantics in German locative phrases.

The second part of the book concentrates on the evolution of spatial language. First the focus is on the formation and acquisition of spatial language by proposing strategies in the form of invention, adoption, and alignment operators. The book shows the adequacy of these strategies in acquisition experiments in which some agents act as learners and others as tutors. It shows next in language for-

Preface mation experiments that these strategies are sufficient to allow a population to self-organize a spatial language system from scratch. The book continues by studying the origins and competition of language strategies. Different conceptual strategies are considered and studied systematically, particularly in relation to the properties of the environment, for example, whether a global landmark is available. Different linguistic strategies are studied as well, for instance, the problem of choosing a particular reference object on the scene can be solved by the invention of markers, which allows many different reference objects, or by converging to a standard single reference object, such as a global landmark.

The book demonstrates that the theory of language evolution by linguistic selection leads to operational experiments in which artificial agents self-organize semantically rich and syntactically complex language. Moreover, many issues in cognitive science, ranging from perception and conceptualization to language processing, had to be dealt with to instantiate this theory, so that this book contributes not only to the study of language evolution but to the investigation of the cognitive bases of spatial language as well.

This book would not have been possible without the hard work of the people at Sony Computer Science Laboratory Paris and the A.I. Lab at the Vrije Universiteit Brussels. Many of them have left traces in software and ideas that provide the background against which a book like this one becomes possible. Most notably I would like to thank the current and past members of the AI lab in Brussels and Sony CSL Paris who I have met and who have made contributions to the various software systems that underly the experiments described in this book: Katrien Beuls, Joris Bleys, Joachim De Beule, Wouter van den Broeck, Remi van Trijp and Pieter Wellens.

Martin Loetzsch and Simon Pauw had big impact on many issues discussed in this book. I am indebted to all of them for long discussions that have tremendously shaped my way of thinking and for their collaboration on different aspects of spatial language, conceptualization and embodiment.

Last but not least, I would like to thank Luc Steels who has had tremendous impact on the intellectual ideas put forth in this book, provided the necessary environment to conduct this research, and who continues to be an inspirational and visionary figure for future work.

Spatial language is a vast topic. This book focusses on locative phrases, which are phrases that single out objects in the physical environment with the comMU- NICATIVE INTENTION to draw attention to these objects. The following shows an example of a locative phrase from German.

(1) der Block rechts der Kiste von dir the.nom block.noM right.PREP the.GEN box.GEN from.PREP your.DAT aus perspective

‘The block to the right of the box from your perspective’

Phrases like this can be seen as highly complex tools that help dialog partners to establish spatial reference. The utterance conveys to the hearer a number of instructions such as (1) apply the spatial relation right, (2) use a particular landmark and (3) take the perspective of the interlocutor. These instructions, when applied properly, allow an interlocutor to identify the object in question. The syntactic structure, i.e. the words and the grammatical relations of the utterance, encode which concepts and categories should be used and how the instructions work together. For instance, the fact that the hearer’s perspective on the scene should be taken is conveyed by the phrase von ... aus (‘from ... your perspective’).

Languages vary widely in how they solve the problem of spatial reference - including both how they conceptualize space and how they talk about it (Levinson & Wilkins 2006; Levinson 2003). Spatial position of objects can be expressed using a variety of syntactic means including case, adpositions, particles, and verbs. But, maybe more importantly, there is a breathtaking variety in how people conceptualize space, which spatial relations they know, what counts as a landmark, how perspective is used, etc. Just to give a few simple examples, Spanish has three basic proximal distinctions, while German has two. In Barcelona people make active use of the topology of the surrounding landscape, referring regularly to the seaside and mountainside when giving navigation instructions. In other languages ‘uphill’ and ‘downhill’ are used to refer to proximal objects (Levinson 2003).

These examples show that spatial language is a highly developed tool for establishing reference in a spatial environment. How did spatial language become this way? There is an emerging view now that the most plausible answer to this question is that spatial language is a COMPLEX ADAPTIVE SYSTEM (see Steels 2000a for the general idea of language as a complex adaptive system), that is constructed and changed by its users for the same purpose it is used for today, namely to describe spatial scenes, establish reference to objects in the environment, give instructions for navigation, etc. This process is, of course, not the same process of construction that a group of engineers use when they are building a bridge. In such classic engineering problems, a team of people with a more or less complete view of the problem designs a top-down solution. By contrast, nobody has a global view on the state of a language. Rather, language lives in the individuals of the language community. Every individual has its own views on the state of the language, i.e. what words and grammatical relations are available.

When we combine the evidence from the complexity of particular spatial languages, such as German locative phrases, and the variation that can be seen across languages, it seems reasonable to consider results from a science that routinely deals with complexity and variation — biology. Biological species are highly complex solutions to particular environmental and social challenges. The solutions found by each species exhibit a high degree of variation. This simple observation has forced biology to come up with precise models and predictions to explain the origins of species. It comes as no surprise, then, that theories of language, particularly language evolution and language change, have adopted concepts from biology related to variation, complexity and the emergence of order in biological systems.

This book defends the SELECTIONIST THEORY OF LANGUAGE EVOLUTION, which exploits biological concepts to explain how language is shaped by the communicative needs and environmental conditions that a community or population faces. The theory hypothesizes that agents create variation within their language and select working solutions based on how successful they are in communication (COMMUNICATIVE SUCCESS), how complex they are in processing (COGNITIVE EF- FORT) and other factors.

Studying language change from the perspective of communicative intentions requires a great deal of insight into how humans or artificial systems can realize their specific communicative intentions in social interactions in the physical world. Such holistic explanations necessitate a WHOLE SYSTEMS APPROACH (Steels 2001), in which great care is taken to ensure that perception, conceptualization and linguistic processing systems are integrated to an extent that interaction be-

1.1 Locative spatial language tween agents is possible. Only when all of this machinery is in place can one attempt to examine questions of language change.

In particular, a whole systems approach requires an operational theory of language. How are utterances processed? How is space conceptualized? How is linguistic knowledge represented? How does language interact with the perception of the physical reality? A whole systems approach requires concrete answers to each of these important questions. The resulting burden placed on operational models is of course far greater than for high-level explanations or logical reasoning about these processes. But concrete, mechanistic accounts allow much greater insights into the phenomena studied. In the best case, a successful model of language evolution in a whole systems approach validates many aspects of the theory of language and language change at the same time.

This book contributes to the understanding of spatial language in two ways. First, it provides a detailed operational reconstruction of German locative phrases using a whole systems approach. Second, it explores the evolution of spatial language within the same computational framework. The two parts together argue for (1) the validity of the approach to language, and (2) the validity and explanatory power of the selectionist theory of language evolution.

1.1 Locative spatial language

If one wants to make an interesting claim about how language evolves, one needs a solid idea what language actually is, how linguistic knowledge is represented, and how to organize linguistic processing. These questions are best answered by reconstructing a complex natural language phenomenon such as German locative phrases. Such phrases are used for establishing reference to static objects and identifying them by denoting their spatial position (Miller & Johnson-Laird 1976). They can be distinguished from other parts of spatial language that are dealing with motion or navigation (Eschenbach 2004).

German locative phrases can be analyzed in terms of components or systems which together form a locative phrase. (1) consists of three parts: a SPATIAL RE- LATION, which is combined with a LANDMARK and a PERSPECTIVE.

Spatial Relations The defining quality of locative spatial phrases are that they contain locative spatial relations such as rechts (‘right’), vorne (‘front’), nah (‘near’), nördlich (‘north on’ and so forth). These relations are called locative because they encode static spatial relationships and do not refer to change of position in time. In (1), rechts (‘right’) is the locative spatial relation. In this book we study three classes of spatial relations. Proximal relations are based on distance estimations. Examples of proximal relations in German are nah (‘near’) and fern (‘far’). The second class is called projective relations and includes direction-based spatial relations such as links (‘left’) and vor (‘front’). The last class considered are absolute relations such as nördlich (‘north’) and östlich (‘east’). These are also direction-based, but the direction is related to a geocentric reference system such as the magnetic poles of the earth.

Landmarks A spatial relation is at least a binary and always relates to something. This something is typically called LANDMARK. In (1), the landmark is expressed in the determined noun phrase der Kiste (‘the box’) immediately following the spatial relation.

Perspective For certain spatial relations perspective is important. (1) features a perspective that is marked via the phrase von ... aus (‘from ... viewpoint’). The marker expresses that the viewpoint on the scene is the hearer.

1.2 A theory of language evolution

Theories of language evolution have to explain the evolution of language by defining the role and contribution of four different factors on language: biology, cognition, social cognition, and culture (Steels 2009; 2011c).

Biology To study language evolution from the biological perspective is to ask questions about the relationship of biology, in particular genetics and ecology, with linguistic behavior. The question can be roughly split into two parts. First, what is the biological influence on the general capacity for language in the human population? Second, one can ask for the influence of biology on the particular language spoken by individuals. The first is a general question for the processing capabilities that need to be present for language. This includes that humans require sufficient memory and powerful neural circuitry for processing language, but also production organs for speech and auditory capacities. The second question is how much the biological basis determines the particular language individuals speak. In other words, how much the lexicon and/or the grammar of a language are influenced by genetic conditions.

Cognition Biology has provided us with neural circuitry that enables distinct cognitive capabilities. The cognitive perspective on language asks: what

1.2 A theory of language evolution are the basic cognitive processing mechanisms underlying production and parsing of language, interpretation, conceptualization, but also categorization, perception etc.? Language depends on a number of capabilities that may or may not be prior to language, such as temporal clustering of events, spatial navigation, perception-action systems (Rizzolatti & Arbib 1998; Arbib 2002; Steels & Spranger 2012; 2008), memory and so on and so forth. For instance, some have linked the evolution of language to an increase in capacity for storing cognitive categories and their interrelations (Schoenemann 1999). Another strand of cognitive influences on language evolution are general cognitive operators such as analogy and learning operators, for instance sequential learning (Christiansen et al. 2001).

Social Cognition Inevitably, language is a social phenomenon that occurs when humans interact. Social cognition researchers, for instance, are interested in the social mechanisms that are needed for children to acquire language, but also in the social mechanisms that are prerequisite for the emergence of language. Proposals include things such as “theory of mind” (Dunbar 1998) which is the capacity to understand another individual’s state of mind, “joint attention” (Carpenter et al. 1998) which is the ability to track interlocutor gaze and mutual attentiveness to the same object, “social learning skills” such as imitation learning (Tomasello 1992) and the ability and the urge to “share intentions” (Tomasello et al. 2005). Many of these mechanisms are deeply rooted in biology. For instance, Dunbar (2003) and Worden (1998) argue that theory of mind is a necessary preadaptation for language and that it has evolved via natural selection.

Culture Language is a cultural phenomenon that is undergoing steady change on the cultural level. New words, speech sounds, morphemes, semantic and syntactic structures arise all the time in language (Steels 2011c). This manifests in the incredible amount of cross-cultural variation on all levels of linguistic processing (Evans & Levinson 2009), for example, phonemes (Maddieson 1984; Oudeyer 2005), spatial semantics (Levinson 2003), and syntax (Levinson & Wilkins 2006). This evidence points to strong cultural negotiation processes in which continuous invention is channeled to produce complex useful communication systems. Many of such processes orchestrating change and diversification have been identified. Grammaticalization, for instance, tries to explain the shift from lexical items to grammatical items (Hopper & Traugott 2003). Others have pointed to generational change as the trigger for development in language (Smith, Kirby &

Brighton 2003). The question from the perspective of cultural evolution is what are the mechanisms that bring about change in language and what are the principles with which agents conventionalize language up to the point that interlocutors have a chance of understanding each other.

I emphasize the cultural point of view in this book. That is, my primary concern is with change in language on the cultural level independent of changes in the human biology. Language change occurs on a smaller time scale than, for instance, the adaptation of anew biological organ, let alone a new species. There is absolutely no doubt that languages evolve fast. One just has to look through a text by Shakespeare or Goethe to see that a few hundred years can have impact on vocabulary and grammatical structure. It took Vulgar Latin a mere 1500 years to evolve into about a dozen different languages such as French, Italian, Portuguese or Catalan (e.g., see Pope 1952 for French). If we observe languages today, we can easily see that new words are invented all the time. In academic and technological contexts, for instance, new concepts arise all the time. Roughly 30 years ago vocabulary such as email or website did not even exist. What drives change in language, in what circumstances does it take place and what are necessary requirements for language change to occur? These are questions that cultural evolution theories of language have to address.

1.2.1 Language systems and language strategies

Cultural theories of language evolution have to take a close look at individual trajectories of language change (Steels 2011c). For instance, how did the Russian aspectual system emerge or why does English have a system of determiners and Russian not? How do spatial language systems develop over time? In other words, cultural theories of language evolution must provide models for the emergence and evolution of concrete LANGUAGE SYSTEMS (Steels 2011c). Language systems package a particular SEMANTIC SYSTEM (e.g. a set of spatial categories) and a particular way of expressing these distinctions (e.g. a corresponding set of lexical items). The absolute German system, for instance, consists of four absolute spatial categories and the corresponding strings, e.g. nördlich (‘north’), südlich (‘south’). These spatial categories are the basic building blocks of absolute spatial conceptualization in German. They can be compositionally combined with landmarks to build complex spatial phrases. Interestingly, the German locative systems effectively consist of different conceptualization strategies that have distinct but converging evolutionary trajectories. For instance, the absolute system is connected to the invention of the compass, whereas projective systems often

12 A theory of language evolution at least in part can be traced back to body parts (Traugott & Heine 1991). Nevertheless, many locative spatial relations are used in the same syntactic context.

Spatial language systems such as the proximal or projective system are characterized by a degree of cohesion and systematicity that points to an underlying principle that organizes acquisition, emergence and coordination. We call the mechanisms organizing a particular language system the LANGUAGE STRATEGY (Steels 2011c). Language strategies have a linguistic and a conceptual part. For example, on the conceptual side absolute spatial categories share that they are part of the same conceptualization strategy which uses absolute directions to the magnetic poles of the earth. Syntactically all spatial relations share that they are expressed in a similar way namely lexically and that they can be expressed as adjective, adverb and preposition.

1.2.2 Selectionist theory of language evolution

In this book I follow the selectionist theory of language evolution (Steels 2011c), which applies the dominant theoretical construct in biology NATURAL SELECTION and uses it to explain language change on the level of language systems and language strategies. Additionally, the concepts of SELF-ORGANIZATION, RECRUIT- MENT and CO-EVOLUTION of syntax and semantics are used as theoretical pillars.

Long-term Influence

Short-term Influence

Communicative outcome

Language strategy

Language system

Utterances

Figure 1.1: The fitness of utterances for communication affects both the language system and the language strategy. The effect of the success of a single utterance on the language strategy is smaller which leads to slower change on the level of the strategy. (Figure adapted from Steels 2011c)

Selection Selectionism rests on two principles: generation of possible variants and selection of variation based on fitness. The most important factor in determining the fitness of a particular language strategy, but also of a particular language system, is communicative success. A communicative interaction between two interlocutors is successful if the communicative intention of the speaker is reached. For instance, if the speaker wanted to draw attention to some object, the communication is successful if the hearer pays attention to that object. Communicative success drives selection on the levels of the language system, but also on the level of language strategies (see Figure 1.1).

Variation occurs in the systems for two reasons. First, agents are actively trying to solve problems in communication (Steels 2000a). Agents introduce new categories, new words and grammar when they detect problems that they cannot solve using the current language they know. Second, language is an inferential communication system (Sperber & Wilson 1986) which means that the information provided in an utterance is often incomplete and ambiguous. Interpreting phrases is an active process in which the hearer is fusing information from the context, from the dialogue and his knowledge about the language to arrive at the best possible interpretation. In this process of course hearers might interpret the utterance differently then intended. This is the second source of variation.

Self-organization Steels (2011c) assumes that selection is not enough to explain language change and proposes another driving force in the evolution of language: self-organization — a concept used to account for complex phenomena in physical and biological systems. In short, self-organization is a way to explain how global structure arises out of local interaction of subunits (Camazine et al. 2003). An example from biology for self-organization is swarm behavior in a school of fish. Each individual fish locally controls its behavior based on the estimation of the position and direction of its immediate neighbors. On the global level this leads to consistent swarm behavior. Self-organization is typically seen as a complementary mechanism to selection, although there is some discussion on how to reconcile the two mechanisms. Kauffman (1993), for instance, proposes the following idea. Local components and the interaction rules are determined by selection, whereas the global emergent behavior is explained using self-organization. Applied to the swarm behavior this means that the anatomy of fish as well as the perceptual feedback loop are a product of natural selection. The global emergent swarm behavior is the product of self-organization.

Similar to swarm behavior, agents in a population evolving a language have to achieve global coherence in the language they use. Each agent

12 A theory of language evolution has its own private representations of the language that they speak and they can adjust their own representations based on local interactions with peers. How, from local interactions, agents can agree on a globally shared communication system is the problem of alignment. Psychologists have found that interlocutors align on all levels of linguistic processing even over the course of a few interactions, i.e., dialogue (Garrod & Doherty 1994; Pickering & Garrod 2004). Similar mechanisms applied over a long time span are required for driving populations to self-organize a sufficiently shared communication system (Steels & Kaplan 2002).

Recruitment The last problem for an account of how languages change in the selectionist theory of language evolution is the problem of language strategy generation. The hypothesis is that language strategies are recruited by assembling basic cognitive operations (Steels 2007). For instance, an absolute spatial conceptualization strategy involving distinctions such as “north” and “south” consists of basic categorization mechanisms and the ability to track ones own direction. The two abilities are assembled into the strategy which encompasses the different absolute spatial distinctions. The process is called RECRUITMENT because the cognitive mechanisms which are assembled could, in principle, have evolved or could be learned independently from language.

Co-evolution One of the tenants of the theory of linguistic selection is that syntax and semantics co-evolve. The idea is that recruitment of conceptualization strategies and the invention of new semantic distinctions and spatial relations trigger evolution of the syntax of a language (Steels 1997; 1998). For instance, presumably when the absolute system in German emerged based on a new way of construing reality, this at the same time triggered the invention of new words.

1.2.3 Evolutionary explanations

In every science one has to define what counts as an explanation. This book is guided by what counts as an evolutionary explanation in biology, ethology and psychology (Tinbergen 1963; Dunbar 1998). In order to explain a complex trait from the evolutionary perspective one has to provide explanations on four different levels: FUNCTION, MECHANISM, ONTOGENY and PHYLOGENY.

Function An explanation for a particular behavior has to show what the behavior is good for, i.e. what is its purpose. For Darwinian biology, the function of a behavior has to be explained in terms of its impact on survival or, more precisely, on the production of offspring. For evolutionary linguistics this turns into the question of how a particular language system or a particular strategy helps an agent to be more successful in communication. For example, one can explain particular spatial language systems with respect to their ability to help agents solve communicative problems in spatial navigation and spatial reference.

Mechanism Besides function, one has to identify the mechanisms that give rise to the behavior. This is actually called “causation” by Tinbergen (1963) and it refers to the cause and effect relations that generate a particular behavior. For instance, one can explain how aggressive behavior is generated by looking at changes in hormone levels in an organism, e.g., testosteron causes aggressive behavior. For spatial language this entails a detailed operational model of the production and parsing of spatial language.

Ontogeny The next question is how a particular behavior is acquired. To answer this question one has to identify the developmental steps that the behavior undergoes, but also what is the ontogenetic basis of the behavior. What is learned and what is instinct? For spatial language this requires insights into how spatial language is learned.

Phylogeny A fourth part of every evolutionary explanation has to identify the evolutionary history of a behavior. What are the sequential stages of evolution of a behavior? What are the prerequisites of a behavior? How do evolutionary older behaviors influence the behavior under question? These questions have to be answered with respect to the function of the behavior. In other words, one needs explanations of how the behavior evolved to fulfill its current function. For language evolution scholars have to identify how a particular strategy evolved over time. Was it adapted from an older strategy? How did syntax and semantics of the language system under consideration co-evolve over time?

1.3 Main hypothesis

This book provides experimental evidence for the theory of linguistic evolution. The hypothesis is that spatial language syntax and spatial semantics co-evolve through a cultural process based on selection, self-organization and recruitment.

1.4 Contributions

This book explores the hypothesis for the different components of spatial language: spatial relations, landmarks and perspective. Computational experiments show the emergence of spatial relations, the negotiation of the use of landmarks and perspective. I also explore different strategies for expressing spatial conceptualizations: lexical and grammatical strategies.

1.4 Contributions

This book provides detailed accounts of the function, mechanisms, ontogeny and phylogeny of spatial language.

1. The first contribution is an explanation of the mechanisms behind spatial language for German locative phrases in a complete reconstruction including perception, semantic and syntactic processing. Once the mechanisms are in place, we test the function and impact of components of spatial language in experiments by removing the component in question and examining the effect the removal has on communication.

2. The second contribution is to explain steps in the co-evolution of spatial syntax and spatial semantics through computational models.

This book follows a whole systems approach which allows us to define external criteria for the progress in each of the objectives. The defining moment for the underlying conception of language is communication. A communication system is SUCCESSFUL if it allows robotic agents to achieve their communicative goals such as drawing the attention to an object in the environment.

1.4.1 Evolutionary stages

One way of understanding evolutionary processes is to try to identify evolutionary stages. Over the years, different steps in the evolution of language have been identified involving varying degrees of specificity (Bickerton 1999; Jackendoff 1999; Steels 2005). All of these proposals, while differing in detail and the exact number of stages, agree that language evolution starts at some pre-grammatical stage and increases in complexity to the form of language, in particular, grammar that we see today. Obviously any evolutionary account of language has to show how the current state of complexity of language can be traced back to earlier simple stages.

This book orients itself alongside Steels (2005), who proposes a number of stages of complexity which are relevant for this book: SINGLE-WORD UTTERANCES, MULTI-WORD UTTERANCES, and GRAMMATICAL UTTERANCES (see Figure 1.2).

syntactic i ingle- j~ rammar complexity single-word multi-word g Increase mm auaaasssssssssssssiÃħiħiŮe ek single strategy multiple strategies multiple strategies EFSI categorization partial re-use extreme re-use

Figure 1.2: Co-evolution of syntactic and semantic complexity.

Single-word utterances In this stage agents utter single words that pertain to a particular concept or category used for discriminating objects. Examples for spatial language include utterances that directly refer to spatial regions such as links (‘left’) or nördlich (‘north’). When agents can only express themselves using a single word, this single word necessarily encodes the complete conceptualization strategy. For instance, which landmark or perspective is used for conceptualization is holistically coded in the single word. Since there is no additional information about which conceptualization strategy the term is referring to, agents have to implicitly agree on the precise spatial construal the term is referring to. This limits the re-use of spatial categories in different spatial conceptualization strategies because agents have no way of disambiguating the use of the same spatial relation in different strategies.

Multi-word utterances Single-word communication systems are not very flexible.