The Talking Heads experiment: Origins of words and meanings

Front matter

The Talking Heads experiment

Origins of words and meanings

Luc Steels

Preface

I set up the Talking Heads experiment with a group of brilliant students and collaborators at the end of the nineteen nineties. It was intended to be the first large-scale, open-ended experiment in the emergence of a shared set of grounded concepts and a vocabulary for expressing these concepts by a population of autonomous agents. Inspired by Ludwig Wittgenstein, the experiment took the form of a series of language games, more concretely games of reference about a “world” made up of geometric figures pasted on a white board and observable by the agents through pan-tilt cameras. I wanted to demonstrate with this experiment earlier breakthroughs in the study of language origins and test whether they would hold for large-scale populations and open-ended environments. I also wanted to find out how humans would interact with these agents. So we made it such that human users, after logging in through the Internet, could teach new words to agents or use the words they learned from the agents to play their own language games.

In 1999, the experiment went live in the context of an exhibition called Laboratorium organised in Antwerp (Belgium) by Hans-Ulrich Obrist and Barbara Vanderlinden. After a first experimental run from 27 June 1999 to 3 October 1999, the experiment was repeated as part of a new exhibition called NO1SE organised in Cambridge and London (UK) by Adam Lowe and Simon Schaffer from 22 January to 26 March 2000, with additional installations at the Palais de la Découverte in Paris and several other places.

At the occasion of the 1999 Laboratorium exhibition, the draft of a book was published that described the experiment and the underlying theoretical assumptions in considerable detail. For many reasons, not at least that work continued at great speed on other exciting experiments, the original “pre-edition” of the book never made it to a fully finished officially published work, and circulated only as an “underground” edition. This was disappointing because the Talking Heads experiment was an important breakthrough. Moreover the experiment contained the first inklings of mechanisms that since then have been worked out, enhanced and tested in many experiments which replicated the original results and further enhanced them. The present volume is intended to fill this gap.

Part I of this book contains the original Talking Heads volume which has been only slightly edited to correct for minor mistakes. It is a miracle that the original source files survived and that the figures could be reconstructed. Part II of this book contains additional unpublished background material, reports on different aspects of the experiment, including its scientific results, and a brief overview of further developments in language evolution research that took place on the basis of the experiment.

Dozens of people worked on the Talking Heads. The initial research started in 1997 at the Artificial Intelligence Laboratory of the Free University of Brussels (VUB) funded by a “Geconcentreerde onderzoeks actie” (GOA) of the Belgian government. Joris Van Looveren worked closely with me on a first prototype that was demonstrated in 1997, using an active vision system custom-built by Tony Belpaeme and segmentation algorithms implemented by Danny van Tieghem. Edwin de Jong did the first theoretical investigations of the underlying semiotic dynamics. Once the initiative for a public installation was taken, the main hub became the Sony Computer Science Laboratory in Paris, where I worked together intensely with Frédéric Kaplan and Angus McIntyre, with additional contributions for the teleportation infrastructure by Silvére Tajan and Alexis Agahi. The AI Laboratory of the VUB remained a second hub where important contributions were made most notably by Joris Van Looveren, Tony Belpaeme, Holger Kenn and Mario Campanella. To all of them I am grateful that we were able to create such an extremely exciting experiment that stimulated many thousands of people to think about language and its origins in new ways.

I also thank the curators of the Laboratorium Exhibition in Antwerp (Hans-Ulrich Obrist and Barbara Vanderlinden) and the members of its curating board (Bruno Latour and Carsten Hiller), the curators of the NO1SE exhibition in London (Adam Lowe and Simon Schaffer), artist Olafur Eliasson, with whom I collaborated on the Look into the Box piece at the Musé d’art Moderne in Paris, artist Anne-Mie Van Kerckhoven for collaborations for the Chromosophy laboratory in Aachen and the many organisers and helping hands who made the other installations possible. Sylvia Spruck Wrigley has helped to improve the 1999 pre-edition (now Part I of the book) under considerable time pressure and Marleen Wynants was crucial in the last phases of this publication with comments, professional advice, photographs, and support.

Part II of the book discusses what happened after the Talking Heads experiment. Our research went through various boom and bust cycles. In the good years there was money to hire new people and push the research forwards, but then bad years would come and the team disintegrated again due to lack of resources. This made progress less substantial than it could have been but had the advantages that waves of new young people were given a chance to contribute. The first wave started working after the Talking Heads experiments were finished. At Sony CSL there was first exciting work by Pierre-Yves Oudeyer and Frederic Kaplan pursuing the origins of turn taking and symbol usage with the arBo robots, thus exploring even earlier stages in the origins of language.

Thanks to the FP6 ECAgents project and the FP7 ALEAR projects of the European Commission, a new team could be formed around 2004, which included Joris Bleys, Joachim De Beule, Bart de Vylder, and Jelle Zuidema at the VUB in Brussels. Meanwhile, we got access through the Sony Computer Science Laboratory to the QRIO humanoid robots thanks to Masahiro Fujita and Hideki Shimomura. A new team formed in Paris which included Nancy Chang, Katya Gerasimova, Martin Loetzsch, Vanessa Micelli, Michael Spranger, Simon Pauw and Remi Van Trijp. I thank them all for major contributions to the experiments reported in the second part of the book. I also thank Stefano Nolfi for coordinating the ECAgents project with a firm hand and Manfred Hild and his team for creating the new humanoid myon robot that we used in later experiments.

vi

In 2008/2009 I was a fellow at the Wissenschaftskolleg in Berlin, which was an ideal environment to reflect on our research program and how we could proceed. Many new ideas came out of that year, in particular, a realisation that for several of the key puzzles we were trying to solve, inspiration could be found in evolutionary biology, and this led to a new path which only now is beginning to be explored. It is good to know that after a major collapse in funding around 2009, there is again a team of young people assembling to push research on language games further. They include Emilia Garcia Casademont in Barcelona, Miquel Cornudella and Paul Van Eecke in Paris, and Yana Knight in Brussels. Financing remains precarious but the future is in their hands!

The present new edition is not only special because of the historical significance of the Talking Heads experiment but also because it is the first one in a new series “Computational Models of Language Evolution” published by the Language Science Press. The intention of this series is to make available through Open Access in-depth models of language evolution that have been validated using agent-based computational simulations. I am grateful to Martin Haspelmath and Stefan Miiller (the editors of the Language Science Press) for making it possible to publish in Open Access research results which do not quite fit in the standard mode. I thank Maria Ferrer Bonnet for help in the final stage of adapting bibliographies. And I am grateful to ICREA for time to create a revised and extended version of the Talking Heads Book and to the Institut de Biologia Evolutiva in Barcelona for providing such an excellent working environment. I thank in addition Remi van Trijp for his help in making this series a reality and Annemie Maes for her encouragement in the final stretches to finish this book.

Barcelona, October 2014.

vii

Preface I The 1999 Talking Heads book 1 Introduction 1:5. “Beyond. Turing sc 9.8 oe wy ae Oh eee ee Sa OR owe Se a8 16 “Whe: book 3.45 2-4) vg Ba a dere det dee he ow PaaS Bale weed 2 Preview 21a Deleporting? jj): da key eae a hes Be 2 i a PS 21:2. ‘Therobots.: 2.55.5 5254 ¢4 aa ka ee a eee ee eee 24:3... Te agents oc Goh ed. te God ch de Me eae ta ote ites onde dole a a ols 24.4 Interactivity’ sa seca whee Woh ee oe eR ee he de 24° “Lexicalisationy eier Ona E e ages eel a SO A S ee eo 214.2" > <Ariew Word. 3 ios. ss Gove beget do ee Foe ee ee Eee iv

216 “Conclusions’, s 233.4. Per aun yee cee dd habe E i Gee aw eae A 47 3 Perception 49 3.1.1 Artificial sensors and actuators . . .. ooo a 50

3.1.2 Natural sensing . .. ooa a 51

31:3- Behaviours 4i ace de ea a ale al e a ir A aea 51

3:2- Segmentation ss aes hy ey s ie Se ee E oR Ee E 53 32l - Feature extraction otgan ei a a a E EE a Steed as S 54

3.2.2 Divergent perception . . . s. oosa a 56

3.2.3 The sieve architecture ... aooaa a 56

3:3- Sensory Channels” 2 sn. ge) aa Gon aia aa R ie ee Been 58

3:3:5 ‘Sealing: 24.2 gin cca tie eke Se ee ee ee A 61

3.316) ° Saliency* nz) 2 ieee, os Se hee ead Avan ak ie dy ete, Rt bar 62

3.4 Methodology. ... 2... 0... 0... . ee 65 3.4.1 Puttingup scaffolds... et kaara a EAER E E a 66

3.4.2 Idealisation and realism .... noaa aaa 67

39 The GEOM Worlds si uie e oa a aE Ble ae EAE O Ea 70 3.6° ‘Conclusions: o daba goia e AEE ana G aae & Sp alee ei 72 4 The Discrimination Game 75 4.1.1 -Tbheempiricist ssanie n ioei a E A o A EE 77

4.1.2 The rationalistť’s stance . . .. oaoa a 78

42 Selectionism: < oaiae aa d4 pea a A a a ar A eee 81 4.2.1 Principles of selectionism . .. aoaaa a 81

4.2.3 The tree metaphor ~ gas ties oaii oea E R O a 83

4.2.5 Comparing approaches s a-seerias a n er p a R E 84

4.3 Discrimination trees . .. . aoaaa a 85 4.3.1 Making distinctions . . . . aa a 85

4.3.4 The Pachinko machine . .. oaaao a 89

45° Conclusions: cdas a hee ee EA ee ee eee Re Ew ee 102 5 The Naming Game 105

5.1.5 Characterising the lexicon . .. ooa a 111

5.16: ‘Monitoring: es 0.4 03 e kien e a AAE E e we oe 113

517- Measuring lexical coherence poca r a a e e a 115

TA OCURE UP: aee r a E E T ADEE AE ee 116 5.2.1 Coping with new meanings .. . ooo a 116

5:3: -Self-organisation e e s ra a el oD a E a a Ra he Be ee 122 5.3.1 Winner-take-all processes . .. ooa a 123

5.3.3 Increasing-returns economics . . . osoo 125

54:2: “Vanguage:Contact: peri la baane Bee bk we hoe de ee ee eee A 129

55° “Conclusions<-i.4 4-5. o¥.8 Ba eh ES EY EAS RU a Ae eH 131 6 The Guessing Game 133

6:14: -REpalk processes: 20 sow ae sein eh See Bel: T y ae ae 143

622) “SYMONYMY sf et ea ee eee Oe 6 S a ald oe ee 144 6:3 Ambiguity 22.05 ois Vad geet ie beet hohe eee ee 149 xi

II xii

64 Seale up: soni thee ed a dS BE GOR On Gok Gut bat Regan T - Conclusions... 4 na ie ee Mh AEG a ee aN Paws Grounding 7.1.2 Concept acquisition .. 2... 0.0... 00000. 2 eee eee 7.2.2 Semioticlandscapes es oia ra a a E 72:9- Competitiondiagrams -i sore aa a I E E E 72:4. RMF coherence: soeh nara ee a a eee eee 73-. The ideal ‘language... s.'..0:305.¢ 42 a ot Sick bl Bele Ge aa 731. ‘Totakeoherence: 5. 35 6.45 E 2 eed He OE Pe a ean 741. “The story of-fepl. > 3, <-4 5 eS at we le Oe a TS TA”, The StoryOf XU: word yes Noten goede AOR aed Hed He ee ge 743 “Theentry of 03-05 2.02.0 or a ee ee ee ee ee 79- Roussedu's paradox: «$e: koh. ei ay RK ee ee ted 7.5.2 Ontological coherence o ey ia graa de ER ER EA S a 7.6 ‘Conclusions. erae e bake Pe Fo eee eM E E S

Installations and experiments

8.1 8.2 8.3 8.4 8.5

The Laboratorium exhibition... 2... 2... e a ee The installation 2... . aaa Conclusions: 2-4. 6-6 6 4-844 a see Wak ae oP ae he ee Ae

9 The second series (2000-2001) 231 Sav! “The exhibition's oad weed ob eS oak ae a ee eS 231

9.6) Conclusions: Crp Gade a eds Fede tek Ree eo Be 264

II Beyond the Talking Heads 267 10 Beyond the Talking Heads experiment 269

10:3: “Conclusions® fa a 348 Ak Cae EE AS gee ae sy ek A 294

11 Language strategies for humanoid robots 295 Hii Cholengesra riir eee ee OE E ee ee ee eo 297

11.1.2 -Semiotic-networks* a %. 3-5 44084 bec 3 Pho es oo Gk ea N 300

112. Action Games 2 i203. 2a ae We BA ee ee A eee he ole 304

1A’ Conclusion: <4 3. Sy 2h eo Ye abe Me ae ware EE ww Ghd 321

12 Language evolution 323 12.2 Fitness. landscapes: e se 2... ee 329 12.2.1 Fitness landscapes of language systems . . . oouo aaa 330

12.4 Generation of new strategies... 2... . ee ee 343

12.4.1 A meta-strategy for generating new conceptualisation strategies 344 xiii

12.4.2 Meta-strategies for generating new lexicogrammatical strategies 347

Nameé:index 2... 8X & 6 bid ede BBE a hk ae oY ed eee db ae 369