Language Evolution and Computation Bibliography

We make the case for developing a Computational Comparative Neuroprimatology to inform the analysis of the function and evolution of the human brain. First, we update the mirror system hypothesis on the evolution of the language-ready brain by (i) modeling action and action recognition and opportunistic scheduling of macaque brains to hypothesize the nature of the last common ancestor of macaque and human (LCA-m); and then we (ii) introduce dynamic brain modeling to show how apes could acquire gesture through ontogenetic ritualization, hypothesizing the nature of evolution from LCA-m to the last common ancestor of chimpanzee and human (LCA-c). We then (iii) hypothesize the role of imitation, pantomime, protosign and protospeech in biological and cultural evolution from LCA-c to Homo sapiens with a language-ready brain. Second, we suggest how cultural evolution in Homo sapiens led from protolanguages to full languages with grammar and compositional semantics. Third, we assess the similarities and differences between the dorsal and ventral streams in audition and vision as the basis for presenting and comparing two models of language processing in the human brain: A model of (i) the auditory dorsal and ventral streams in sentence comprehension; and (ii) the visual dorsal and ventral streams in defining "what language is about" in both production and perception of utterances related to visual scenes provide the basis for (iii) a first step towards a synthesis and a look at challenges for further research.

Mirror neurons for manipulation fire both when the animal manipulates an object in a specific way and when it sees another animal (or the experimenter) perform an action that is more or less similar. Such neurons were originally found in macaque monkeys, in the ventral premotor cortex, area F5 and later also in the inferior parietal lobule. Recent neuroimaging data indicate that the adult human brain is endowed with a “mirror neuron system,” putatively containing mirror neurons and other neurons, for matching the observation and execution of actions. Mirror neurons may serve action recognition in monkeys as well as humans, whereas their putative role in imitation and language may be realized in human but not in monkey. This article shows the important role of computational models in providing sufficient and causal explanations for the observed phenomena involving mirror systems and the learning processes which form them, and underlines the need for additional circuitry to lift up the monkey mirror neuron circuit to sustain the posited cognitive functions attributed to the human mirror neuron system.

The Mirror System Hypothesis of the evolution of the human language-ready brain emphasizes the parity requirement for communication through language. It gives key roles to mirror neurons for manual actions and the evolution of brain mechanisms for an imitation of the praxis. Both the imitation and language require integration of mirror neurons with diverse neural systems. The visual data on shape and pose of an object are processed in parietal cortex and passed to an area of the premotor cortex called F5. Many single neurons in F5 fire most strongly when the monkey executes a limited range of manual actions, with distinct neurons related to such as a precision pinch, tearing paper, or breaking peanuts. The Mirror System Hypothesis holds that the brain mechanisms, which support language, evolved in part by an elaboration of Broca's area atop the mirror system for manual action. The Mirror System Hypothesis has several elements such as language demands parity and mirror neurons provide a basis for parity. The hominin evolution mentions that mirror neurons for manual actions preceded those for vocal actions, thereby lending support to gestural theories of language origin. Ape gestures are flexible and learnable, in contrast to the mostly innate system of primate vocalizations, exemplifying processes whereby a practical action may become ritualized to serve as a communicative action. Imitation is essential for language and mirror neurons are essential for imitation but do not support imitation in and of themselves.

Because language doesn't fossilize, it is difficult to unambiguously time the evolutionary events leading to language in the human lineage using traditional paleontological data. Furthermore, techniques from historical linguistics are generally seen to have an insufficient time depth to tell us anything about the nature of pre- modern-human language. Thus hypotheses about early stages of language evolution have often been seen as untestable ''fairy tales''. However, the discovery of human-unique alleles, associated with different aspects of language, offers a way out of this impasse. If an allele has been subjected to powerful selection, reaching or nearing fixation, statistical techniques allow us to approximately date the timing of the selective sweep. This technique has been employed to date the selective sweep associated with FOXP2, our current best example of a gene associated with spoken language. Although the dates themselves are subject to considerable error, a series of different dates, for different language-associated genes, provides a powerful means of testing evolutionary models of language if they are explicit and span the complete time period between our separation from chimpanzees to the present. We illustrate the potential of this approach by deriving explicit timing predictions from four contrasting models of ''protolanguage.'' For example, models of musical protolanguage suggest that vocal control came early, while gestural protolanguage sees speech as a late addition. Donald's mimetic protolanguage argues that these should appear at the same time, and further suggests that this was associated with Homo erectus. Although there are too few language-associated genes currently known to resolve the issue now, recent progress in the genetic basis for dyslexia and autism offers considerable hope that a suite of such genes will soon be available, and we offer this theoretical framework both in anticipation of this time, and to spur those developing hypotheses of language evolution to make them explicit enough to be integrated within such a hypothesis-testing framework.

Much of the debate concerning the question ''Was Protolanguage Holophrastic?'' assumes that protolanguage existed as a single, stable transitional form between communication systems akin to those of modern primates and human languages as we know them today. The present paper argues for a spectrum of protolanguages preceding modern languages emphasizing that (i) protospeech was intertwined with protosign and gesture; (ii) grammar emerged from a growing population of constructions; and (iii) an increasing protolexicon drove the emergence of phonological structure. This framework weakens arguments for the view that the earliest protolanguages were not holophrastic while advancing the claim that protolanguages became increasingly compositional over time en route to the emergence of true languages.

We focus on the evolution of action capabilities which set the stage for language, rather than analyzing how further brain evolution built on these capabilities to yield a language-ready brain. Our framework is given by the Mirror System Hypothesis, which charts a progression from a monkey-like mirror neuron system (MNS) to a chimpanzee-like mirror system that supports simple imitation and thence to a human-like mirror system that supports complex imitation and language. We present the MNS2 model, a new model of action recognition learning by mirror neurons of the macaque brain and augmented competitive queuing, a model of opportunistic scheduling of action sequences as background for analysis of modeling strategies for ``simple imitation'' as seen in the great apes and ``complex/goal-directed imitation'' as seen in humans. Implications for the study of language are briefly noted.

Mirror neurons may hold the brain's key to social interaction - each coding not only a particular action or emotion but also the recognition of that action or emotion in others. The Mirror System Hypothesis adds an evolutionary arrow to the story - from the mirror system for hand actions, shared with monkeys and chimpanzees, to the uniquely human mirror system for language. In this volume, written to be accessible to a wide audience, experts from child development, computer science, linguistics, neuroscience, primatology and robotics present and analyze the mirror system and show how studies of action and language can illuminate each other. Topics discussed in the fifteen chapters include: What do chimpanzees and humans have in common? Does the human capability for language rest on brain mechanisms shared with other animals? How do human infants acquire language? What can be learned from imaging the human brain? How are sign- and spoken-language related? Will robots learn to act and speak like humans?

TABLE OF CONTENTS
Preface
Part I. Two Perspectives:
1. The mirror system hypothesis on the linkage of action and languages Michael Arbib
2. The origin and evolution of language: a plausible, strong-AI account Jerry Hobbs
Part II. Brain, Evolution and Comparative Analysis:
3. Cognition, imitation and culture in the great apes Craig Stanford
4. The signer as an embodied mirror neuron: neural systems underlying sign language and action Karen Emmorey
5. Neural homologies and the grounding of neurolinguistics Michael Arbib and Mihail Bota
Part III. Dynamical Systems in Action and Language:
6. Dynamical systems: brain, body and imitation Stefan Schaal
7. The role of vocal tract gestural action units in understanding the evolution of phonology Louis Goldstein, Dani Byrd and Elliot Saltzman
8. Lending a helping hand to hearing: a motor theory of speech perception Jeremy I. Skipper, Howard C. Nusbaum and Steven L. Small
Part IV. From Mirror System to Syntax and Theory of Mind:
9. Attention and the minimal subscene Laurent Itti and Michael Arbib
10. Action verbs, argument structure constructions, and the mirror neuron system David Kemmerer
11. Linguistic corpora and theory of mind Andrew Gordon
Part V. Development of Action and Language:
12. The development of grasping and the mirror system Erhan Oztop, Nina Bradley and Michael Arbib
13. Development of goal-directed imitation, object manipulation and language in humans and robots Iona D. Goga and Aude Billard
14. Affordances, effectivities and the mirror system in child development Patricia Zukow-Goldring
15. Implications of mirror neurons for the ontogeny and phylogeny of cultural processes: the examples of tools and language Patricia Greenfield.

The Mirror System Hypothesis (MSH) of the evolution of brain mechanisms supporting language distinguishes a monkey-like mirror neuron system from a chimpanzee-like mirror system that supports simple imitation and a human-like mirror system that supports complex imitation and language. This paper briefly reviews the seven evolutionary stages posited by MSH and then focuses on the early stages which precede but are claimed to ground language. It introduces MNS2, a new model of action recognition learning by mirror neurons of the macaque brain to address data on audio-visual mirror neurons. In addition, the paper offers an explicit hypothesis on how to embed a macaque-like mirror system in a larger human-like circuit which has the capacity for imitation by both direct and indirect routes. Implications for the study of speech are briefly noted.

This article offers a conceptual framework for integrated analysis of subprocesses in action and language, based on goal-directed action. Anatomical substrates are discussed in the companion paper (Arbib and Bota, 2003) which approaches ``Integrative Models of Broca's Area and the Ventral Premotor Cortex'' within the context of explaining why the evolution of the human brain yielded mechanisms which support language in a multi-modal vocal-manual-facial system rather than privileging the vocal mode. Arbib and Bota (2003) examine homologies between different cortical areas in macaque and human to revisit the Mirror System Hypothesis (MSH) of Rizzolatti and Arbib (1998)--the notion that the mirror system for grasping (which has its frontal outpost in premotor area F5 of the macaque) provides the substrate for the evolution of the language-ready brain which supports parity of communication. They also offer a critique and extension based on the work of Aboitiz and Garci(1997; Aboitiz et al., 2006). Arbib and Bota (2003) also discussed the utility of neuroinformatics in relating information across diverse cortical atlases and evaluating degrees of homology for brain regions of interest in different species (for discussion, see Deacon, 2004; Arbib and Bota, 2004).

The article analyzes the neural and functional grounding of language skills as well as their emergence in hominid evolution, hypothesizing stages leading from abilities known to exist in monkeys and apes and presumed to exist in our hominid ancestors right through to modern spoken and signed languages. The starting point is the observation that both premotor area F5 in monkeys and Broca's area in humans contain a 'mirror system' active for both execution and observation of manual actions, and that F5 and Broca's area are homologous brain regions. This grounded the Mirror System Hypothesis of Rizzolatti & Arbib (1998) which offers the mirror system for grasping as a key neural 'missing link' between the abilities of our non-human ancestors of 20 million years ago and modern human language, with manual gestures rather than a system for vocal communication providing the initial seed for this evolutionary process. The present article, however, goes 'beyond the mirror' to offer hypotheses on evolutionary changes within and outside the mirror systems which may have occurred to equip Homo sapiens with a language-ready brain. Crucial to the early stages of this progression is the mirror system for grasping and its extension to permit imitation. Imitation is seen as evolving via a so-called 'simple' system such as that found in chimpanzees (which allows imitation of complex 'objectoriented' sequences but only as the result of extensive practice) to a so-called 'complex' system found in humans (which allows rapid imitation even of complex sequences, under appropriate conditions) which supports pantomime. This is hypothesized to provide the substrate for the development of protosign, a combinatorially open repertoire of manual gestures, which then provides the scaffolding for the emergence of protospeech (which thus owes little to non-human vocalizations), with protosign and protospeech then developing in an expanding spiral. It is argued that these stages involve biological evolution of both brain and body. By contrast, it is argued that the progression from protosign and protospeech to languages with full-blown syntax and compositional semantics was a historical phenomenon in the development of Homo sapiens, involving few if any further biological changes.

We analyze how data on the mirror system for grasping in macaque and human ground the mirror system hypothesis for the evolution of the language-ready human brain, and then focus on this putative relation between hand movements and speech to contribute to the understanding of how it may be that a schizophrenic patient generates an action (whether manual or verbal) but does not attribute the generation of that action to himself. We make a crucial discussion between self-monitoring and attribution of agency. We suggest that vebal hallucinations occur when an utterance progresses through verbal creation pathways and returns as a vocalization observed, only to be dismissed as external since no record of its being created has been kept. Schizophrenic patients on this theory then confabulate the agent.

This paper contributes to neurolinguistics by grounding an evolutionary account of the readiness of the human brain for language in the search for homologies between different cortical areas in macaque and human. We consider two hypotheses for this grounding, that of Aboitiz and Garci[Brain Res. Rev. 25 (1997) 381] and the Mirror System Hypothesis of Rizzolatti and Arbib [Trends Neurosci. 21 (1998) 188] and note the promise of computational modeling of neural circuitry of the macaque and its linkage to analysis of human brain imaging data. In addition to the functional differences between the two hypotheses, problems arise because they are grounded in different cortical maps of the macaque brain. In order to address these divergences, we have developed several neuroinformatics tools included in an on-line knowledge management system, the NeuroHomology Database, which is equipped with inference engines both to relate and translate information across equivalent cortical maps and to evaluate degrees of homology for brain regions of interest in different species.

Walter's Machina speculatrix inspired the name Rana computatrix for a family of models of visuomotor coordination in the frog, which contributed to the development of computational neuroethology. We offer here an 'evolutionary' perspective on models in the same tradition for rat, monkey and human. For rat, we show how the frog-like taxon affordance model provides a basis for the spatial navigation mechanisms that involve the hippocampus and other brain regions. For monkey, we recall two models of neural mechanisms for visuomotor coordination. The first, for saccades, shows how interactions between the parietal and frontal cortex augment superior colliculus seen as the homologue of frog tectum. The second, for grasping, continues the theme of parieto-frontal interactions, linking parietal affordances to motor schemas in premotor cortex. It further emphasizes the mirror system for grasping, in which neurons are active both when the monkey executes a specific grasp and when it observes a similar grasp executed by others. The model of humanbrain mechanisms is based on the mirror-system hypothesis of the evolution of the language-ready brain, which sees the human Broca's area as an evolved extension of the mirror system for grasping.

This chapter argues that the ability to imitate is a key innovation in the evolutionary path leading to language in the human and relates this hypothesis to specific data on brain mechanisms. In this context, imitation involves more than simply observing someone else's movement and responding with movement that in its entirety is already in one's own repertoire, imitation involves 'parsing' a complex movement. What marks humans as distinct from their common ancestors with chimpanzees is that whereas the chimpanzee can imitate short novel sequences through repeated exposure, humans can acquire (longer) novel sequences in a single trial if the sequences are not too long and the components are relatively familiar. This chapter will take us through seven hypothesized stages of evolution: (1) grasping; (2) a mirror system for grasping; (3) a simple imitation system for grasping; (4) a complex imitation system for grasping; (5) a manual-based communication system; (6) speech, which I here characterize as being the open-ended production and perception of sequences of vocal gestures, without implying that these sequences constitute a language; and (7) language.

This article recalls Cajal's brief mention of consciousness in the Textura as a function of the human brain quite distinct from reflex action, and discusses the view that human consciousness may share aspects of 'animal awareness' with other species, but has its unique form because humans possess language. Three ingredients of a theory of the evolution of human consciousness are offered: the view that a pres of intended activity is necessarily formed in the brain of a human that communicates in a human way; the notion that such a pres constitutes consciousness; and a new theory of the evolution of human language based on the mirror system of monkeys and the role of communication by means of hand gestures as a stepping-stone to speech.

In monkeys, the rostral part of ventral premotor cortex (area F5) contains neurons that discharge, both when the monkey grasps or manipulates objects and when it observes the experimenter making similar actions. These neurons (mirror neurons) appear to represent a system that matches observed events to similar, internally generated actions, and in this way forms a link between the observer and the actor. Transcranial magnetic stimulation and positron emission tomography (PET) experiments suggest that a mirror system for gesture recognition also exists in humans and includes Broca's area. We propose here that such an observation/execution matching system provides a necessary bridge from `doing' to `communicating', as the link between actor and observer becomes a link between the sender and the receiver of each message.