Projects in the Second Phase of the SPP:
The Theory of Event Coding (TEC) provides a novel theoretical lens to help us understand the mechanisms underlying a variety of social psychological phenomena that we here refer to as “social resonance”. An interdisciplinary approach towards these questions promises to allow us to go even further: Through understanding how MOO creates social resonance with an Other, we can create feedback loops that move beyond the traditional one-shot manipulation of classic social psychology paradigms. Thus, instead of simple mimicry of basic repetitive behaviors (e.g., foot tapping; Chartrand & Bargh, 1999), our vision for future work provides the foundation for understanding the development of social resonance in real-time. In the short term, this implies better ecological validity. In the longer term, our approach promises to help us understand also large-scale and interactive “social resonance” phenomena – e.g., positive collective emotions (online, face-to-face), or more problematic phenomena such as social exclusion or the propagation of hate speech. Overall, the proposed project thus aims to (1) evolve and test TEC to account for the social psychological phenomenon of social resonance, and (2) to lay the necessary technical foundations towards a fruitful application of new methods in this field.
Co-I: Prof. Dr. -Ing. Tanja Schultz, Dr. Stefan Zachow
Postdocs: Stephan Verschoor
PhD Students: Abdul Haq
The “experience of controlling one’s own actions, and, through them, events in the outside world” (Haggard & Chambon 2012) lies at the heart of the Sense of Agency. While forms of agency may be found in direct encodings of sensorimotor experiences, we propose that more explicit, accessible forms require abstractions away from the actual sensorimotor dynamics, i.e. events. The result may be called an agentive self, which can become ‘aware’ of its own experiences as well as the consequences of its actions in the world.We aim at revealing critical computational components, including learning and processing biases, for the development of an agentive self in robots. Over the three years, we aim at first modeling spatial action-effect binding, to implement a simple form of agency. We will then enhance the architecture to model event-effect anticipations, focusing on the anticipatory crossmodal congruency paradigm, which shows how our minds project our body parts onto future positions before even starting to execute the required motion to reach the position. Finally, we will tackle tool-mediated event-effect anticipations, which we expect to first show in experiments with human participants.Our computational model takes ideomotor theory, comparator models, and the free energy principle (active inference) as the point of departure. Over recent years, including research work within the SPP’s first funding period, we have implemented these principles in various artificial systems and robots. Our deep active inference model enables robots to learn generative models from continuous raw sensory information and to plan in a model-predictive manner. Furthermore, inspired by our contribution to theories of event-predictive cognition, we have also implemented event-predictive systems, which convert relative distances and orientations, into event encodings, enabling agents to plan goal-directly on an event scale.By combining our expertise in adaptive robotics and deep artificial neural networks (Donders) with our expertise in experimental cognitive psychology and neuro-cognitive modeling (Tübingen), we aim to isolate “the mechanisms and prerequisites that allow an [artificial] agent to develop a self” and the scrutinization of the “roles of agency”, fostering the development of more effective event control. Moreover, we expect to identify core mechanisms of self-plasticity in tool-use. Meanwhile, we envisage improving the robot’s agentive processing abilities via the development of compact event-predictive encodings. Beyond the actual project, we expect to contribute to systems that can explain their influence on the environment and that learn to identify its causality. While we will focus on individual robots in this project, we hope that the realization of an agentive self will also facility the development of social interaction competencies – considerations which we hope to discuss with other SPP projects during the second funding phase.
PI: Prof. Martin Butz
PhD Students: Johanna Theuer
What are the prerequisites for the development of an artificial self? How does it change during development and bodily changes? Are the computational models suitable for explaining properties of the self in humans? These are the questions we have addressed in our project “Prerequisites for the development of an artificial self” within the first phase of the SPP. An important aspect that was raised during our collaborations within the SPP are the ways of measuring selfhood in humans and artificial agents. This addresses questions of adaptivity and predictability for a sense of agency or a sense of control, and the effects of bodily interactions of agents with their environment. Answering these questions will also help evaluating the selfhood of different implementations of artificial intelligent agents.
We address measures of the self in this project in the following steps: First, we start with a computational predictive model that allows for a basic sense of agency and body ownership and implement it in different experimental setups of robot interaction. We investigate measures of the self in two methodologically different ways: based on properties of the computational models and their instantiation in artificial agents, and based on behavioural observations inspired by variations of a sensorimotor Turing Test. Finally, we investigate whether the developed measures can account for testing disturbances of the self, in particular in cases where there is an imbalance between predicted and perceived information.
PhD Students: Yasmin Kim Georgie
Student Assistant: Ilja Porohovoj
Forming the sense of agency is believed to follow a Bayesian integration framework. That is, during interaction with the environment, humans take into account prior beliefs regarding the acting agent and integrate this information with the likelihood that they were responsible for the observed outcome.
Here we will directly test the predictions of the Bayesian integration framework when forming the sense of agency. Through a series of psychophysical experiments we test participants’ sense of agency while manipulating the prior beliefs and the sensory feedback using haptic robots integrated in a virtual reality environments. These experiments focus on the formation of the sense of agency during different object manipulation tasks in the presence of a virtual external agent that acts at different magnitudes and under different uncertainties. Each experiment is designed to control the effect of a single contributor in the integration framework while keeping the other factors unaffected. Our results will assist in developing a model of active self for use in robotics, and may also be critical for developing control of assistive robotics (such as rehabilitation robots) and human-robot interactions where the sense of self agency is important for rehabilitation success and successful collaboration.
PIs: Prof. Dr. David Franklin, Mathias Hegele
Postdocs: Dr. Raz Leib, Johannes Keyser
PhD Students: Clara Günter, Christoph Schneider, Yiming Liu
The general aim of the project is to reveal necessary conditions for the emergence of internal representations associated with the self, when dealing with truly embodied and situated agents. This will be based on the study of Helmholtz machines that implement
prediction and recognition as prerequisite for optimal control. Furthermore, the aim is to study to what extent these processes generate high integrated information in the sense of Tononi’s Integrated Information Theory (IIT) of consciousness. This will provide insights about the mechanisms that underly the phenomenal self. Based on information theory, which is quantitative in nature, we expect to identify transitions between qualitatively different kinds of embodiments, thereby relating our work to Metzinger’s orders of embodiment.
The work of this project will be based on crucial insights from the first period of the DFG SPP The Active Self which suggest refined research directions for the second period. In particular, we will develop a hierarchy of controller architectures with increasing granularity, ultimately leading to neuronal architectures. Here, we will benefit from our recent work on Helmholtz machines, which were originally proposed by Dayan et al. Corresponding
learning algorithms suggest a close connection to the Friston’s Free Energy Principle, which will provide a conceptual and formal basis for the project. Having developed controller architectures with various granularities, we will study corresponding information flows in sensorimotor loops of robotic systems, in collaboration with Verena V. Hafner’s group. We want to verify the increase of information integration within a controller when the learning
is involved and incorporates a forward model for prediction and an inverse model for control.
PIs: Prof. Dr. Nihat Ay
PhDs: Carlotta Langer, Jesse van Oostrum
In (con-)joint activities, such as lifting and moving an object, interpersonal coordination is a primary requirement. Moving as a coordinated unit is not likely to succeed at first, as we may fail in predicting the partner’s dynamic capabilities by overgeneralizing from representations such as our own body schema. Subsequent coordination attempts, however, may gradually improve our predictions of our partner’s action space. We argue therefore that by repeated exposure to the movement dynamics of a partner in a joint action task, we will learn to predict their dynamics and generate a body schema representation of the confederate. Thus, we consider mechanisms responsible for the minimization of prediction error crucial to the experience-dependent adjustment of self-representations as well as the acquisition of partner representations, which facilitate the distinction between the influence of self and other with respect to the concurrent demands of the joint task.
In order to research the plasticity of self and partner representations, this project will investigate collaborative learning that occurs in a goal-directed joint dexterity task. Changes in performance will be correlated with altered representations during and following training sessions, in which pairs are practicing a “collaborative hotwire” task, which is a simulated object transporting task, under several levels of spatiotemporal difficulty. In addition to measurements of movements and interaction forces and torques, our project will assess participants’ representations of self and partner. For example, body schema and body image, but also sense of agency as a function of leader-follower relationships and self-other-overlap will be recorded during and after joint practice. Using human-human collaboration as a performance benchmark, we will apply the paradigm to human-robot-collaborations aiming not only to simulate collaborative performance and acquisition of representations of self and other, but also to test the quality of robotic control models in the form of a joint action “Turing test”.
Co-PI: Dr. Leif Johannsen
PostDoc: Arianna Curioni
PhD: Johannes Heidersberger
Even though the scientific question whether the concept of a “Self” is necessary to explain human behaviour is still open, it is obvious that the concept does play a role in everyday behaviour. Laypeople attribute a self to other humans, but also to non-human animals or technical systems, and they treat them accordingly. For instance, agents that are perceived as having a self are treated more carefully and politely, and they receive more empathy. But what are the criteria for attributing a self to another agent? In this question we do not presume the existence or nonexistence of the self. Uncoupled from the actual self we are interested only in the core characteristics that determine whether we attribute a self to other agents.
Identifying those characteristics is the scope of the first phase of the project which consist of a series of experiments exploring the effect of nine perceived characteristics on mind attribution. We will investigate the characteristics causality, speed, equifinality, behavioural efficiency, learning and context sensitivity, additionally social sensitivity, attention sharing and helping. We manipulate the behaviour of small robots to either suggest the presence or absence of these characteristics and are then interested in the mind attribution ratings of our participants.
PIs: Bernhard Hommel, Arvid Kappas
Co-PI: Francesco Maurelli
PhD Students: Kristina Nikolovska, Jan Pohl
In the MoReSpace project, we will investigate the extent to which the transfer of learning is responsible for the development of a “self”, and hypothesize that a conflict-driven attention model plays a major role. In the first part of our project, we investigate the transfer of previously learned action-effect associations to new, unexpected environmental dynamics. Here, we put a strong focus on cognitive plausibility and motivate our model with psychological phenomena such as “haptic neglect”. The phenomenon occurs, for example, when the computer mouse is inverted and the mouse pointer is directed in the opposite direction in each case. In such scenarios, psychologists have found reduced perception of the haptic and proprioceptive senses. Our hypothesis is that this is due to a conflict-driven attention mechanism that improves the ability to deal with such dynamics. We will evaluate our model on a physical robot, and we will theoretically substantiate it with our collaboration partners from psychology. In the second part of the project, we will focus on imitation learning. Our hypothesis is that the attention model captures some psychological properties that are important for the human ability to change perspective and to imitate. We hypothesize that this will lead to novel methods of imitation learning for robots. We expect these methods to lead to significant improvements in the learning performance. We will evaluate this empirically and reproducibly.
Sense of agency, the feeling of being in control of one’s actions and their perceivable results in the environment, is crucial in constructing a sense of self (Gentsch & Schütz-Bosbach, 2015; Haggard, 2017). Agency is most relevant during goal-directed behavior and has been proposed to be fundamentally bound to the processing of affective information (Gentsch & Synofzik, 2014; Ly, Wang, Bhanji, & Delgado, 2019). Moreover, experimental work on the relation of agency experience and action regulation indicates that affective processing might represent a core mechanism that mediates the influence from sense of agency to objective action regulation (Legault & Inzlicht, 2013; Murayama et al., 2015). Despite the evidence for the potentially important relation of sense of agency and affective processing, most experimental research so far studied agency in non-affective contexts. The current project shall extent previous work on sense of agency, affective processing, and action regulation to investigate how sense of agency modulates the processing of affective feedback, how agency-related changes in affective processing influence our ability to self-regulate behavior, and under which circumstances affective information determines the emergence of agency experience. Answering these questions will contribute to the DFG Priority Program “The Active Self”, by elucidating affective processing as one potential core mechanism underlying the sense of self during goal-directed actions.
PIs: Prof. Dr. Simone Schütz-Bosbach (TODO), Dr. Jakob Kaiser
PhD: Maren Giersiepen
In our second phase project we will extend the results of the first phase project that investigated influences of the early caregiver-child interactions on the developing minimal self in the first two years. The second project will focus on preschool children. The preschool age is the first age that the currently most widely accepted indicators of the minimal self (sense of agency, interoception, body ownership) yield unambiguous evidence. While we accept that the self is a moment-to-moment construction and is grounded in sensorimotor processes, we postulate – following influencing theories such as attachment theory – that the minimal self is also malleable for social interactions and its consequences (e.g. attachment representations). Four studies will be carried out to test these notions in which we include methods of attachment theory (Adult Attachment Interview, Story Stem Assessment, Strange Situation Procedure), direct observation of the caregiver-child-interaction and operationalizations of the indicators of a minimal selfhood (Enfacement Illusion, Interoceptive Accuracy, Judgements of Agency, Intentional Binding) in preschool children and adults. The results will help us understand the influences of social interactions on the developing minimal self and how representations of earlier interactions affect the construction of the minimal self in the here and now.
Dynamic environments are characterized by their unpredictability. We investigate how a self might emerge when acting in these unpredictable environments. More specifically the COMPAS project aims to explore the role of sense of control (SoC) in the emergence of self and implement a computational cognitive model of the SoC in situated action.
We continuously develop an experimental paradigm to investigate what part SoC plays in decision making in dynamic environments. We also explore various metrics of eye movement behavior as possible implicit measures of Sense of Control. Additionally, we are refining a computational model of deep reinforcement learning which simulates an individual’s control experience in situated action. The computational model is based on theoretical processes underlying the sense of control. We investigate whether SoC can provide a control mechanism shaping the learning of robust and adaptive policies.
PIs: Prof. Nele Russwinkel, Prof. Stefan Kopp
PhDs: Nils Heinrich, Annika Österdiekoff
Breathing grounds the individual in the current sensory experience, which is the essence of a minimal self. Despite being a vital motor task that is performed on average 20 000 times a day, it is surprising that little is known about how breathing modulates neural and sensorimotor processes.
Emerging evidence in research with adults showed that neural oscillations entrain to respiration, and respiration modulates cognitive and affective functions. However, it is entirely unknown how these interactions work in the developing brain.
In this project, we will investigate how breathing alters neural and sensorimotor processes and how active modulation of the breath elicits short-term and long-term changes in the developing brain. With an interdisciplinary developmental approach combining cutting-edge neuroscientific methods with robotics simulations, we will provide a mechanistic explanation on how sense of self, operationalized through respiratory interoceptive sensitivity, continues to develop in early years of life. We expect our interdisciplinary project to generate groundbreaking insights into brain-body interactions in development.
PI: Dr. Ezgi Kayhan
The overall goal of this project is to provide a thorough and fine- grained characterization of disturbances of the minimal self in patients suffering from schizophrenia. Furthermore, the data from experiments in patients and healthy controls will be used to model their behavior in a neurocomputational model with the ultimate aim to extract general principles of the construction of an active self. Schizophrenia represents a severe mental disorder, in which self-disturbances are reported as a disintegration of subjectivity (often termed ‘self- disorders’, SD). Patients report a long-time persisting feeling of self- transformation, bodily self-alienation, and permeability of ego- boundaries. Symptoms include disturbed stream of consciousness, self-awareness, corporeality, demarcation, and existential reorientation, all of which are interrelated. They can be assessed in great detail using the “Examination of Anomalous Self-Experience” (EASE) instrument (Parnas et al., 2005), a semi-structured clinical interview focusing on the experiential and phenomenological anomalies of schizophrenia spectrum disorders. From a computational perspective, disturbances of the self in schizophrenia such as disturbed sense of agency (SoA) or sense of ownership (SoO) are described in terms of deficits in predicting sensory consequences, which may account for several aspects of SD. In our project, we will assess the dimensions of self-disturbance in terms of neurocognitive mechanisms linked to the subdomains of the EASE. Combining neuro-computational modelling with data from behavioral experiments and electrophysiological recordings, we will continue to carry out and extend the experimental tasks, which have been previously established by our collaborators and us within the DFG SPP “active self” in patients with schizophrenia (ICD-10: F20.0) and healthy controls. In interdisciplinary collaborations for the second funding period, we will use experimental data on impaired SoA and SoO in schizophrenia to inform the development of anatomically plausible computational models as well as a humanoid robot and focus on altered prediction of sensory input. The implementation of a disturbed sense of self resembling the symptoms of schizophrenia within a neurocomputational model and a humanoid robot is a worldwide unique approach. This “lesion model” of self-disturbances will broaden our understanding of mechanisms underlying the development of a self. Ultimately, our results will advance the understanding of schizophrenia as a disorder of the basic sense of self and promote the development of new therapies focusing on the reconstitution of the active self.
Dr. Laura Kaltwasser, Dr. Martin Voss, Prof. Dr. Fred Hamker
Our project proposal addresses the co-development of a body schema and of agency (predicting the sensory consequences of one’s own actions) in an interdisciplinary way. The interdisciplinary components are: i) Theoretical/conceptual: Based on data from human studies and basic neuro-scientific findings, we developed a concrete design concept of integral mechanisms required for explaining body ownership and agency on a neural systems level of brain function. This conceptual model has guided experimental setups to further evaluate this model together with our partners. ii) Computational neuroscience methods: The conceptual model (representing parietal cortex, basal ganglia, cerebellum, and brainstem function) will be implemented at the neural level (firing-rate, synaptic plasticity), and data obtained from the model will be compared to data from our collaborators and co-applicants. iii) Robotics: The models will be tested in simulation, and importantly, on a humanoid robot (iCub) in a real-world scenario resembling studies with humans. Although at its core our project is not a robotics project, but rather focused on model building and understanding biological principles of brain function, in the long run, the obtained results may lead to autonomous robots that possess the human-like neurocognitive architecture of a minimal self as an integral aspect of their behavior.
Projects of the First Phase of the SPP:
The (active) self is intimately coupled with sensorimotor interactions. Therefore, any theory of the self, in particular when addressing the functional role of the self, has to be developed within the paradigm of embodied cognition. Based on this paradigm, this project aims at integrating two theories that are considered to be essential for understanding the construction of a minimal self: Metzinger’s theory of embodiment orders and Tononi’s integrated information theory of consciousness. Their consistent integration will be achieved in terms of conceptual and mathematical work, thereby incorporating existing and developing new information-theoretic measures such as predictive information, morphological computation, synergy, integrated information, etc. The project will illuminate the functional role of embodied information integration, thereby relating it to prediction, regulation, and control in the agent’s sensorimotor loop. The mathematical work will be tested and guided in terms robotics experiments which will ultimately provide exemplifications of the theory.
The certainty that we are inextricably linked with our body is a fundamental feature of self-experience. Yet, the unity of body and self is surprisingly vulnerable. This vulnerability becomes particularly evident in amputees who predominantly report ongoing awareness of a phantom body part that is physically missing. Moreover, prosthetic devices can restore amputees’ body integrity and there is evidence that real ownership can be perceived for the artificial limb. There is growing evidence indicating that these perceptions involve a widely-distributed fronto-parietal network integrating body-related multisensory input in peripersonal space. Furthermore, distortions of peripersonal space representation might be causally associated with reduced body ownership sensations. However, no study so far has directly compared amputees perceiving or not perceiving prosthesis ownership. The proposed project will use the advancements that have made in the application of the well-known rubber limb illusion paradigm in which participants transfer the sensation of ownership from their real limb to an artificial limb. Based on a survey, we seek to characterize distorted bodily self-experiences in large cohorts of amputees with and without prosthesis ownership compared to healthy controls. In subgroups of these participants, we will use functional and structural magnetic resonance imaging to identify discriminating neural processes as well as connectomic signatures associated with body ownership experiences. Finally, we will experimentally extend peripersonal space representation, and will assess associated changes in ownership sensations for the prosthetic device. It is assumed that these studies will further elucidate the neuropsychological mechanisms underlying the perceived unity between the body and the self and will eventually pave the way for innovative therapeutic approaches for the treatment of disturbed bodily self-processing.
The abilities to control one’s own actions in a goal-related way and to understand the goals and intentions behind the actions of others are important aspects of the agentive self. Both aspects develop during infancy and depend on the so-far acquired own agentive experience. It has been hypothesized that cognitive representations of own actions are partially used to plan own actions and to understand the actions of others. However, when mapping observed motions onto cognitive action representations, challenging problems of correspondence, perspective, and motor inference need to be solved. Although a critical role of the mirror neuron system (MNS) is supposed, the actual encodings and computational processes involved as well as their ontogenetic development remain elusive. The planned project seeks to fill this explanatory gap by combining insights and further experimental evaluations from developmental psychology with machine learning-oriented cognitive modeling. This interdisciplinary collaboration promises benefits in a bidirectional manner: Developmental psychology will be augmented with a functional, computational model of the cognitive development of action understanding. Machine-learning and cognitive-systems research will profit from the identification of inductive biases that foster the emergence of action understanding. In eye-tracking and EEG studies with infants and driven by the modeling efforts, the project will assess in further detail which cues and cue combinations of agency (e.g., human visual appearance, self-propelledness, production of salient action-effects, own action experience) are most relevant for infants’ ability to anticipate the goals of observed actions. The computational models will combine our current biological-motion model with our theory of event-predictive cognition. The current model expects, for example, that perceptual highlighting the final goal will support anticipatory action observations. By modeling the concrete scenarios, we will also generate more concrete behavioral predictions. Overall, we expect to answer critical developmental and cognitive-science questions. For example, for which types of observed actions will eye-tracking- and EEG-derived signals for MNS activity be detectable? Do own action experiences or observations of others’ actions influence subsequent action understanding in infants of different ages? Can agency-cue augmentations facilitate the learning of computational models of action understanding? Thus, the project will contribute to the thematic focus of the SPP 2134 by interweaving predictions generated by (neuro-)cognitive modeling with insights from developmental psychology to foster understanding on how infants (i) plan and control own goal-related actions as well as (ii) anticipate action goals and infer the underlying intentions of others.
Overall, the project will shed further light on the development of the agentive self, the MNS, and the resulting social competencies.
Humans possess very sophisticated learning mechanisms that allow them, for example, to learn a sports discipline or task, and to transfer certain movement patterns and skills from this discipline to improve their performance in another discipline or task. This is possible, even if the transfer task is not immediately related to the initially learned task [33]. It is this important transfer learning ability that enables humans to solve problems they have never encountered before, in ways that are beyond the current capabilities of robots and artificial agents. However, the neural foundations of transfer learning and its role in the emergence of a self are still mostly unknown, and there exists no generally accepted functional neural model of transfer learning.
In this project, we will investigate to what extent the transfer of learning is responsible for the development of a self. We will, therefore, present a computational ideomotor approach, and hypothesize that the transfer is possible due to a hierarchical structure of action-effect associations, such that training a specific narrow low-level task indirectly trains higher cognitive skills that are involved in other low-level tasks. For example, manipulating objects and balancing are two low-level tasks that both involve mental rotation. Consequently, according to our hypothesis, the mental rotation skill will benefit from balance training, which in turn also triggers an improvement of the grasping task. In the context of the minimal self [8], the transferred patterns and skills correspond to ideomotor action-effect associations. Their transfer, therefore, constitutes an important aspect of the plasticity of the self that we refer to as ideomotor transfer.
We will address our hypothesis by implementing a computational and neurocognitively plausible neural network architecture evaluated on a physical humanoid robot. We will teach the robot to perform two different tasks, namely balancing and grasping. We select these tasks because they involve common cognitive functions like mental rotation, and there are also analogies in the physiology of the limbs that are primarily used in these tasks (e.g. elbows correspond to knees). Yet, the tasks are different enough to require a non-trivial so-called far transfer [33] between motor skills that are not immediately related to each other. We hypothesize that the transfer of these skills is internally realized by the emergence of higher cognitive action-effect associations that are relevant for both tasks. We will identify and evaluate these high-level ideomotor associations by means of computational clustering over neural activation patterns during alternating training sessions.
We expect this research to result in interdisciplinary contributions to fields related to the plasticity of selfhood and action. Specifically, we expect that a) neuroscientists and psychologists can use our model to answer questions about transfer learning and its functional relation to the plasticity of the self, b) our approach helps to anchor cognitive theories about embodiment and transfer learning in a neural active and functional model, and c) our model leads to novel approaches for intelligent robotics, where transfer learning will facilitate the integration of new sensors and actuators by reducing the amount of training required for a robot to fulfill a certain task.
The current project investigates how humans are informed by their physical body to construct a representation of themselves (minimal self). Studies in experimental psychology and cognitive neuroscience have distinguished several facets of minimal selfhood that are based on sensorimotor experience of both the outer and inner body, such as feelings of internal sensations (i.e., interoception) and of outward actions (i.e., sense of agency). Despite recent theoretical Bayesian frameworks viewing selfhood as emerging from a tight coupling between processes of motor control and autonomic control, interoception and agency have mostly been studied in isolation. Unfortunately, potential interactions between these two fundamental dimensions of selfhood have received little experimental attention so far. The current project, INTERACT, will focus on the reciprocal relationship between interoception and action by investigating established neurocognitive markers of implicit self-awareness such as sensorimotor attenuation and heartbeat-evoked potentials. In the first two subprojects, INTERACT will assess how internal physiological states associated with motor actions influence our sense of agency (subproject A), and vice versa, how having a sense of agency affects back on our visceral body awareness and homeostatic control (subproject B). The third subproject (C) will assess how these signals (i.e., inside and outside the body) and their integration allows for flexible goal-directed behavior. The planned studies will aim to achieve their aims by applying methods from cognitive psychology and neuroscience, such as electroencephalography, peripheral physiology measures, psychophysics and behavioral studies in healthy volunteers. The proposed project will form part of the groundbreaking call of the Priority Programme (SPP 2134) for investigating minimal selfhood and will address three of its five key questions. The aforementioned studies will have significant theoretical and clinical implications for the understanding of the dynamic nature of human selfhood across various disciplines. First, autonomic signals, and their continuous flow between viscera and brain, may provide an additional, powerful window of insight into the dynamics of self-construction such as personal agency, and for testing “feeling-based” self-representation in artificial agents. Second, the possibility that interoceptive signals may be integral to the functional significance of agency states for self-regulation holds the promise of better explaining individual differences or individual fluctuations in flexible goal-directed behavior and may lead to developing new therapeutic interventions. The long-term objective of this research will be to investigate the clinical implications of these findings. This latter endeavor will require clinical cooperation which shall be established in the first funding period and could become a core focus during the second funding phase of the SPP 2134.
The minimal self (MS) can be defined as the way we perceive ourselves in a specific situation, i.e., how body parts are perceived as belonging to one’s own body (body ownership), and which actions or events are produced by these body parts (agency). Recent developments in virtual / augmented reality techniques allow detailed parametric manipulation of action-sensation interactions and contingencies. Herewith, it might be an optimal method to investigate the embodiment of tools into the body schema (BS) and body image (BI), and the association between body-ownership and agency as well as their mutual dependency on action-related sensory feedback. Our project will contribute to the question of which functional mechanisms underlie the construction of the MS, and to which extent the MS is plastic and affected by sensorimotor experiences. Our proposed work program will apply an Augmented Reality (AR) approach to study short-term and long-term plasticity of the MS. First, we will focus on the analysis of short-term effects of tool-use trainings in AR. Then, we will examine the more long-term consequences of aging as a proxy for altered sensorimotor abilities and reduced involvement in and experience of specific sensory-motor skills in everyday activities. The impact on BS and BI, more specifically on perceived body ownership and agency, will be investigated for both types of plasticity, including the analysis of biosignal data (EEG). In three experiments, participants will learn to control a virtual tool, consisting of a hand-like tool mounted to a stick, held by the subject’s real hand in AR, to select and grasp target objects among distracting objects and to fit them into a target whole adjusted to the shape of the respective object. We will further complement the traditional hypothesis-driven analysis on averaged data with a more dynamically data-driven perspective: For each single trial of the virtual tool-use task, we process the related multimodal data streams and derive an online prediction of the level of body ownership plasticity. Such an online prediction of BO plasticity can be used for giving feedback to the user, or for optimizing training duration and intensity by terminating a training episode after an individual time period. For this purpose, we will calculate features from recorded kinematic data (from motion tracking and myoelectric signals) and neural data (from high-density EEG), as well as behavioral features. By studying the MS with adult and aging samples, we can learn more about how the mature human brain maintains a sense of MS while adapting to changes to its natural bodily boundaries. By systematically studying the MS in humans, we can then build upon this knowledge when building artificial systems and robots designed to develop a MS. We argue that the question of the extent to which having a body with clear physical boundaries is necessary for the MS, is indeed highly relevant for humans as well as for future robots.
PIs: Prof. Dr. Benjamin Godde, Dr. Dennis Küster & Dr. Felix Putze
Our project proposal addresses the development of a body schema and of agency (predicting the sensory consequences of one’s own actions) in an interdisciplinary way. The interdisciplinary components are i) Theoretical/conceptual: Based on data from human studies and basic neuro-scientific findings we outline a first concrete design concept of integral mechanisms required to explain body ownership and agency on a neural systems level of brain functions. This conceptual model will be a first starting point and not a complete theory. ii) Computational Neuroscience methods: The models (parietal cortex, basal ganglia, cerebellum, brainstem) will be designed at the neural level (firing-rate, synaptic plasticity) and data obtained from the model will be compared to existing data at different levels. iii) Robotics: The models will be tested in simulation and importantly, on a humanoid robot (iCub) in a real-world scenario to mimic studies with humans. Although this project is in its core not a robotics project and rather focused on model building and understanding the biological principles, in the long run, the obtained results may lead to autonomous robots that possess the human-like neurocognitive architectures of a minimal self as an integral aspect of their behaviour.
Researchers commonly identify two main phenomena as characterising a minimal Self in humans: the sense of body ownership – I feel corporal sensations as uniquely belonging to my own body – and the sense of agency – I feel being in control of my own actions. In this project, we will investigate the principles and prerequisites for the development of a minimal self from a developmental robotics perspective. We will study the computational processes that are necessary for multimodal sensorimotor predictions and adaptation to changing conditions during development. First, we will identify prerequisites and methods to provide basic skills for self-experience to artificial systems – such as the capability of the robot to detect its own body and to distinguish between self-generated movements and those generated by other individuals. We will investigate multi-modal (e.g. proprioceptive, tactile, visual and auditory) body representations and predictive capabilities – within a predictive coding framework – and their role in own-body perception. We will also investigate whether multi-modal self-perception increases cognitive abilities, for example self-other distinction skills. Secondly, we will investigate the adaptation of these processes while the artificial agent is undergoing sensorimotor experience and growing corporal conditions. In particular, simulation experiments will be carried out, where humanoid robots will be characterised by evolving morphological conditions that simulate growth and bodily changes in biological systems. Experiments on a real robot will be carried out as well with changing morphological conditions. We will evaluate our models within two different scenarios.
In the field of philosophical psychology, the relation between selfhood and intersubjectivity is currently debated. Recently, it has been stressed that the sense of self does not only include a differentiation between oneself and others, but it also contains the person’s readiness to be affected by others. The ENFACEMENT effect illustrates this readiness: In case of synchronous stimulation of the own face and the face of a partner, it has been shown that the facial features of the partner have been incorporated into the self-face representation. In the present interdisciplinary project and in cooperation with further projects within the priority program, we aim to 1) develop an experimental setting in the framework of Transformed Social Interaction and construct individualized face avatars that can be manipulated in real-time interactions with a subject and 2) to use this methodology to embed enfacement into the dynamic facial communication process in which it may occur in real life. By doing so, we will investigate the plasticity of the self-face illusion, postulating that enfacement would occur if the transmission process of intention and emotion related facial cues is successful. Additionally, we assume that the transmission process as well as enfacement itself will depend on empathetic personality traits and physical attractiveness and trustworthiness attributions to the interacting partner. Finally, we expect enfacement to facilitate emotion recognition and expression abilities.
Recent research suggests that what belongs to the self is quite malleable. The Theory of Event Coding predicts: the more features overlap between self and other, the more pronounced self-other integration. Such integration leads to increased ownership and agency over the others, and feature-migration between self and other. TEC could thus explain the malleability of self-experience.
To test TECs predictions regarding ownership, agency, feature-migration, we will manipulate the degree of feature overlap between participants and avatars and the degree of control that participants have over the avatar (movement features). Additionally we will manipulate participants’ attentional weighting and meta-control states.
“Cogito, Ergo Sum” – is it “My Thinking” makes me a Self and opposes me to others? Against the background of progressive industrial automation, artificial intelligence and collaborative humanoid robots, this age-old question now draws increasing public attention to a quite modern topic: Can robots become coequal work-related partners of men? The planned investigation skips the topic’s philosophical, ethical, political, or cultural connotations. Instead, we replace Descartes’ extremely self-centered view by introducing “Paired Abstract Automatons”. One of the principally coequal automatons we call “Self” and the partner “Other”. We base concepts like social interaction, partnership, goal pursuit, autonomy, automation, enslaving, or tool use on this theory, knowing that all these concepts need a self-other layout to become definable and that the partners require special signaling to transmit task related commands and to understand each other’s intentions.
Here, we limit the discourse to sensorimotor control and haptics in men as well as to the corresponding technical aspects in machines. In this context, we look for core mechanisms that enable discriminating between self-made and foreign-made sensory inflow. Therefore, we include controller-plant arrangements in the framework of paired abstract automatons, and assign the role of a Self to the controller and the role of an Other to the plant. By a suitable definition of controller and plant, which includes also the reverse relationship, this approach is applicable to a wide range of scenarios, e.g. two interacting machines, a human bearing a prosthesis or an exoskeleton, or a human interacting with the own limbs or even with conspecifics.
Our goal is to identify and experimentally validate a solution for haptic self-other discrimination in humans, to implement the respective solution into a robot, and to provide reciprocal signaling with a syntax that both sides can easily understand.
PIs: Prof. Dr. Karl Theodor Kalveram & Prof. Dr. Mario Kupnik
The overall goal of this project is to provide a thorough and fine-grained characterization of disturbances of the minimal or basic self in patients suffering from schizophrenia. Furthermore, the data from experiments in a cohort of patients will be used to model their behavior in humanoid robots with the ultimate aim to extract general principles of the construction of an active self. Schizophrenia represents a severe mental disorder, in which self-disturbances are reported as an incapacity to pre-conceptually grasp the meaning of the world and a loss of ‘common sense’. Patients report a long-time persisting identity void and feeling of self-transformation. Symptoms include disturbed stream of consciousness, self-awareness, corporeality, demarcation, and existential reorientation, all of which are interrelated. They can be assessed in great detail using the “Examination of Anomalous Self-Experience” (EASE) instrument (Parnas et al., 2005), a semi-structured clinical interview focusing on the experiential and phenomenological anomalies of schizophrenia spectrum disorders. From a computational perspective, disturbances of the self in schizophrenia such as disturbed sense of agency (SoA) or sense of ownership are described in terms of false active inference. Patients show reduced precision in sensorimotor predictions which may lead to sensory attenuation deficits, abnormal eye movements and altered awareness of own actions and body. Dimensions of self-disturbance will be assessed in terms of neurocognitive mechanisms linked to the scales of the EASE. We will use experimental tasks that directly correspond to the subscales of the EASE interview investigating self-awareness, bodily experiences and demarcation (self-other distinction) in patients with first-episode schizophrenia (ICD-10: F20.0) and healthy controls. Structural equation models will be used to empirically test the relationships between the different subscales of the EASE with neurocognitive testing data. Second, we will use data from a novel visuomotor paradigm investigating anomalies in SoA in schizophrenia to model the patient’s behavior in a humanoid robot. The pattern of disturbed behavior in the task will be compared to different computational models implemented into an embodied agent, in order to create a lesion model of disrupted SoA. Our project will investigate how internal constraints on the active self in schizophrenia affect sensorimotor behavior and cognitive processes. The implementation of a disturbed sense of self within a robot presents a worldwide unique approach. This “lesion model” of self-disturbances will broaden our understanding of mechanisms underlying the development of a self. Ultimately, our results will advance the understanding of schizophrenia as a disorder of the basic sense of self and promote the development of new therapies focusing on the reconstitution of the active self.
PIs: Dr. Laura Kaltwasser & Dr. Martin Voss
Across centuries, one fundamental question has occupied human thought, shaped religions, cultures, scientific practices and beyond: What is the self? Whereas some claimed that the self is a concept beyond the reach of the physical world, others argued that it is a “bundle of perceptions” (Hume, 1739). However, modern psychology and cognitive neurosciences, largely thanks to advances in research methods, have attempted to ground the self to the body, and considered embodiment as the starting point for a science of the self. Our proposal capitalizes on these attempts to bring a theoretically grounded neuroscientific perspective into the investigation of the self, by addressing the formation of self through embodied interactions with others early on in development.
In the proposed project, we aim to conduct a longitudinal investigation into the emerging sense of self in infancy, operationalized through interoceptive sensitivity, in relation to embodied interactions with the primary caregiver early on in life. By examining the developmental aspects of the self, across three time points, we aim to answer the question of how the minimal self is constructed over the first one and a half years of life through interactions. Grounding our investigation in the recent interpretations of Predictive Processing framework (Fotopoulou & Tsakiris, 2017), we test the hypothesis that one of the core mechanisms that allows the infant to construct a dynamic model of minimal self is the social and embodied interactions with the caregiver.
In the first work package, we will longitudinally investigate the sense of self, operationalized through interoceptive sensitivity, when the infant is 3, 9, and 18 months of age. This will allow us to examine the developmental trajectory of interoceptive sensitivity in the first one and a half years of life. In three experiments, we will examine the behavioral and neural markers of interoceptive sensitivity in young infants.
In the second work package, we will investigate interoceptive sensitivity of the infants’ mothers by examining the behavioral and neural markers of their cardiac interoceptive sensitivity. We hypothesize that caregivers who can better detect their interoceptive signals should be more sensitive to the infant’s needs, and consequently make the environment more predictable for the infant, which would then support the development of interoceptive sensitivity of the infant.
In the third work package, we will conduct a thorough investigation into the embodied interactions between the mothers and their infants in order to define the behavioral, physiological and neural mechanisms that contribute to the interoceptive sensitivity of the infants, thus, supporting the formation of minimal self. In order to measure the neural aspects of the interaction process, we will use an innovative research method named hyperscanning in which the brain activity of two people is measured in a naturalistic interaction (Babiloni & Astolfi, 2014). This method will allow us to investigate the dynamic brain activity during a real-time interaction process.
PI: Dr. Ezgi Kayhan
This project seeks to develop a first computational cognitive model of the predictive active self in situated action. The goal is to develop a model of the execution and control of situated action in an embodied cognitive architecture that allows for (a) detailed explanation of specific hypotheses about the mechanisms and processes underlying the sense of agency and the active self; (b) simulation of situated action in different environmental contexts along with predictions about the subjectively perceived sense of agency, (c) empirical validation and model fitting through comparison with empirical data obtained from experimental studies with human participants in identical settings. To that end, the project has two main objectives: First, to develop an integrated computational architecture for embodied cognition and situated spatial action and to test it against human behavioral data in successive experimental settings. Second. to realize within this architecture a computational concept of the sense of agency and the active self and its role in online perception and control of situated action in a spatial environment. The modeling effort will be complemented by empirical studies, both with the model in simulation as well as with human participants, to inform and validate the computational model and its predictions obtained in simulation. This will include studies on situated actions under different kind of sensory feedback (predictable, predictably distorted, unpredictably distorted or fully novel) as well as with different kinds of action complexity (simple vs. complex action structure, single vs. dual tasks). Finally, the resulting computational framework shall be made available to other researchers within and beyond the SPP “Active Self” to provide a platform for testing hypotheses and help integrate communities from cognitive science, artificial agents, and robotics.
PIs: Prof. Dr. -Ing. Stefan Kopp & Prof. Dr. -Ing. Nele Rußwinkel
Abstract: Our actions typically aim at distal effects beyond our own body, for example, when we manipulate a real or virtual tool (‘remote effects’ according to James, 1890). At the same time our body movements normally produce ‘resident’ effects at the body. There can discrepancies between remote and resident effects, for example, when the seen movement of a tool contradicts the felt movement of the hand. Such discrepancies strongly impair action production. We aim to study how such discrepancies impact explicit and implicit measures of agency and ownership regarding distal events, thus the integration of such events into the minimal self. We assume that body-related representations are suppressed to the extent they interfere during action production. We aim to reveal the properties and consequences of such ‘body-related suppression’, such as its strategic components, and its impact on the processing of body-related signals. The project will thus specify an important mechanism that supports the emergence of an active self.
High-order body representations, such as body schema, encode metric properties of the body as well as its movement dynamics and play a crucial role in sensorimotor control. Across the lifespan, our body underlies gradual changes in size and weight which demand the continuous updating of any embodied self-representations. Predictive mechanisms, important for adapting balance control to internal and external constraints, are grounded on experience-dependent presumptions and as experience alters our internal body representations, any predictions based on these representations will be altered as well. In daily life, when the stability of body balance is compromised, a dependency on interpersonal balance support may emerge. During physical contact with another individual, we experience a clear distinction between self and other, perhaps based on representations of our own and the other’s body dynamics. It seems, therefore, reasonable to assume that in the process of keeping light touch with another individual, we create representations of the partner’s body dynamics in addition to our own. We believe that tactile interactions between two individuals for postural stabilization will provide strong evidence for the existence of mutual body representations, based on predictive coding mechanisms, that lead to spontaneous interpersonal postural coordination phenomena, such as interpersonal sway entrainment. In this proposed 36-month, interdisciplinary research programme, we will, therefore, observe in healthy individuals the lifespan development of any embodied selfrepresentations for balance control. We will record and analyse the body kinematics and forces acting in single participants as well as in pairs of participants to characterise the control of body sway during quiet upright standing with light interpersonal touch. Research approaches in the domains of human psychology and movement science will merge with robotic engineering to develop a computational model of probabilistic self-motion representation that predicts the own sway dynamics and balance state as well as those of a contacting partner. We will use the model to simulate natural selfother distinctions during light interpersonal touch. We will test the computational model’s validity against human nature by implementing an adaptive robotic setup for the provision of light haptic support to human individuals. An accurate robotic model will result in the emergence postural coordination between human and robot that resemble the features of “weakly” coupled, spontaneous humanhuman interpersonal postural interactions. Finally, we will extend the robotic system into a haptic, postural “Turing test” for challenging human individuals’ ability to use haptic feedback for the phenomenological distinction between a real human or a simulated partner during haptic interactions for balance support.
PIs: Prof. Dr. Dongheui Lee, Dr. Leif Johannsen
In addition to linguistic information, natural human speech contains extralinguistic cues about the speaker’s identity and acoustic spatial cues concerning the speaker’s location. It remains poorly understood how extralinguistic and acoustic spatial cues contribute to perceived self-ownership of voice. In order to systematically investigate the influence of these non-linguistic cues on selfvoice perception, the current project will develop novel paradigms of dynamic 3D Audio feedback. Behavioural and non-invasive electrophysiological measures will be recorded from adult human volunteers.
PI: Dr. David Magezi
Influential developmental theories have proposed that infants’ development of an implicit or active self is supported by their parents’ behavior. Moreover, it has been claimed that the active self is the developmental foundation and precursor of the emergence of a conceptual self at the end of the second year of life. The current project aims at examining both proposals. To this end, one longitudinal study is proposed that ranges over the first two years of life. It will be explored whether and which parent-child interaction characteristics relate to the emergence of the active self. In addition, it will be examined whether the active self that emerges in the first year of life relates to the development of the conceptual self at the end of the second year of life.
PI: Prof. Dr. Markus Paulus
The objective of this project is to understand the bidirectional link between the minimal self and sensorimotor as well as cognitive skills from a developmental embodied cognition perspective. By integrating cognitive, developmental and movement science, we tackle one of the ultimate goals of the SPP call by exploring how core mechanisms (i.e., internal models, reafferences) generate a self in the course of ontogenetic development (i.e., across the lifespan) or the acquisition of expertise (i.e., training sensorimotor and cognitive skills). Specifically, we will answer one of the five questions of the SPP call: “How does creating and having a self work back on sensorimotor skills and cognitive processes?” From a developmental embodied cognition perspective, this question cannot be answered without considering the bidirectionality between the minimal self and sensorimotor and cognitive skills. Theoretically, we propose the mechanism that optimizing the internal model by integrating sensorimotor reafferences and cognitive feedback will lead to a stabilization of the minimal self. In turn, the stability of the minimal self will foster sensorimotor and cognitive skills. To scrutinize this mechanism, we will conduct two large-scale studies. In Study 1 we will investigate with young children to adults how sensorimotor and cognitive skills influence the minimal self. In Study 2 we will investigate how manipulating the minimal self influences sensorimotor and cognitive skills. Our theoretical contribution will be an empirically tested mechanism of the minimal self, namely the optimization of the internal model through the use of sensorimotor reafferences and cognitive feedback. This mechanism will add to the development of an integrative theoretical framework. In turn, our minimal-self tests, kinematic data from movements in complex movement tasks can promote the turing test, which can be tested in robots.
“Phenomenal experience in the here and now” and “perception of ourselves to be in a particular situation” are two facets of the active self that is ultimately part of consciousness, of the “feeling of what happens”. A prerequisite to consciousness is intentionality, the capacity of humans and other animals to have mental states that are “about” things in the world. The active self, we argue, requires intentionality of two directions of fit and six psychological modes. World-to-mind, the direction of fit of the motoric flavor of intentionality, includes intention-in-action (ongoing movement behavior), prior intention (planned actions that are not yet initiated), and desires (orientation to outcomes that motivate action). Mind-to-world, the direction of fit of the perceptual flavor of intentionality, includes perception (immediately available), memory (available through retrieval), and beliefs (linking percepts, actions, and outcomes).Body ownership, a core signature of the self, entails perception of body parts, and memory to enable their recognition. Agency, a second core signature of the self, entails intention-in-action to generate movement, but also prior intention to generate sequences of actions toward a goal. Agency thus also entails desires of particular outcomes. Contingency learning may be construed as the establishment of beliefs that link percepts and actions to outcomes.Our goal is to provide a process account for how the sensory-motor grounding of intentionality can give rise to an active self. Within the theoretical framework of Dynamic Field Theory, intentional states are modeled as self-stabilized patterns of activation linked to the sensory and motor surfaces in embodied dynamic architectures. In a developmental perspective we will study visual exploration, exploration of actions and contingencies, and imitation. To account for patterns of looking that embody curiosity, visual recognition, and recognition of infants’ own body, we will model looking behavior based on the psychological models of intention-in-action, perception, and memory. Prior intention, desire, and belief will be implemented in a process model of contingency learning. Second-level intentionality in which intentional states are formed about the systems own internal representations will be used to account for imitation. On the hypothesis that only actions within an infants’ behavioral repertoire are imitated, mental simulation will be shown to be critical to deferred imitation and self-agency, the intentional state of experiencing the self as the cause of one’s actions. Mental imagery will be shown to provide the substrate for self-perception.We will examine and illustrate the process models through simulation. As a heuristic device, we will implement models on a vision-based robot to examine the extent to which they capture all necessary processes. Comparison to developmental experiments will subject the theoretical accounts to empirical test.
PI: Prof. Dr. Gregor Schöner
Associated Projects:
If an actor takes the perspective of an avatar (or robot), spatial dissociations can arise between the perspective of the actor and the perspective of the avatar. Well-known cognitive phenomena (such as spatial stimulus-response compatibility) are predicted to be influenced by the actor’s identification with the avatar. The present project aims at the conditions in which an actor adopts avatar’s perspective and asks to what extent the actor identifies with the avatar, integrates him into his/her body schema, or even becomes one with him. Thus, our goal is to gather further information about concepts like the minimal self, agency, embodiment and ownership.
PI: Prof. Dr. Jochen Müsseler
“Predictive Robots” is a Marie Skłodowska Curie Individual Fellowship funded by the EU-H2020 Research and Innovation programme (2018 call, grant agreement No. 838861).
Within this project, models of infants development and of brain processes are applied into artificial systems. In particular, this project investigates: (1) predictive models for adaptive robot behaviours, where the interplay between online learning, episodic memory mechanisms, deep neural networks, and predictive processes is exploited for implementing intelligent exploration behaviours in robots; (2) predictive processes for enhancing robot perception, where raw sensory input are combined with sensory predictions in order to produce more informative data; (3) predictive processes as a prerequisite for an artificial Self, aiming at providing insights in the understanding of the mechanisms behind subjective experience and in its possibility in robots.
The aim of the project is also to apply these computational models onto different robotic platforms, including humanoids, marine drones, and soft robots.
The project is hosted by the BioRobotics Institute of the Scuola Superiore Sant’Anna (Pisa, Italy) at Prof. Cecilia Laschi’s lab.”
SELFCEPTION is an interdisciplinary project that combines robotics and cognitive psychology. I investigate brain-inspired computational models for i) building robots that learn to recognize their own body for improving interaction and ii) investigate the self construction in humans using synthetic models.
Recent evidence suggests that self/other distinction will be a major breakthrough for improving interaction and might be the connection between low-level sensorimotor abilities and voluntary actions, or even abstract thinking. The project follows the hypothesis that the “sensorimotor self” learning will permit that humanoid robots could distinguish between the machine and the other agents during interaction. For that purpose, SELFCEPTION proposes combining advanced sensorimotor learning with new multimodal sensing devices, such as artificial skin, in order to permit the robot to acquire its perceptual representation.
The project pursues the materialization of the next generation of perceptive robots: multisensory machines able to build their perceptual body schema and distinguish their actions from other entities. We already have robots that navigate and now it is the time to develop robots that interact.