\documentclass{article}
\usepackage{fullpage}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage{xcolor}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage[natbibapa]{apacite}
\usepackage{eso-pic}
\AddToShipoutPictureBG{\AtPageLowerLeft{\includegraphics[scale=0.7]{powered-by-Authorea-watermark.png}}}

\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}

\usepackage{authblk}

\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}


\iflatexml
% \documentclass[]{sig-alternate-05-2015} % Not (yet) supported by LaTeXML.
\fi
\usepackage[]{cite}

% \usepackage[]{microtype} % Not (yet) supported by LaTeXML.

\usepackage[hidelinks]{hyperref}

\usepackage[]{graphicx}

\usepackage[]{booktabs}

\usepackage[]{paralist}

% \usepackage[]{algpseudocode} % Not (yet) supported by LaTeXML.

\usepackage[]{color}
\usepackage[]{xspace}

\newcommand{\eg}[][]{e.g.,\xspace}

\newcommand{\etal}[][]{et al.\xspace}

\newcommand{\ie}[][]{i.e.,\xspace}

\newcommand{\etc}[][]{etc.\xspace}

\newcommand{\vs}[][]{vs.\xspace}

\newcommand{\resp}[][]{\textit{resp.}\xspace}
\newcommand{\alone}[][]{\textsf{alone}\xspace}

\newcommand{\human}[][]{\textsf{human-presence}\xspace}

\newcommand{\robot}[][]{\textsf{robot-presence}\xspace}
\newcommand{\C}[1][]{\textbf{c#1}\xspace}
\setlength{\paperheight}{11in}
\setlength{\paperwidth}{8.5in}
\usepackage[
  pass,% keep layout unchanged 
  % showframe,% show the layout
]{geometry}
\graphicspath{{figs/}}


\begin{document}

\title{This Love Affair of Ours with Psychology}


\author[ ]{Séverin Lemaignan}

\affil[ ]{}
\vspace{-1em}


\date{}

\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup

\selectlanguage{english}
\begin{abstract}
The field of Human-Robot Interaction lies at the intersection of several
disciplines, and is rightfully perceived as a prime interface between
engineering and social sciences. In particular, we entertain close ties
with social and developmental psychology. Examples of building upon
classic results from psychology to explore the intricacies of
human-robot interaction are countless; the methodologies and insights
that psychology brought to HRI are invaluable.

This article discusses, however, the less positive side of this `love
affair'. Building on failed attempts to replicate well-accepted
experiments on social facilitation, we discuss our possible
over-reliance and somewhat blind acceptance of classic results in
psychology. Firstly, we suggest that the Human-Robot Interaction
community should transform the `love affair' with psychology into a
regular `business relationship', and secondly that our research field
does not need to shy away from developing its own reference tasks.%
\end{abstract}%



\textbackslash{}setcopyright

acmcopyright

\textbackslash{}doi

10.475/123\_4

\textbackslash{}isbn

123-4567-24-567/08/06

\textbackslash{}conferenceinfo

HRI'17,March 6--9, 2017, Vienna, Austria.

\textbackslash{}numberofauthors

1

\section*{Introduction: Our Love Affair with
Psychology}\label{introduction-our-love-affair-with-psychology}

The field of Human-Robot Interaction, and in particular, the field of
social HRI is nurtured by the weaving of different scientific
perspectives. As a community, we recognise that the technical fields of
engineering, control theory or computer science do not provide much
tooling for the scientific investigation of the `Human' and
'Interaction' parts of HRI. For this reason, we take inspiration and
ground much of our research in established results from social sciences
-- in our field, primarily social psychology, developmental psychology,
and sociology. As academics, we pride ourselves on standing at the
intersection of these many fields, being able to understand and be
understood by programmers, engineers, as well as psychologists. In this
sense, our field embodies the basic idea of cognitive sciences: building
bridges across disciplines to gain new insights on complex scientific
challenges.

That said, the demographics of the academics working in HRI are skewed
towards engineering background TBD: any data to support that? We could
try to go over last year HRI's author list, and quickly check the
backgrounds?: one often becomes a researcher in HRI by first building
robots and then looking at how the machines might interact with humans.
While some of us do have a primarily academic background in psychology,
many do not. This is not per se an issue: as capable, rigorous
scientists, we can read and understand the literature of social science,
and take inspirations or reproduce tasks, protocols, results. This is
actually how science is supposed to work.

We think however that a `second order' effect might be underestimated:
because many of us are `consumers' of the psychology literature rather
than `producers' and active contributor to the psychology community, we
might not always share the same common-grounds with these neighbouring
academic fields.

This has two consequences: first, as we are generally less familiar with
different social science academic communities, we tend to be less
critical and we would not automatically question their findings as we
would in our own community. This effect is reinforced by the perceived
maturity of academic fields like social or developmental psychology,
versus the youth of human-robot interaction.

Second, we build assumptions on how research is conducted in other
communities based on our own experience. As our background is often in
exact sciences, we would intuitively expect evaluation methods to
deliver as much as possible robust, exact, clear-cut results. Results
that are always reproducible. And we are certainly embarrassed whenever
our results do not draw such a clear, legible picture.

It is reasonable to think that, consciously or not, we assume the same
`exact science' mindset across all scientific discipline, including
social sciences. Accordingly, we do not tend to question their
established results.

However, over the last years, a growing evidence has built up that show
that many of these `classical', `established' results from social
sciences -- psychology in particular -- might not be as solid and clear
as assumed: to much ado, a recently published
article~\cite{rpp} evidenced the difficulty to
reproduce several key results from psychology: out of 100 studies
sampled from a range of psychology sub-fields, only 36\% of the
replication studies found significant results whereas 97\% of the
original studies reported significant effects. The results of two third
of 100 studies could not be properly replicated: whatever the reasons
might be (from publication bias, to sociological changes in the
population, to small effect sizes), it calls for exerting caution
whenever we build upon supposedly established results.

Mention that findings true 50 years ago might not be true anymore today?

We are convinced that many researchers in HRI are already aware of these
issues. The purpose of the article is to make this concern (``we rely
too much and too blindly on results from psychology that might not
hold'') surface within the HRI community. To illustrate the point, we
present hereafter two unsuccessful attempts at reproducing the effect of
social facilitation in presence of robots: social facilitation is an
effect where the mere presence of a (silent, passive) external agent
influences one's behaviour: the participant would perform better or
worst, depending on the task and on the expectations. A large body of
psychology literature evidence this effect, that has been studied in
robotics as well. However careful our experimental design was, we have
not been able to reproduce the effect.

\section*{A Case in Point: Social
Facilitation}\label{a-case-in-point-social-facilitation}

\subsection*{Context: Studying the Mere Presence Effect in Social
Facilitation}\label{context-studying-the-mere-presence-effect-in-social-facilitation}

Background and Related Theories

In 1898, Norman Triplett~\cite{Triplett1898} observed
that cyclists pedal faster in presence of rivals than when they are
alone. He studied this effect on children by using a fishing reel that
they need to turn as quickly as possible and found the same effect,
although a later analysis of his work by
Stroebe~\cite{Stroebe2012} showed that there was no
significant difference in his findings. This effect has later been
termed as `social facilitation' by
Allport~\cite{Allport1924} to describe the increase in
response due to the presence of others who are performing the same task.
Social facilitation occurs by two types of condition: `co-action
effects' like Triplett's examples, and `audience effects', in which only
the mere presence of an observer effects the performance of a person
performing the task. In order to explain the audience effects,
Zajonc~\cite{Zajonc1965} proposed the drive theory,
which states that the audience enhances the exhibition of dominant
responses in a person. In the case of a well-mastered task (`simple
task'), the performance is facilitated, whereas, for the tasks that are
new or require learning (`complex tasks'), the performance is inhibited.

Many other theories underpin social facilitation: anticipation of
evaluation~\cite{Cottrell1972}, attentional
conflict~\cite{Sanders1975}, monitoring
~\cite{Guerin1983},
self-awareness~\cite{Duval1972} and self-presentation
~\cite{Bond1982}.

Factors

A meta-analysis by Bond and Titus~\cite{Bond1983}
compares 202 published and 39 unpublished studies on social
facilitation. They provide a list of 13 factors that might impact social
facilitation (like the participants' age, the number of observers, the
role of the observers, the familiarity of the observers, etc.). The
meta-study shows that the performance speed (`quantity') is increased
for the simple tasks and the performance accuracy (`quality') is
decreased for the complex tasks. The performance quantity is measured by
the latency to respond, time it takes to complete a task and the number
of responses per unit time. The performance quality is measured by the
number of errors. The study also shows that the visibility of the
observers to the subjects has a slightly bigger effect than the
non-visibility, although the difference was not significant. Examples
from the literature are use of a one-way
mirror~\cite{Ganzer1968, Criddle1971}, use of a video
camera~\cite{Geen1973,Terry1993}, or a desktop image on
a computer screen~\cite{Gardner2008}. On the other
hand, Bernard Guerin~\cite{Guerin1983} argues in a
review that the majority of studies with social facilitation effects had
observers or the experimenters watching the subject while the task was
performed, that is, they were not busy with other tasks. However, he
states that there should be some uncertainty if the observer will
evaluate the subjects, which meant that there should be evaluation
potential which the subject should not know beforehand. He also draws
attention the ceiling and floor effects for the tasks. He states that
the task should be difficult enough so that the measurements can vary
between subjects and conditions.

Tasks

Following Zajonc, the literature on social facilitation distinguish
between `simple tasks' and `complex tasks'. Examples of simple tasks
include letter cancellation, multiplication, and vowel cancellation;
examples of complex tasks include concept formation, anagrams, digit
span, and pursuit rotor tasks. Tasks such as letter-copying and paired
associates can be either simple or complex depending on the task
structure. McCaffrey et al.~\cite{McCaffrey1996} also
presented significance levels of each of these tasks in the literature.
It shows that visual perception and construction tasks such as
letter/word copying~\cite{Guerin1989, Terry1993,
Gardner2008} and motor tasks such as physical
activities~\cite{Strube1981} are good tasks in terms of
significance as simple tasks, and memory or learning tasks such as
paired associates~\cite{Cottrell1967, Geen1973,
Guerin1983} and visuomotor tasks as in rotary pursuit
performance~\cite{Lombardo1975, Miller1979} have higher
significance for social facilitation as complex tasks.

Cheating as a reinforcing factor

Self-presentation theory~\cite{Bond1982} also suggests
conformity to normative behaviours to gain approval of another person.
For example, in the case of an embarrassing situation such as cheating,
this should prevent the subject from engaging in the cheating behaviour
due to social pressure. There might be several factors that effect
cheating behaviour as examined by several researchers, such as the
importance of the task, risk of being caught, probability of
success~\cite{Vitro1972}, belief in
free-will~\cite{Vohs2008}, knowledge of peer
performance~\cite{Hill1969}, higher gain of money or
grades, penalty~\cite{Nagin2003} or conformity to
cheating behaviour in peers~\cite{Fosgaard2013}. In the
study by Vohs and Schooler~\cite{Vohs2008}, in order to
observe the cheating behaviour, a computer-based mental arithmetic test
was used. The participants were made aware that there is a glitch in the
program which shows the correct answer to the problem but they could
close the answer window by pressing a key after the problem appeared.
They were also told that the experimenter would not know whether they
pressed the bar, even though the number of presses was recorded, but
they should try to solve the problems honestly. Their results revealed
that those who were given an essay prior to the test that stated the
lack of free-will cheated more frequently than others.

Social Facilitation In Robotics

The audience effect has been studied in HRI by
Schermerhorn~\cite{Schermerhorn2008} and Riether et
al.~\cite{Riether2012}. Schermerhorn compared the
effect of the robot presence during easy and difficult arithmetic tasks
with alone and robot-presence conditions. A significant two-way
interaction between gender and robot was found, because the male
subjects performed worse during the difficult task when the robot was
present. Overall, a marginally significant effect of robot presence was
found. Riether et al. on the other hand, compared alone, human-presence,
and anthropomorphic robot-presence conditions with four different tasks
with easy and complex conditions: anagram solving, numerical distance,
finger tapping and motor reaction task. They observed that in the
anagram solving, numerical distance and finger tapping tasks, there were
significantly larger difference scores than the alone group for both
robot and human conditions, but there was no significant difference
between the robot and the human observer conditions. This suggests that
people regard robots as social beings. After the experiment, the
subjects were asked to fill a survey in which they gave higher
observation impression scores for the robot condition than the human
observer, implying that they thought someone else was watching through
the eyes of the robot or due to novelty effects leading to distraction.

Following the findings from social facilitation literature, we decided
to explore the mere presence of two robotic platforms (the Aldebaran NAO
and Pepper) through a social facilitation task. We anticipated that
there would be a difference between the two platforms due to their
appearance that would be useful to make other HRI researchers aware of.
While the studies aimed to compare the social facilitation of two
different robots, it was important to establish two baselines first: one
with no observers, and one with the social facilitation elicited by a
human. This would essentially be a first step in replicating the finding
from psychology. Assuming this replication was successful, the robots
could then be tested to see how they compare both with each other and
our psychology-inspired baselines.

We ran two distinct studies, with a total of three different tasks.
Because no effect could be found between the alone condition and the
human condition in any of our tasks, we did not actually pursue the
studies to include the robots.

\subsection*{Social Facilitation: First
Attempt}\label{social-facilitation-first-attempt}

The first study was run between-subjects with two conditions: alone
condition against human-presence. Participants were recruited on the
university campus and taken to a room in the campus library for the
experiment. The experimenter would take the participant to the room and
tell them to follow instructions on the tablet, then the experimenter
would leave. In the human observer condition, a second experimenter
would already be sitting in the room and would remain there for the
duration of the experiment (as per
Figure~\ref{fig:setup}).\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/setup/setup}
\caption{{Layout of the experiment rooms. The participant (\emph{A}) is sitting at
a table, with their back to the door. The tasks are performed on a
tablet (\emph{B}). When present, the social agent (human observer) is
placed at \emph{C}.%
}}
\end{center}
\end{figure}

Tasks

The literature distinguishes between the effects of mere presence on
simple tasks and complex tasks. We sought to elicit differences in both
of these task types. Each participant therefore performed two tasks,
followed by a brief questionnaire. Both the tasks and the questionnaire
were administered on the tablet. The first task was designed to be a
repetitive visuomotor task (the `shape matching' task); the second one
required recollection and comprehension of spoken information (the
`story' task). As such, we examine the effect of social facilitation on
both low- and high-cognitive tasks.

The `shape matching' task is a game where the participants are asked to
match a colourful target shape with another one, of the same shape, but
a different colour (Figure~\ref{fig:shape-matching}).
The target shape as well as the eight possible responses are random
combinations from the sets \{red, yellow, green, purple, blue, white\}
and \{square, cross, star, circle\}. After the participant touches a
shape to select it as an answer, a new random set is shown on screen.
This is repeated 200 times. By using the same random seed for all
participants, the stimuli sequence was kept identical for all
participants.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/shape-matching/shape-matching}
\caption{{Screenshot of the shape matching task. Participants are instructed to
tap on the picture matching the target's shape (seen at the top), but
with a different colour. In this example, the participant has to tap the
green square.%
}}
\end{center}
\end{figure}

The task can be repeated for up to 200 rounds of random shapes. After 75
rounds, a button labelled ``Give up'' appears on screen, giving the
participant the option to skip to the second task. The wording of the
label was intentionally chosen instead of a more neutral ``Stop'' or
``Continue to next task'' to elicit a stronger social response (``Giving
up'' being more socially costly than simply ``continuing to the next
task''), thereby increasing the contrast between conditions
(self-presentation effect). During this first task, we record three
metrics: the reaction time for each round, the number of correct and
incorrect responses, the total number of rounds completed (dependent on
when the ``Give up'' button was used). We also asked the participants to
give an estimate of how many rounds they thought they had completed,
between 0 and 300 and if they felt like they were being observed during
the test.

We need to give the results on the answer of being observed in the
questionnaire, I remember that we said they were felt like they were
being observed in the alone condition as well (for both studies), that
would be a good argument for the Hawthorne effect

The second task (`story' task) involves listening to a short
pre-recorded text (1min 56sec) and answering eight questions about this
text. The text11Recording and transcript available on-line, at {[}hidden
for blind review{]}. details the history of a fictional country named
``Brookland'' and includes a range of facts: names of places
(``{[}they{]} sailed to Port Danford''), dates (``Springland was settled
in the year 2503''), terminology (``Settlers or 'squatters' began to
move deeper into the territories''), and situations (``Women were
outnumbered five to one''). The text was based upon the settling of
Australia, but with key details and place names changed. This was so
that the information would certainly be novel, without sounding
implausible. The eight multiple-choice questions are administered
immediately after the end of the text. Each question provides a choice
of four answers (Figure~\ref{fig:text-questions}). The
score of each participant (number of correct answers) is the performance
metric for this task.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/text-questions/text-questions}
\caption{{Screenshot of four of the eight questions administered immediately after
listening to a short text about a fictional country.%
}}
\end{center}
\end{figure}

Hypotheses

Based on Zajonc's drive theory~\cite{Zajonc1965}, our
hypotheses were the following:

\begin{itemize}
\tightlist
\item
  In the `shape matching' task, the presence of a human would lead to
  \textbf{better performance}: fewer mistakes, faster reaction times.
\item
  In the presence of a human, the `Give up' button would be used less
  frequently (or later in the game) due to the social pressure
  (self-presentation theory).
\item
  In the presence of a human, participants would perceive that they
  complete fewer rounds of shape matching than they actually do, due to
  social facilitation.
\item
  In the `story' task, the human presence would lead to the impairment
  in performance such that the participants will recall less facts.
\end{itemize}

Protocol \& Data Collection

We recruited 45 participants after exclusion (25 for the alone condition
and 20 for the human condition, 16 males, 29 females, balanced across
conditions) on campus. The participants age was M=20.36 (SD=2.53). We
ensured that all participants who enrolled were not colour-blind (due to
the necessity of seeing colour accurately for the shape matching task)
and that they were native English speakers (to prevent comprehension
issues due to language in the story task).

Participants were first given information sheets describing the
experiment (simply entitled ``Learning with a touchscreen'', so as not
to disclose the role of the mere presence of the observers). They then
gave consent to participate, compliant with the university ethics
committee rules. Participants were told in writing and verbally that
whether or not they decided to withdraw early from the study, they would
receive compensation equivalent to EUR6 (as an Amazon voucher). We made
this point explicit to make sure the participants knew that, even if
they quit the shape matching game early (i.e., between rounds 75 and
200), they would still receive the full compensation amount.

Results

We did not observe any difference between the two conditions concerning
the number of time required to reach the limit of the 75 shapes, average
reaction time, number of shape completed, ratio of correct matching or
recall performance (cf. Table \ref{tab:results1}).

This means that we do not observe any social facilitation effect in this
study. H1 is not supported: the presence of a human by-stander does not
impact the performance for the simple task (no significance difference
between number of mistakes and reaction time). Neither is H2: the
'Give-up' button is similarly not used in both cases. Similarly, the
presence or absence of a human did not impact the estimation ratio
(ratio of perceived shape matching done by the real number) nor the
recall of facts, invalidating both H3 and H4.

Results for the shape matching task. No significance has been observed
on any of the six metrics reported: time to reach 75 shapes, average
reaction time, number of shapes completed, ratio of correct shape, ratio
of perceived matching and recall performance.

Metric

Alone condition \textbackslash{}(M(STD)\textbackslash{})

Human condition \textbackslash{}(M(STD)\textbackslash{})

\textbackslash{}(p\textbackslash{})-\textbackslash{}(value\textbackslash{})

Time to 75 shapes (s)

\textbackslash{}(117.7(30.01)\textbackslash{})

\textbackslash{}(110.63(17.25)\textbackslash{})

\textbackslash{}(.349\textbackslash{})

Average reaction time (s)

\textbackslash{}(1.70(0.47)\textbackslash{})

\textbackslash{}(1.58(0.26)\textbackslash{})

\textbackslash{}(.305\textbackslash{})

Number of shapes completed

\textbackslash{}(196(11.5)\textbackslash{})

\textbackslash{}(198(7.8)\textbackslash{})

\textbackslash{}(.522\textbackslash{})

Ratio of correct shapes

\textbackslash{}(0.98(0.02)\textbackslash{})

\textbackslash{}(0.99(0.01)\textbackslash{})

\textbackslash{}(.082\textbackslash{})

Ratio of perceived matching

\textbackslash{}(0.52(0.34)\textbackslash{})

\textbackslash{}(0.50(0.32)\textbackslash{})

\textbackslash{}(.88\textbackslash{})

Recall performance

\textbackslash{}(4.81(1.27)\textbackslash{})

\textbackslash{}(5.11(1.49)\textbackslash{})

\textbackslash{}(.473\textbackslash{})

Explain how our design decision could have impacted the results maybe
later

\subsection*{Second Attempt}\label{second-attempt}

Reflecting on the lack of effect observed in our first attempt, we
designed a second experiment to address the possible failures of the
first one.

Specifically, we chose to have the human observer closer to the
participant (aiming for greater human influence), a stronger moral
component (aiming for a greater influence of the human presence), a more
difficult task (stronger incentive for behavioural differences -- i.e.,
cheating -- between conditions), money reward dependent on performance
(stronger, clearer incentive for behavioural differences between
conditions) and finally, regarding the methodology, we decided to move
away from primarily using reaction times as metric, so as to avoid any
natural performance limit.

Task

Based on these constraints, we designed a new task involving mental
arithmetic. Participants were required to calculate the result of a set
of non-trivial mental additions. The additions each had exactly three
2-digit numbers to sum, one carry (a digit that is transferred from one
column of digits to another), and their results ranged from 100 to 200.
Participants had 5 minutes to perform as many additions as possible.
Each correct answer would earn them a small financial reward of \selectlanguage{ngerman}£0.20
(Figure~\ref{fig:sums}).

Critically, following the design of Vohs and
Schooler~\cite{Vohs2008}, a supposed `glitch' was
showing a pop-up dialogue before each addition. This dialogue was
designed to look like a spurious debug dialogue and contained the
expected answer. The participants were explicitly shown by the
experimenter that the correct answer was erroneously displayed in the
dialogue. They were instructed to ignore the dialogue and to dismiss it.
This `bug' was explained to the participant as being caused by a new
operating system on the laptops used for the test (``Our previous
computers did not have this issue''). The bug made it practically easy
for participants to cheat: by briefly glimpsing at the debug dialogue
before dismissing it, they could immediately know the correct answer,
and earn money faster.

The dialogue could be dismissed by pressing `enter' on the keyboard.
`Enter' was also the key used to move to the next question. As such, a
double-press would move to the next question and close the dialogue
before it could be seen. Through this mechanism, it was possible to
measure how long it took participants to close the dialogue, and infer
whether they had cheated.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/sums/sums}
\caption{\selectlanguage{ngerman}{Screenshot of the sums task. Participants have 5 minutes to perform as
many mental additions of three numbers (with a result between 100 and
200) as possible. Each correct answer earns £0.20. A pop-up dialogue
with the answer is shown before each new addition. The participants are
instructed that this pop-up is a bug and that they should ignore it and
dismiss it.%
}}
\end{center}
\end{figure}\selectlanguage{ngerman}

Hypotheses

The literature ~\cite{Feinberg2010, Wechsung2014}
suggests that social presence during a complex cognitive task like this
one should lead to worse performance. Accordingly, our hypotheses were
the following:

\begin{itemize}
\tightlist
\item
  In the presence of a social agent, participants will be more honest
  (i.e., they will look at the answer on the dialogue pop-up less).
\item
  In the presence of a social agent, participants will complete fewer
  correct questions.
\end{itemize}

Protocol \& Data Collection

As outlined previously, while our plan was to run four conditions
(alone, human presence, NAO presence, Pepper presence), we first ran the
two baseline conditions: alone and human observer. 15 participants were
recruited in the alone condition, 16 participants in the human
condition.

The experimental setup was similar to
Figure~\ref{fig:setup} with two differences: when
present, the human observer was sitting at the table, facing the
participant, and the tablets were replaced with laptops with a keyboard
to facilitate the input of the answers. For each participant, we
recorded how many additions were attempted, the total gain (i.e., the
number of correct answers), and the time to calculate each of the
additions.

Results

Based on the data (31 participants for a total of 633 additions), the
average time to dismiss the debug dialogue was 1185ms and the average
time to provide an answer was 9980ms. Based on these values, we
conservatively consider cheating as taking more than 0.8 seconds to
dismiss the spurious debug dialogue and taking less than 5 seconds to
calculate the sum and providing a correct answer. It results in 147
cheating rounds (23.2\% of all rounds).

Looking at these results per condition, we find 77 rounds involving
cheating from 316 rounds in the human condition (24.4\%) and 70 rounds
involving cheating from 317 rounds in the alone condition (22.1\%). TBD:
T-Test. This result shows that 1) participants do cheat relatively
often, 2) however the presence of a human observer does not
significantly impact the cheating behaviour of the participants,
providing no support for H1.

In term of performance, participants in the human presence condition
gave 28 wrong answers out of 239 rounds with no cheating (11.7\% were
wrong answers), while participants in the alone condition gave 25 wrong
answers out of 247 (10.1\%). TBD: again, T-test. Again, there is no
significant performance difference between the two conditions, providing
no support for H2. Therefore, neither of our hypotheses are supported.
Due to the absence of any effects between the human and alone
conditions, we did not pursue the study with robots.

\section*{Discussion}\label{discussion}

Several factors might explain our failing at observing any effect of
social facilitation or social inhibition: the small effect sizes of
social facilitation, certain weaknesses in the psychology literature and
a broader publishing bias toward positive result that might mask how
brittle some effects are.

The difficulty of observing social facilitation

What does this failed attempt at reproducing a `classic' result of
social psychology tell us? Beyond possible experimental confounds, our
failure at reproducing these results is likely due to the small effect
size of social facilitation. In their meta-analysis of studies on social
facilitation, Bond and Titus~\cite{Bond1983} showed
that the overall mean effect sizes are low, ranging from 0.03 to 0.36.
According to Cohen~\cite{Cohen1977}, effect size of 0.2
should be regarded as small, an effect size of 0.5 as medium, and 0.8 as
large.

complete/develop

Social facilitation or inhibition is as well affected by a combination
of several other psychological effects: the observer effect (also known
as the Hawthorne effect); demand characteristics; cultural conventions;
personality orientation; {[}\ldots{}{]}. These effects are potential
confounds, and adequately accounting for each of them in the
experimental design proved problematic.

For example, Liad Uziel~\cite{Uziel2007} compared the
effect sizes of studies in the literature that examined personality
effects for subjects with negative orientation (trait anxiety and low
self-esteem) and positive orientation (extraversion and high
self-esteem). The results suggest that the effect of orientation is
higher than the task complexity in determining the performance, that is,
the subjects with positive orientation had an improve in performance for
both simple and complex tasks in the mere presence of a person, and
similarly, those under negative orientation had an impairment in the
performance in both types of tasks. However, these results were only
based on a few studies that examined the personality effects, hence, the
author states that the results cannot be generalised.

how was it done in other studies? was it accounted for at all? ¡- I
couldn't find anything else but the above point

We believe that these comments on, on one hand the small effect sizes,
and on the other hand the complexity of multi-faceted psychological
effects, are applicable beyond the specific case of social facilitation.

complete/develop

Weak methods in older psychology literature

Beyond the caution that must be observed when studying one specific
psychological effect, a broader range of methodological issues with
older research in psychology might explain why some results in
psychology are incorrectly believed to be reliable.

For instance, Bond and Titus~\cite{Bond1983}
meta-analysis of research on social facilitation claims to have
exhaustively examined every publications prior to the publication of the
meta-analysis itself (in 1983). As a matter of fact, the oldest study
that they refer dates from 1898, and 35 out of the 241 were published
prior to 1965. As such, social facilitation is a good example of an old,
classical psychological effect. It however also hints to the fact that
its characterisation might have relied on weak research methodologies by
today standards. Bond and Titus raise in that regard interesting points:
only 100 out of the 241 studies state that the experimenter was in a
different room in the \textbackslash{}alonecondition(and in 96 studies,
we know the experimenter was in the room). This would be seen today as a
serious confound. Similarly, Bond and Titus report that 72.3\% of the
total participants were undergrad students, pointing to a possible
serious demographic bias.

Biases in scientific publishing: the `file drawer' problem

Coined in 1979 by Robert
Rosenthal~\cite{Rosenthal1979}, the file drawer problem
refers to the bias introduced into the scientific literature by mainly
publishing positive results, and rarely negative or non-confirmatory
results. As a consequence, an effect could be reported and believed
reliable, simply for the lack of literature showing the contrary.
Rosenthal proposes to account for this problem by reporting in
meta-analysis the `fail-safe N' measure: N is the number of null effects
that would be required to make the original result non-significant.
Rosenthal proposes to consider an effect resistant to the `file drawer
problem' of unreported null effects iff the fail-safe N is above
\textbackslash{}(5k+10\textbackslash{}), with
\textbackslash{}(k\textbackslash{}) the number of reported effects.

Bond and Titus report the fail-safe N for some of the effects of social
facilitation. For instance, their meta-analysis show that the
performance quantity of participants for complex tasks reliably
decreases in presence of an observer (even thought the effect size is
small). 54 effects are reported, and they note that the fail-safe N
value is 160: 160 is clearly smaller than
\textbackslash{}(5\textbackslash{}times 54+10=280\textbackslash{}) and
as such, this result could well be subject to the problem of unreported
null effect. The fact that social presence inhibit the performance in
complex tasks is not a robust result in the face of the bias towards
publishing only positive results.

The fail-safe N for the `quantity' for simple tasks is 6,183 and the
`quality' for complex tasks is 5,697. That is why, I wrote in the
literature review that quantity increases with simple tasks and quality
decreases with complex tasks. So I don't know how we should add this,
otherwise it might be seen as we didn't put into account the fail-safe N
in the first place

A weighted calculation of the fail-safe number has been
proposed~\cite{rosenberg2005file} that address some of
the concerns with Rosenthal proposal, and while not systematically
reported in the literature, this metric is a valuable tool for HRI
researchers when assessing how robust a result in psychology is.

\section*{Conclusion}\label{conclusion}

While we have build this paper around the example of social
facilitation, the points raised above in the discussion are broadly
applicable to the literature in social sciences. The limited
reproducibility of results in psychology evidenced
in~\cite{rpp} supports this statement, even though
variations might be observed regarding why each of the reported effects
could not be replicated.

We are however not suggesting that HRI should not build on psychology
anymore. Quite the contrary. The authors of this paper have strong ties
with psychology, and do routinely ground their work in theoretical
frameworks provided by psychology. If anything else, we do encourage the
community to keep on building new links with all the neighbouring
academic fields, and psychology is quite possibly the first of those.

However we need to be remain lucid: results in psychology are not a gold
standard. We must exert a critical questioning.

\begin{itemize}
\tightlist
\item
  working hand in hand with psychologists would be desirable
\item
  submit our protocols to psychologists for review?
\item
  co-design protocols?
\item
  submit our results for discussion with psychologists?
\end{itemize}

\section*{ACKNOWLEDGEMENTS}\label{acknowledgements}

Acknowledgments redacted for double blind review. Vertical space added
below to simulate actual length of section.

\selectlanguage{english}
\FloatBarrier
\end{document}

