Christian Guckelsberger Minor rephrasing and some more work on the recommender case study.  about 9 years ago

Commit id: 2a1531b64f22ee465b84530343d1c624d13694d1

deletions | additions      

    Binary files a/.DS_Store and b/.DS_Store differ        

analyses current evaluation procedures used in computational  creativity, and provides a much-needed set of customisable evaluation  guidelines, the \emph{Standardised Procedure for Evaluating Creative  Systems} (SPECS) \cite{jordanous:12}. The Originally designed to evaluate the concept of creativity, the  three stepprocess of  SPECS process firstly  requires the evaluator to define the concept(s) they are evaluating the system on (originally SPECS was designed to evaluate the concept of creativity). on.  This definition is then converted intotestable  standards that can eventually  be used to test and  evaluate individual systems, or comparatively evaluate multiple systems. We give a slightly modified version of her earlier evaluation  guidelines, in that rather than attempt a definition and evaluation of  {\em creativity}, we follow the three steps for \emph{serendipity}. %\newpage         

\begin{quote} {\em Using Step 1, clearly state what standards you use to evaluate the serendipity of your  system. }\end{quote}  \noindent Here we need to identify testable standards from With  our definitionof computational serendipity.  in other words, mind,  we now state propose the following standards for evaluating serendipity in computational systems. They represent  the key parts of our definition in a form that can be evaluated as allows  to what assess the  degree to which  they are or are not met.   With our definition in mind, we propose the following standards for evaluating serendipity in computational systems: met:  %% Serendipity relies on a reassessment or reevaluation -- a \emph{focus shift} in which something that was previously uninteresting, of neutral, or even negative value, becomes interesting. 

\begin{quote} {\em Test your serendipitous system against the standards stated in Step 2 and report the  results.}\end{quote}  \noindent We will develop several examples of devote  the application of this framework entire next section \ref{sec:computational-serendipity}  to examples from computing testing our framework  in Section  \ref{sec:computational-serendipity}. respect to existing computational systems focussing on serendipity.  \paragraph{Example.}  To get a feel for the likelihood score, briefly consider the case of         

and his dog's fur and became curious about how they worked. After  examining them under a microscope, he realised the possibility of  creating a new kind of fastener that worked in a similar  fashion. fashion, laying the foundations for the hook-and-pile fastener.  % \cite[p. x]{roberts}  \end{itemize} 

abductive inference (what might cause a clear patch in a petri  dish?); analogical reasoning (de Mestral constructed a target domain  from the source domain of burs hooked onto fabric); and conceptual  blending (Kekul\'e (Kekul\'e, discoverer of the benzene ring structure,  blended his knowledge of molecule structure with his vision of a snake biting its tail). The bridge may also rely on  new social arrangements, such as the formation of cross-cultural  research networks. 

hypothesis, a new use for a material substance, and so on. The  outcome may contribute evidence in support of a known hypothesis, or  a solution to a known problem. Alternatively, the result may itself  be a {\em new} hypothesis or problem. The result may bea  ``pseudoserendipitous'' in the sense that it was {\em sought}, while  nevertheless arising from an unknown, unlikely, coincidental or  unexpected source. More classically, it is an \emph{unsought}         

revised criteria, and also show forays of computational intelligence  into domains known for serendipity in their everyday cultural context.  We then turn to a more elaborated thought experiment that evaluates  these ideas in the course of developing a new system design.First, a  bracketing remark.  \paragraph{Minimally As a contrast, we introduce the notion of \emph{minimally  serendipitous systems.} systems}:  According to our standards, there are various ways to achieve a result with \emph{low} serendipity: if the observation was likely, if further developments happened with little skill, and if the the value of the  result was low, then we would not say the outcome was serendipitous.  We would be prepared to attribute ``minimal serendipity'' to cases 

\paragraph{Recommender systems.}   % Stress distinction between serendipity on the system- vs. serendipity on the user's side.  As discussed in Section \ref{sec:related}, recommender systems are one  of the primary contexts in computing where serendipity is seen to play  a role. As we noted, these systems mostly focus on discovery.  Although this describes the mainstream of recommender system  development, it seems that there are some architectures that also take  account of invention. We have in mind invention such as the  Bayesian methods (surveyed surveyed  in Chapter 3 of \citeNP{shengbo-guo-thesis}). \citeNP{shengbo-guo-thesis}.  The current discussion focuses on possibilities for serendipity on the system side, drawing  on the observation that recommender systems do  not only  \emph{stimulate} serendipitous discovery: they discovery, but  also have the task of \emph{simulating} \emph{simulate}  when this is likely to occur.  A recommendation is typically provided if the system suspects Most discussions of serendipity in recommender systems focus on suggesting items to a user  thatthe  item  will be likely to introduce new  ideas that are unexpected, but  close to what the userknows, but that will be unexpected. Typical discussions of  serendipity in recommender systems focus on this. However, user  behaviour (e.g.~following up on these recommendations) may also serve  as a \textbf{serendipity trigger} for the system, and change the way  it makes recommendations in the future. Note that it  is typically the  system's \emph{developers} who adapt the system; even in the Bayesian  case, already interested in. If  the latter connection exists, such a  system has limited responsibilities. Nevertheless, the  impetus to develop increasingly autonomous systems is present,  especially in complex domains where hand-tuning reaches its limits. must be called pseudoserendipitous.  Current systems have at least the makings of a \textbf{prepared mind}, including both a \emph{user model} and a \emph{domain model}, both of which can be updated dynamically. A \textbf{bridge} to a new kind of recommendation may be found by pattern matching, and especially by looking for exceptional cases: new elements are introduced into the domain which do not cluster well, or different clusters appear in the user model that do not have obvious connections between them. The intended outcome of recommendations depends on the organisational mission: to make money, to provide a good user experience, etc. mission, and can in most cases be situated between making money and empowering the user.  The serendiptious \textbf{result} would be learning a new approach that helps to address these goals. \textbf{Chance} will only have a significant role in the system if it  has the capacity to learn from user behaviour.  %% The typical commercial perspective on recommendations is related to  %% the process of ``conversion'' -- turning recommendations into  %% clicks and clicks into purchases.  Note that it is typically the system's \emph{developers} who adapt the system; even in the Bayesian case, the system has limited autonomy. Nevertheless, the impetus to develop increasingly autonomous systems is present, especially in complex domains where hand-tuning is either very cost-intensive or infeasible. As an approach towards more autonomy, user behaviour (e.g.~following up on these recommendations) may serve as a \textbf{serendipity trigger} for the system, and change the way it makes recommendations in the future.  Combined with the ability to learn, \textbf{curiosity} could be described as the urge to make ``outside-the-box''\footnote{\citeA{abbassi2009getting}.} recommendations specifically for the purposes of learning more about users, possibly to the detriment of other metrics over the short term. Measures of \textbf{sagacity} would relate to the system's ability to  draw inferences from user behaviour. For example, the system might do A/B testing to decide how novel recommendation strategies influence conversion. Again, currently this would typically be organised by the system's developers. The \textbf{value} of recommendation strategies  can be measured in terms of traditional business metrics or other organisational objectives.  %%%  \begin{table}[ht!]  {\centering \renewcommand{\arraystretch}{1.5} 

\end{table}%  \normalsize  %%%  recommendations specifically for the purposes of learning more about  users, possibly to the detriment of other metrics over the short term.  Measures of \textbf{sagacity} would relate to the system's ability to  draw inferences from user behaviour. For example, the system might do  A/B testing to decide how novel recommendation strategies influence  conversion. Again, currently this would typically be organised by the  system's developers. The \textbf{value} of recommendation strategies  can be measured in terms of traditional business metrics or other  organisational objectives.  A \textbf{dynamic world} which nevertheless exhibits some regularity  is a precondition for useful A/B testing. As mentioned above the         

The connection to the key condition and components of serendipity % AJ four dimensions instead of components? ****  introduced in our literature survey are as follows:  %  The \textbf{serendipity trigger} is denoted by $T$.  %  The \textbf{focus shift} takes place with the identification of         

\end{enumerate}  \end{quote}  \noindent This definition can be summarised schematically as follows: follows, with letters referencing to the key condition and components introduced in the literature survey:  % \input{schematic-tikz}  {\centering         

\subsection{Related work} \label{sec:related}  An active research community investigating computational models of serendipity exists in the field of information retrieval, and specifically, in recommender systems \cite{Toms2000}. In this domain, \citeA{Herlocker2004} and \citeA{McNee2006} view serendipity as an important factor for user satisfaction, alongside accuracy and diversity. Serendipity in recommendations variously always  require the system to deliver an suggest  \emph{unexpected} and items, which have to be considered as either  \emph{useful}, \emph{interesting}, \emph{attractive} or \emph{relevant} item. by the user.  % \cite{Herlocker2004} \cite{Lu2012},\cite{Ge2010}.   Definitions differ as to the requirement of \emph{novelty}; \citeA{Adamopoulos2011}, for example, describe systems that suggest items that may already be known, but are still unexpected in the current context. While standardized measures such as the $F_1$-score or the (R)MSE are used to determine the \emph{accuracy} of a recommendation (as very close to what the user is known to prefer), there is no common agreement on a measure for serendipity yet, although there are several proposals \cite{Murakami2008, Adamopoulos2011, McCay-Peet2011,iaquinta2010can}.  In terms of our model, these systems focus mainly on producing a \emph{serendipity trigger} and predicting their potential  for serendipitous discovery on  the user and in support side  of discovery, the user,  but they include aspects of user modeling which could bring other elements into play, as we will discuss in Section \ref{sec:computational-serendipity}. Recent work has examined related topics of \emph{curiosity}  \cite{wu2013curiosity} and \emph{surprise} \cite{grace2014using} in