Anna Jordanous modifying the text on SPECS to cover different layers of evaluation (successfully? any comments?)  about 9 years ago

Commit id: 9945ad62247af2d4f584e45574987be079fe72fb

deletions | additions      

       

analyses current evaluation procedures used in computational  creativity, and provides a much-needed set of customisable evaluation  guidelines, the \emph{Standardised Procedure for Evaluating Creative  Systems} (SPECS) \cite{jordanous:12}. The three step process of SPECS requires the evaluator to define the concept(s) they are evaluating the system on (originally SPECS was designed to evaluate the concept of creativity). This definition is then converted into testable standards that can be used to evaluate individual systems, or comparatively evaluate multiple systems.  We propose give  a slightly modified version of her earlier evaluation guidelines, in that rather than attempt a definition and evaluation of  {\em creativity}, we follow the three steps for \emph{serendipity}. 

\begin{quote} {\em Identify a definition of serendipity that your  system should satisfy to be considered serendipitous.}\end{quote}  \noindent As above. Our computational definition of serendipity is as given in Section \ref{sec:our-model}.  %% This situation can be pictured schematically as follows:         

\begin{quote} {\em Using Step 1, clearly state what standards you use to evaluate the serendipity of your  system. }\end{quote}  \noindent Here we need to identify testable standards from our definition of computational serendipity. in other words, we now state the key parts of our definition in a form that can be evaluated as to what degree they are or are not met.  With our definition in mind, we propose the following standards for evaluating serendipity in computational systems: %% Serendipity relies on a reassessment or reevaluation -- a \emph{focus shift} in which something that was previously uninteresting, of neutral, or even negative value, becomes interesting. 

evaluated as useful, by the system and/or by an external source.}  \item[\emph{(\textbf{B - Dimensions})}] \emph{Serendipity, and its  various dimensions, can be present to a greater or lesser degree.  If the criteria above have been met, we consider the system (and optionally,  generate ratings as estimated probabilities probabilities)  along several dimensions: %  \emph{($\mathbf{a}$ - \textbf{chance})} how likely was this trigger to appear to  the system? 

\medskip  \emph{Then aggregating  $\mathbf{a}\times\mathbf{b}\times\mathbf{c}$ gives a likelihood score: low likelihood $\mathbf{a}\times\mathbf{b}\times\mathbf{c}$  and high value is $\mathbf{d}$ are  the criterion criteria  we use to say that the event was ``highly serendipitous.''} \item[\emph{(\textbf{C - Factors})}] \emph{Finally, if the criteria  from Part A are met, and if the event is deemed ``highly