Xavier Holt edited Timeline_Evaluation_Methodology_Gold_Timeline__.md  almost 8 years ago

Commit id: 42e461e198ceea87f29105de4ebb36e12d6cdade

deletions | additions      

       

## #  Timeline Evaluation Methodology ### ##  Gold Timeline Generation This approach generates a timeline that is in some sense 'correct' or 'best'. Different system generated timelines are then compared against the standard. The closer or more similar by some measure to the standard, the better the generated timeline is.  ### ##  Entity Selection Process We propose to select twenty figures who are central to the US presidential election. Current forerunners Bernie Sanders, Donald Trump, Hillary Clinton, John Kasich and Ted Cruz will be included. Additionally we will select a number of figures with less news coverage to allow evaluation on models with more sparse input. Gold-standard generation will involve presenting crowd workers with a list of news URLS. They will then be asked to indicate whether or not to include an article in a gold-standard timeline.  ### ##  Metric Based Comparisons Once a gold standard has been found, there are several approaches used to evaluate a candidate timeline.   ### ##  Crowd-Vote asda