Members: Lorena Barba, Ann Gabriel, Richard Price, Nancy Davenport, Mark Newton, Lacey Earle, Abel PackerIn thinking through the future of peer review, we considered four properties that would be desirable in a peer review system:Moving from a 2-person-system to a many person systemCurrently academic papers are peer reviewed by ~2 people: a journal editor will send out a submission to two peer reviewers to solicit their thoughts.It would be good if there was a peer review system, both pre-publication and post-publication, that encouraged readers to share their thoughts and evaluations of the paper. This is what we mean by a ‘many person system’. This system is normally called “post-publication peer review”, though it’s worth noting that getting feedback from readers will work in fields where preprints and drafts are shared.Peer Review of code and data-setsHistorically the only form of scholarly output that gets peer reviewed is the paper. Since peer review, and venue of publication, is one of the primary means for academic promotion, there is not an incentive to share data-sets and code. It would be good to have a system that did peer review these items, as that would incentivize academics to share them.Closed vs open; anonymous vs signed We discussed the question whether peer reviews should be kept private, which is the norm, or whether it would be good for them to be open. We also discussed the orthogonal distinction between the peer reviews being anonymous, which is the historical norm, or whether they should be signed (non-anonymous).Discoverability of all peer reviews on a paper throughout life-cycleWe discussed the fact that if you are looking at a published paper, it would be nice to know if there are comments and peer reviews on a prior version of the paper, say a public pre-print.Case StudiesWe discussed some case studies of developments in peer review.Journal of Open Source SoftwareThe Journal of Open Source Software was founded by one of the members of our group, Lorena Barba. The way it works is that authors submit some code, and a one-page write up of what the code does. The code is then peer reviewed by people familiar with the relevant programming languages.      Here is what the one-page write-up looks like:And here is what the list of publications looks like:          Academia.edu     We discussed Academia.edu’s Sessions feature. Sessions are a way for authors to crowd-source peer review on their draft papers. Sessions last 20 days, and the feedback on the paper appears on the right-hand margin of the paper.              At the end of the 20 days, the session is closed, and no further comments are possible.      Survey on Open/Closed and Anonymous/Signed     Ann Gabriel was in our group, and she shared some survey data that Elsevier had gathered on experiments around open and closed peer review.      Elsevier tried open peer review for five journals: what this meant is that peer reviewers are told in advance that their peer reviews will be published openly, alongside the paper. Furthermore, peer reviewers will have the choice whether to sign their public peer reviews, or keep them private.      After the experiment, peer reviewers were surveyed for their opinions. There were 40 respondents:     ·      95% said publishing review reports didn’t influence their recommendation.  ·      45% provided consent to reveal their names.  ·      98% said they will accept further review invites for the journal.     Other data included:     ·      10 out of 14 peer reviewers thought that publishing of peer reviews should become common practice.   ·      70% of editors said the reports are more in depth and constructive.   ·      40% of editors said that the peer review reports are more helpful to make their decision.      Further Questions     Some questions that we thought warranted further discussion were:     More modern formats like HTML to allow more seamless commenting     When papers are published in HTML form, in-line comments are possible, which are not possible with formats like PDF.      Formalization of open peer reviews: citable via DOIs     When open peer reviews can be cited, there will be incentive to want one’s peer reviews to be open.      TOP-like framework to think about open/closed spectrum     The TOP framework is a series of standards that correspond to greater degrees of data transparency guidelines that a given journal might have. E.g. level 1 means that a journal will state whether data is available for a given paper; level 2 means that data is posted to a trusted 3rd party data repository; level 3 means that the study has been replicated by an independent 3rd party prior to publication.      There was a question about whether a similar set of steps could be drawn up for levels of open-ness for peer review.And here is what the list of publications looks like:          Academia.edu     We discussed Academia.edu’s Sessions feature. Sessions are a way for authors to crowd-source peer review on their draft papers. Sessions last 20 days, and the feedback on the paper appears on the right-hand margin of the paper.              At the end of the 20 days, the session is closed, and no further comments are possible.      Survey on Open/Closed and Anonymous/Signed     Ann Gabriel was in our group, and she shared some survey data that Elsevier had gathered on experiments around open and closed peer review.      Elsevier tried open peer review for five journals: what this meant is that peer reviewers are told in advance that their peer reviews will be published openly, alongside the paper. Furthermore, peer reviewers will have the choice whether to sign their public peer reviews, or keep them private.      After the experiment, peer reviewers were surveyed for their opinions. There were 40 respondents:     ·      95% said publishing review reports didn’t influence their recommendation.  ·      45% provided consent to reveal their names.  ·      98% said they will accept further review invites for the journal.     Other data included:     ·      10 out of 14 peer reviewers thought that publishing of peer reviews should become common practice.   ·      70% of editors said the reports are more in depth and constructive.   ·      40% of editors said that the peer review reports are more helpful to make their decision.      Further Questions     Some questions that we thought warranted further discussion were:     More modern formats like HTML to allow more seamless commenting     When papers are published in HTML form, in-line comments are possible, which are not possible with formats like PDF.      Formalization of open peer reviews: citable via DOIs     When open peer reviews can be cited, there will be incentive to want one’s peer reviews to be open.      TOP-like framework to think about open/closed spectrum     The TOP framework is a series of standards that correspond to greater degrees of data transparency guidelines that a given journal might have. E.g. level 1 means that a journal will state whether data is available for a given paper; level 2 means that data is posted to a trusted 3rd party data repository; level 3 means that the study has been replicated by an independent 3rd party prior to publication.      There was a question about whether a similar set of steps could be drawn up for levels of open-ness for peer review. And here is what the list of publications looks like:          Academia.edu     We discussed Academia.edu’s Sessions feature. Sessions are a way for authors to crowd-source peer review on their draft papers. Sessions last 20 days, and the feedback on the paper appears on the right-hand margin of the paper.              At the end of the 20 days, the session is closed, and no further comments are possible.      Survey on Open/Closed and Anonymous/Signed     Ann Gabriel was in our group, and she shared some survey data that Elsevier had gathered on experiments around open and closed peer review.      Elsevier tried open peer review for five journals: what this meant is that peer reviewers are told in advance that their peer reviews will be published openly, alongside the paper. Furthermore, peer reviewers will have the choice whether to sign their public peer reviews, or keep them private.      After the experiment, peer reviewers were surveyed for their opinions. There were 40 respondents:     ·      95% said publishing review reports didn’t influence their recommendation.  ·      45% provided consent to reveal their names.  ·      98% said they will accept further review invites for the journal.     Other data included:     ·      10 out of 14 peer reviewers thought that publishing of peer reviews should become common practice.   ·      70% of editors said the reports are more in depth and constructive.   ·      40% of editors said that the peer review reports are more helpful to make their decision.      Further Questions     Some questions that we thought warranted further discussion were:     More modern formats like HTML to allow more seamless commenting     When papers are published in HTML form, in-line comments are possible, which are not possible with formats like PDF.      Formalization of open peer reviews: citable via DOIs     When open peer reviews can be cited, there will be incentive to want one’s peer reviews to be open.      TOP-like framework to think about open/closed spectrum     The TOP framework is a series of standards that correspond to greater degrees of data transparency guidelines that a given journal might have. E.g. level 1 means that a journal will state whether data is available for a given paper; level 2 means that data is posted to a trusted 3rd party data repository; level 3 means that the study has been replicated by an independent 3rd party prior to publication.      There was a question about whether a similar set of steps could be drawn up for levels of open-ness for peer review.We discussed the question whether peer reviews should be kept private, which is the norm, or whether it would be good for them to be open. We also discussed the orthogonal distinction between the peer reviews being anonymous, which is the historical norm, or whether they should be signed (non-anonymous).      Discoverability of all peer reviews on a paper throughout life-cycle     We discussed the fact that if you are looking at a published paper, it would be nice to know if there are comments and peer reviews on a prior version of the paper, say a public pre-print.      Case Studies     We discussed some case studies of developments in peer review.      Journal of Open Source Software     The Journal of Open Source Software was founded by one of the members of our group, Lorena Barba.      The way it works is that authors submit some code, and a one-page write up of what the code does. The code is then peer reviewed by people familiar with the relevant programming languages.      Here is what the one-page write-up looks like:             And here is what the list of publications looks like:          Academia.edu     We discussed Academia.edu’s Sessions feature. Sessions are a way for authors to crowd-source peer review on their draft papers. Sessions last 20 days, and the feedback on the paper appears on the right-hand margin of the paper.              At the end of the 20 days, the session is closed, and no further comments are possible.      Survey on Open/Closed and Anonymous/Signed     Ann Gabriel was in our group, and she shared some survey data that Elsevier had gathered on experiments around open and closed peer review.      Elsevier tried open peer review for five journals: what this meant is that peer reviewers are told in advance that their peer reviews will be published openly, alongside the paper. Furthermore, peer reviewers will have the choice whether to sign their public peer reviews, or keep them private.      After the experiment, peer reviewers were surveyed for their opinions. There were 40 respondents:     ·      95% said publishing review reports didn’t influence their recommendation.  ·      45% provided consent to reveal their names.  ·      98% said they will accept further review invites for the journal.     Other data included:     ·      10 out of 14 peer reviewers thought that publishing of peer reviews should become common practice.   ·      70% of editors said the reports are more in depth and constructive.   ·      40% of editors said that the peer review reports are more helpful to make their decision.      Further Questions     Some questions that we thought warranted further discussion were:     More modern formats like HTML to allow more seamless commenting     When papers are published in HTML form, in-line comments are possible, which are not possible with formats like PDF.      Formalization of open peer reviews: citable via DOIs     When open peer reviews can be cited, there will be incentive to want one’s peer reviews to be open.      TOP-like framework to think about open/closed spectrum     The TOP framework is a series of standards that correspond to greater degrees of data transparency guidelines that a given journal might have. E.g. level 1 means that a journal will state whether data is available for a given paper; level 2 means that data is posted to a trusted 3rd party data repository; level 3 means that the study has been replicated by an independent 3rd party prior to publication.      There was a question about whether a similar set of steps could be drawn up for levels of open-ness for peer review.the only form of scholarly output that gets peer reviewed is the paper. Since peer review, and venue of publication, is one of the primary means for academic promotion, there is not an incentive to share data-sets and code. It would be good to have a system that did peer review these items, as that would incentivize academics to share them.      Closed vs open; anonymous vs signed     We discussed the question whether peer reviews should be kept private, which is the norm, or whether it would be good for them to be open. We also discussed the orthogonal distinction between the peer reviews being anonymous, which is the historical norm, or whether they should be signed (non-anonymous).      Discoverability of all peer reviews on a paper throughout life-cycle     We discussed the fact that if you are looking at a published paper, it would be nice to know if there are comments and peer reviews on a prior version of the paper, say a public pre-print.      Case Studies     We discussed some case studies of developments in peer review.      Journal of Open Source Software     The Journal of Open Source Software was founded by one of the members of our group, Lorena Barba.      The way it works is that authors submit some code, and a one-page write up of what the code does. The code is then peer reviewed by people familiar with the relevant programming languages.      Here is what the one-page write-up looks like:             And here is what the list of publications looks like:          Academia.edu     We discussed Academia.edu’s Sessions feature. Sessions are a way for authors to crowd-source peer review on their draft papers. Sessions last 20 days, and the feedback on the paper appears on the right-hand margin of the paper.              At the end of the 20 days, the session is closed, and no further comments are possible.      Survey on Open/Closed and Anonymous/Signed     Ann Gabriel was in our group, and she shared some survey data that Elsevier had gathered on experiments around open and closed peer review.      Elsevier tried open peer review for five journals: what this meant is that peer reviewers are told in advance that their peer reviews will be published openly, alongside the paper. Furthermore, peer reviewers will have the choice whether to sign their public peer reviews, or keep them private.      After the experiment, peer reviewers were surveyed for their opinions. There were 40 respondents:     ·      95% said publishing review reports didn’t influence their recommendation.  ·      45% provided consent to reveal their names.  ·      98% said they will accept further review invites for the journal.     Other data included:     ·      10 out of 14 peer reviewers thought that publishing of peer reviews should become common practice.   ·      70% of editors said the reports are more in depth and constructive.   ·      40% of editors said that the peer review reports are more helpful to make their decision.      Further Questions     Some questions that we thought warranted further discussion were:     More modern formats like HTML to allow more seamless commenting     When papers are published in HTML form, in-line comments are possible, which are not possible with formats like PDF.      Formalization of open peer reviews: citable via DOIs     When open peer reviews can be cited, there will be incentive to want one’s peer reviews to be open.      TOP-like framework to think about open/closed spectrum     The TOP framework is a series of standards that correspond to greater degrees of data transparency guidelines that a given journal might have. E.g. level 1 means that a journal will state whether data is available for a given paper; level 2 means that data is posted to a trusted 3rd party data repository; level 3 means that the study has been replicated by an independent 3rd party prior to publication.      There was a question about whether a similar set of steps could be drawn up for levels of open-ness for peer review.Members: Lorena Barba, Ann Gabriel, Richard Price, Nancy Davenport, Mark Newton, Lacey Earle, Abel Packer        In thinking through the future of peer review, we considered four properties that would be desirable in a peer review system:     Moving from a 2-person-system to a many person system     Currently academic papers are peer reviewed by ~2 people: a journal editor will send out a submission to two peer reviewers to solicit their thoughts.     It would be good if there was a peer review system, both pre-publication and post-publication, that encouraged readers to share their thoughts and evaluations of the paper. This is what we mean by a ‘many person system’. This system is normally called “post-publication peer review”, though it’s worth noting that getting feedback from readers will work in fields where preprints and drafts are shared.      Peer Review of code and data-sets     Historically the only form of scholarly output that gets peer reviewed is the paper. Since peer review, and venue of publication, is one of the primary means for academic promotion, there is not an incentive to share data-sets and code. It would be good to have a system that did peer review these items, as that would incentivize academics to share them.      Closed vs open; anonymous vs signed     We discussed the question whether peer reviews should be kept private, which is the norm, or whether it would be good for them to be open. We also discussed the orthogonal distinction between the peer reviews being anonymous, which is the historical norm, or whether they should be signed (non-anonymous).      Discoverability of all peer reviews on a paper throughout life-cycle     We discussed the fact that if you are looking at a published paper, it would be nice to know if there are comments and peer reviews on a prior version of the paper, say a public pre-print.      Case Studies     We discussed some case studies of developments in peer review.      Journal of Open Source Software     The Journal of Open Source Software was founded by one of the members of our group, Lorena Barba.      The way it works is that authors submit some code, and a one-page write up of what the code does. The code is then peer reviewed by people familiar with the relevant programming languages.      Here is what the one-page write-up looks like:             And here is what the list of publications looks like:          Academia.edu     We discussed Academia.edu’s Sessions feature. Sessions are a way for authors to crowd-source peer review on their draft papers. Sessions last 20 days, and the feedback on the paper appears on the right-hand margin of the paper.              At the end of the 20 days, the session is closed, and no further comments are possible.      Survey on Open/Closed and Anonymous/Signed     Ann Gabriel was in our group, and she shared some survey data that Elsevier had gathered on experiments around open and closed peer review.      Elsevier tried open peer review for five journals: what this meant is that peer reviewers are told in advance that their peer reviews will be published openly, alongside the paper. Furthermore, peer reviewers will have the choice whether to sign their public peer reviews, or keep them private.      After the experiment, peer reviewers were surveyed for their opinions. There were 40 respondents:     ·      95% said publishing review reports didn’t influence their recommendation.  ·      45% provided consent to reveal their names.  ·      98% said they will accept further review invites for the journal.     Other data included:     ·      10 out of 14 peer reviewers thought that publishing of peer reviews should become common practice.   ·      70% of editors said the reports are more in depth and constructive.   ·      40% of editors said that the peer review reports are more helpful to make their decision.      Further Questions     Some questions that we thought warranted further discussion were:     More modern formats like HTML to allow more seamless commenting     When papers are published in HTML form, in-line comments are possible, which are not possible with formats like PDF.      Formalization of open peer reviews: citable via DOIs     When open peer reviews can be cited, there will be incentive to want one’s peer reviews to be open.      TOP-like framework to think about open/closed spectrum     The TOP framework is a series of standards that correspond to greater degrees of data transparency guidelines that a given journal might have. E.g. level 1 means that a journal will state whether data is available for a given paper; level 2 means that data is posted to a trusted 3rd party data repository; level 3 means that the study has been replicated by an independent 3rd party prior to publication.      There was a question about whether a similar set of steps could be drawn up for levels of open-ness for peer review.