PREreview beta testing phase – Here is how you can help!
Hello everyone! In this post we will detail our beta testing phase, hopefully clarifying what it means to be a beta tester for PREreview and guiding you on how to best contribute. Thank you for taking the time to help us in our mission to make journal clubs worth any researchers' time!
Preprint Journal Clubs: Your Opinions Revealed
In the summer of 2017, we conducted a survey to assess scientists' opinions on the value and potential barriers related to reading and reviewing preprints at journal clubs. In this short article we present and discuss the results of the survey as well as how these results helped us shape our approach at PREreview.
Mozilla Mini Grant Application (June 3, 2017)
Here is how it all started. Two researchers and ASAPbio Ambassadors met at a Mozilla Working Open Workshop
in April 2017. A PhD student (Daniela) and a postdoctoral fellow (Sam) decided to volunteer some of their time to develop guidelines to help researchers from all around the world start preprint journal clubs. We believed this would have contributed positively to spreading the word and value of preprints in the scientific community, as well as helped early-career researchers master their skills in peer review.
During the Mozilla Science Global Sprint, June 2-3 2017, we wrote our application to the first Mozilla Science Mini-Grant. We asked for enough money to support 20 beta testers by covering the cost of snacks and beverage for two preprint journal clubs. And we were awarded
Since then, a lot has happened, including starting PREreview thank to the help of the Authorea team and the support of many others who share our mission.
Since July, our application has been posted on our project GitHub
, but we wanted to have it on PREreview as well. So below is our full proposal. Thank you!
PREreview of bioRxiv article “Subfamily-specific functionalization of diversified immune receptors in wild barley”
This is a review of Maekawa et al. bioRxiv 293050; doi: https://doi.org/10.1101/352278 posted on June 20, 2018. In this paper, the authors mined the transcriptomes of 50 different accession of wild barley, generating a rich library of natural variants of the MLA immune receptor—a classical nucleotide-binding domain and leucine-rich repeat-containing (NLR) protein. They grouped the MLA variants in two subfamilies with all receptors known to be effective against the powdery mildew fungus grouping in one subfamily.
ITQB Preprint Journal Club: 21 September 2018
This is a review of the preprint "A tunable dual-input system for 'on-demand' dynamic gene expression regulation" by Elisa Pedone and colleagues. The ITQB Preprint Journal Club met to discuss the article on September 21, 2018. Dr. Federico Herrera led the discussion and wrote this review. Dr. Zach Hensel edited the review for publication.
UIUC Plant Physiology Journal Club: 2018-09-05
Iulia Floristeanu, Cindy Chan , Steven Burgess (0000-0003-2353-7794), Charles Pignon, Stephanie Cullum, Isla Causon, Pietro Hughes
In the preprint "StomataCounter: a deep learning method applied to automatic stomatal identification and counting” (doi: https://doi.org/10.1101/327494
) Fetter et al. introduced a reliable and automated stomata counting program that is more efficient and accurate than human counting and existing algorithms, with a low false positive rate. The authors report the algorithm can be used for previous uncharacterised species and has a 94.2% transfer accuracy when used on untrained datasets. In addition they provide a publically available webtool which could be very beneficial for researchers working on stomata.
We really enjoyed the paper and found it to be of high interest as the method presented could make the work of a lot of people easier. We were particularly impressed with the precision score of StomataCounter which compares well with other existing algorithms, especially in terms of the transfer accuracy. To our knowledge this is a novel approach, as there are no automated stomata counting technologies available, and it outperforms existing methods. The DCCN appears to overcomes challenges of stomata counting and it correctly identified stomata showing minimal false positives on non-plant and non-stomata covered tissue.
The article was well written and easy to follow. We liked the choice of a Deep Convolutional Neural Network (DCNN) for machine learning and felt the authors could have highlighted the benefits of this method by providing a more detailed justification of why it is superior to other algorithms - as this is what made the paper so interesting to us. The text might be further improved by being more specific about results in the abstract and a clearer statement on supplementary data and sampling used.
We were interested to know how the algorithm performs on grass species, as they have “dumbbell-shaped” guard cells and companion cells. It was unclear to us whether grass stomata have been used among the training images so we wondered if the accuracy would be the same for this particular shape of stomata. Including some text in the discussion about this would be illuminating. In addition for the sake of reproducibility and to aid readers comprehension it would be useful to provide (1) a complete list of samples analysed and (2) ideally the whole training dataset in a public repository such as Zendo or Dryad, as it could greatly benefit future comparative studies
Questions we had it would help to clarify
The authors state “many researchers are likely to manually count stomata” - is this because other methods are not good enough, user interface is too complicated or just people liking more traditional things?
In the methods it is written “final fully connected layers by convolutions”. Does this mean there's no fully-connected layers, only convolutional layers? It sounds like both are included in other parts of the paper.
We were confused by the statement “we argue that the current architecture does not transfer well to different scales and images should be pre-processed to match the training scale of the network,” is that not the reason a fully-convolutional network is used? to avoid the inability to manage different input sizes.
- Consider making legends of Figure 3 and Figure 4 bigger to improve readability
Figure 5 is very informative and summarizes the data well
Figure 6 might be improved by including an inset to focus on the down number.
Figure 7 - scale bars
There was a strong linear correlation at higher magnification between DCNN and human counts is this a standard magnification for this type of experiment?
It would be good to include:
Quantification of the accuracy of the automatic separation between abaxial and adaxial datasets. A failure of separation could potentially have contributed to finding stomata on the "adaxial" cuticle.
Discussion of whether the magnifications (200x and 400x) are suitable for this kind of analysis.
Information about why the 4 sources of images were chosen and discussion of whether they provide enough of a range.
An explanation of why data was which showed <98% was discarded
Stats on Gingko and poplar datasets.
Might consider rephrasing “the mean number of stomata detected in the adaxial, aorta, and breast cancer image sets was 1.5, 1.4, and 2.4, respectively, while the mean value of the abaxial set was 24.1” to better explain the low frequency: to compare with non-stomata dataset vs dataset that has very low number of stomata
Code availability: It would be helpful if the code and custom scripts (such as separation of abaxial and adaxial leaf sides) are made available and linked to a repository.
UIUC Plant Physiology Journal Club: 2018-08-13
Steven Burgess (0000-0003-2353-7794), Samuel Fernandes, Antony Digrado, Charles Pignon, Elsa de Becker, Naomi Housego Day, Lusya Manukyan, Stephanie Cullum, Isla Causon, Iulia Floristeanu, Young Cho, Freya Way, Judy Savitskya, Robert Collison, Aoife Sweeney, Pietro Hughes, Cindy Chan AbstractThe paper “Arabidopsis species employ distinct strategies to cope with drought stress” by Bouzid et al. (https://doi.org/10.1101/341859) investigates whether responses to water limitation vary between closely related species by assessing the growth and survival of A. thaliana, A. lyrata and A. halleri accessions in a dry down experiment. By including multiple accessions of each species the authors were able to analyse variation in response to drought stress within and between species based on eight phenotypic parameters. The authors went on to perform comparative transcriptomic analysis between A. lyrata and A. halleri over a time course of drought treatment and identified differentially expressed genes. GO ontology analysis suggest the species analysed adopt different strategies to cope with drought stress, with A. lyrata employing avoidance and tolerance mechanisms, whereas A. thaliana showed strong avoidance but no tolerance. We were impressed with the amount of work performed and thought the study aims to address an interesting question. During the hour long journal club participants were asked to focus on three aspects of the paper as part of a training exercise, including novelty, interest, soundness as well as writing and presentation.ReviewThere are several published papers looking at the effect of drought stress in Arabidopsis species including A. lyrata (Sletvold and Agren 2011; Paccard et al. 2014) and A. thaliana (Ferguson et al. 2018; Kalladan et al. 2017). We suggest toning down the assertion on Line 28 that little is known about the physiological response to drought in closely related species. Although not in a single paper, this issue has been addressed within a species by Sletvold and Agren (2012) and Davila Olivas et al. (2017) and there is a fairly large collection of papers looking at this within a species. To our knowledge, the novelty of this work lies in analysis of drought responses within and between several species of Arabidopsis in one article. We thought this was an interesting approach and that the authors can make more of this point, highlighting the new information that it yields. The article is well written in clear sentences and it was easy to read. We felt the authors had collected a lot of data and believe this could be explored further in the discussion, particularly the differential expression data. There is a lot of microarray and transcriptomic data available for the response of A. thaliana to drought conditions and would like to have seen some form of comparison between these data and that collected for A. lyrata and A. halleri.In addition, it would help to provide more information about why analysis of Arabidopsis accessions was limited to late flowering varieties. Does excluding accessions which can terminate life cycle early bias the experiment? Termination is a major strategy for survival and may impacts upon the conclusion that “response to depletion in SWC did not reveal significant differences between accessions (line 591).” Further discussion of this conclusion in the context of previous studies would be illuminating as it appears to contrast with some findings, for example Bouchabke et al. (2008) which suggested there are differences in response to depletion of SWC between A. thaliana accessions. Readers would also benefit from discussion about how the results from the phenotypic analysis relates to other studies which have implicated trichome production, rosette leaf size and flowering time as drivers for drought tolerance.We commend the fact the methods are detailed which should aid anyone wanting to replicate the study. Several aspects were highlighted as excellent practices, in particular: the fact that all program versions are provided, software parameters are included and accessions and materials are well catalogued. To build on this we recommend depositing the data (particularly transcriptomic analysis) in a public repository such as GEO, SRA or Zendo as required by some journals. This means data can be built upon in future studies and could increase the likelihood of the paper being cited. Inclusion of extended methods in an accompanying Bioprotocol paper or on sites such as Protocols.io and sharing custom scripts in a github repository (or on Zendo) would make the reporting of methods outstanding. Minor commentsThe paper may benefit from clarify a number of questions raised by participants: How was soil mixture chosen?Line 192 under what conditions were the plants grown in the greenhouse? Line 206: How were the first signs of wilting defined? It might be worth mentioning this earlier in the manuscript (line 206)Line 232 - How was loss of turgidity measured?The colors used in the figures will create difficulty for those individuals with color blindness, using color oracle (https://colororacle.org/) can help address this issue.Figure 2 - no scale bars are includedFigure 2 - how many plants per pot were analyzed? This could have impacted on measurements displayed images suggested this was variable.Figure 5 - we thought it was excellent that unit level data is provided, this could be extended to the other figures too.Figure legends could be improved e.g. by changing “halleri” to “A. halleri”*Formating of legend (line 1326 )extra space and period.The article might benefit from consistent formatting/presentation of figuresA table might be more appropriate for Figures 7 and 8Inclusion of n= numbers in the figures would help readers assess the data.We were confused about how the experiment was designed, why is data for only on biological replicate displayed in figures 7 and 8.References Bouchabke et al. 2008 https://doi.org/10.1371/journal.pone.0001705 Davila Olivas et al. (2017) https://doi.org/10.1111/mec.14100Des Marais, et al. (2012) https://doi.org/10.1105/tpc.112.096180 Huttunen et al. 2010 https://doi.org/10.5735/085.047.0304Kalladan et al. (2017) https://doi.org/10.1073/pnas.1705884114 Paccard et al. (2014) doi: 10.1007/s00442-014-2932-8 Sletvold and Agren (2011) https://doi.org/10.1007/s10682-011-9502-x