Our evaluation of the factors did not agree with the respondents in every case (Table \ref{435874}). Impact Factor was raised frequently in the responses, as a secondary factor if not the primary one, but in some cases the decrease in Impact Factor or ISI category ranking was relatively small. In two cases the respondents explained the decrease as an expected fluctuation, which seemed unlikely when given the extremity of the decline. One respondent proposed that the decline in submissions was due the journal rejecting more, when in fact the acceptance rate went up. Retractions was not a factor identified by any of the respondents but, given our data, it seemed to be the most likely factor in at least four cases.

Our statistical analysis did not find a correlation between acceptance rate and submissions, but some of our respondents highlighted a connection. One editor-in-chief, who has worked on several journals, said “if you get a huge spike in rejection rate, you can get a drop in submissions.”

Additional desk research looked at social media in the researcher community (e.g. scirev.org; letpub.com; pubpeer.com, Twitter, etc.) regarding the journal and year of each case study. In four cases we identified significant negative comments regarding aspects of peer review at the respective journal, concerning either the process or the outcomes.

We also identified a decrease in companies in an industry connected with journals in three of our cases; we suspect that the decline in that industry may correlate with a decline in research output. A respondent to one of our other cases also identified a decline in research funding in the journal’s field as a factor.

The analysis of Impact Factor looked at number of submissions, rather than percentage increase, so the average increase is rough and not generalizable across all journals. Future analysis could include repeating these analyses stratified by subject area, or other factors, to improve the accuracy of the coefficient. Also, future analysis could look at relative ISI subject ranking, rather than absolute Impact Factor.

For each year of analysis there is a very small sample of journals with retractions (often under 30 journals), which may cause issues with the accuracy of the results of the t-tests. This analysis does not distinguish between a journal publishing 1 retraction in a single year versus a journal publishing 15. Some journals issued retractions every year, or most years, so no difference can be analyzed.

The Impact Factor is released midway through the calendar year. Retractions might be published at any point in the calendar year. Future analysis could look at submissions by month, perhaps focusing on the 12-month period after the release of the IF or publication of the retraction.

Because the sample includes journals that started publishing within the timeframe (or shortly before), efforts were made to exclude the first and second year of submissions data. However, the submissions growth of a new journal may not be linear and may be uneven. 

New journals are unlikely to have an Impact Factor and less likely to have retractions.

There are many factors involved in the number of submissions a journal receives and many of them are beyond the control of the editor or the publisher. The number of research papers being written will be dependent in large part on the amount of research funding available and the number of relevant untapped research areas in that field. (One respondent to our cases noted how the technique covered by the relevant journal is now “commonplace” leading to a decline in new research.) Even where research funding unchanged, submission numbers would still fluctuate, dependent on the incidence of negative or null results (which are commonly not submitted for publication.) Therefore, submission numbers are unlikely to rise evenly. Because there are multiple submission options for authors, including both general and specialized journals, there is competition between journals for submissions.

From our data, we identified two factors that have a significant impact on submission numbers. The finding that changes in Impact Factor correlate to changes in submissions is not surprising, as it fits with prior research about authors’ selection process. Because Impact Factor is used by various bodies, including universities and funders, to assess published research, authors will inevitably seek to be published in journals with the highest Impact Factor. One of our respondents noted, “it has been unfortunate that some countries and grant agencies set arbitrary limits of what IF journals will or will not count in certain metrics.”

Our second finding is less expected – that retractions correlate with lower submissions. Prior research indicated that journal reputation is an important factor for authors when choosing where to submit. Our case study research found that negative research comments about a journal’s peer review occurred in years with significant declines in submissions. Though an isolated case, the decline of submissions following editor-in-chief misconduct seems indicative of a correlation between journal reputation and submissions. To distinguish these reputational effects from Impact Factor, we shall categorize them “peer review reputation” (PRR). We propose that PRR is an important selection criterion for authors and that significant negative PRR will lead to a decrease in submission numbers. We believe this is the reason for the correlation between retractions and submission numbers. (Previous researchers have used retractions as a metric for peer review quality; see \citealt{Horbach_2018}).

We do not believe that the correlation between retractions and submission numbers should be considered an incentive for editors or publishers to not publish retractions. Firstly, publishers have a responsibility to maintain the scientific and scholarly literature, which includes the publication of retractions, corrections, and errata. Secondly, whilst retractions correlate with submission numbers, there are many incidences in our data where there was no effect, so it is too simplistic to conclude that retractions cause declines in submissions. Retractions are a quantifiable way of measuring PRR, but certainly not the only aspect of it. only aspect of it. Thirdly, as our case studies indicate, negative PPR is transmitted by many means including word-of-mouth (which would be impossible to quantify) and not just via retractions. Lastly, we suspect that not publishing a retraction, where one was justified, would ultimately lead to a greater decline in PPR than publishing the retraction.

Whilst in many surveys journal reptuation was ranked higher than Impact Factor, it is not possible to use our data to contrast IF and PRR (as measured by retractions) to determine which is the more significant “risk factor” for a decrease in submissions. Our analysis of these two factors involved different tests and different types of analysis, which are not easily comparable. Also, the number of retractions, whilst statistically significant, was relatively small. Changes in Impact Factor correlate to both increases and decreases in submission numbers, whilst retractions only correlate to decreases in submissions. Retractions are also a binary measure; either a retraction was published, or it wasn’t. The cases where multiple retractions were published by the same journal in the same year were too few to test for any correlation between number of retractions and number of submissions – we do not expect such a relationship. Furthermore, it may be that trying to rank IF against PRR would be meaningless in any case.

It is worth noting the previous finding that high IF journals published more retractions \citep*{Fang_2011}. Those authors speculate that the correlation may be due to the high IF increasing the incentive for authors to manipulate their results in order to get published, or may be due to higher IF journals coming under more intense scrutiny. Others have proposed that high IF journals favour papers reporting novel and unexpected results, which are disproportionately likely to be retracted later \citep*{Agrawal_2012}. Regardless of the cause, this correlation between IF and retractions means that the correlation between these two factors and submissions will not be a simple one; it may be impossible to truly isolate each factor in an analysis.

We found no correlation with submissions for factors such as acceptance rate and turnaround times. One might expect authors to care about the expected speed of dissemination and the likelihood of being published, and there is some indication of this in the literature. We suspect the reason that there was no correlation is that these metrics are not routinely made available to authors. Editors and publishers might consider advertising acceptance rates and turnaround times to assess the impact on submissions. \citet{Pepermans_2015} found that, for high-quality papers only, framing the metric as an acceptance rate, rather than a rejection rate, did lead to authors being more likely to submit.

We recommend that editors and publishers consider both PRR and IF when analyzing submission numbers. Given that PRR is likely to be a significant factor in determining journal choice, publishers may want to find metrics other than retractions by which to measure it. Journals need sustained or increasing submissions to remain viable, so editors and publishers should aim to maintain a journal’s PRR, as well as its IF, to continue to attract submissions. Underinvesting in the practices and processes around peer review may lead to smaller submission numbers and thereby smaller revenues in the longer term.