Method
There was a fundamental critique of my approach on Twitter:
“Librarian here. -Search strategy poor -Not explicitly defining
“predatory journals” -Heavy reliance on convenience sampling -Clarify
& test inclusion/exclusion criteria -Mechanism for resolving
inter-rater disagreements? Talk. To. Your. Local. Librarians. NOW.”
That’s why I’d like to present my method in more detail, also to hear
your opinion about it. The normal procedure for a systematic literature
review is to find as much literature as possible via search strings in
order to conduct an ”objective” study. This is usually done by using the
large databases Web of Science or Scopus (or, rather rarely,
googlescholar), or by using smaller subject-specific databases. I have
been dealing with this form of procedure for quite some time, and have
also worked with and published on it myself
(
https://link.springer.com/article/10.1007/s10734-016-0045-5
). In
my opinion, however, the methods have one major disadvantage: there is a
lot of noise, i.e. articles are included that have nothing to do with
the research question. Then the author(s) has to sort out the articles,
which are not related to the research question - whereby the criteria
often remain rather vague. That is why I started my search with other
databases and started with a small number of articles. As I described on
my blog:
“I started my search at the Directory of Open Science (DOAJ). “DOAJ is
a community-curated online directory that indexes and provides access to
high quality, open access, peer-reviewed journals.” (https://doaj.org/)
Secondly, I used the Bielefeld Academic Search Engine (base). Base is
operated by Bielefeld University Library and “one of the world’s most
voluminous search engines especially for academic web resources”
(base-search.net). Both platforms are non-commercial and focus on Open
Access publications and thus differ from the commercial publication
databases, such as Web of Science and Scopus. For this project, I
deliberately decided against commercial providers and the restriction of
search in indexed journals. Thus, because my explicit aim was to find
articles that are open in the context of Open Science.”
In addition, it often happens that authors who are frequently quoted are
included in the literature review without reflecting on why they are
highly quoted (I have also done this myself). Reflecting on my own work,
I then began to study bibliometric procedures and literature reviews. In
the process, I encountered the method of bibliometric review.
Bibliometric review combines bibliometric methods (citation analysis
etc.) with a systematic overview of research fields. In my analysis, I
found that very often high citation rates are equated with quality and
that there is little reflexion on how algorithms work in bibliometric
methods such as co-citation analysis. There is also little reflection on
the sample, how high the noise is in the sample and maybe missing
articles.
Based on these findings, I decided to choose a qualitative approach to
the systematic review. That means to judge qualitatively whether
something belongs in the sample or not. The criteria I have applied I
describe in detail in the next section. In this way, I would like to
point out connections that are present in the articles and that actually
refer to aspects of Open Science. To illustrate the connections I
created a figure with the free software yEd
(https://www.yworks.com/products/yed ) and uploaded it to Zenodo
(https://zenodo.org/record/3371415).
Now the following questions arise:
Do you agree to continue along this qualitative path? In addition, are
there perhaps people who would like to try out the ”classical” way? Then
we could compare which results we come to - but that would be an
additional article :)
Kwok: of the definitions are well defined and clear for a qualitative then it could be fine to use, especially since it seems to be well justified. Thus, before committing to this, it would be useful to see what this approach is and how it is implemented.
Max: I'm very much in for the qualitative approach. When reading through your procedure I was wondering if there are many papers that talk about teaching a specific open science practice (e.g. registered reports) without using the term open science. Would it make sense to assemble a corpus using the term "open science" first, then distill open science practices, and then search for those practices again to see if papers come up that do not refer to "open science"? I'm sure that in management studies there are many papers that talk about open access publishing without the term open science for example.
Tarandeep: I like the way you think, Max.
Helen: I agree with Max re terminology. Open research / scholarship are used in the UK. Happy with the qualitative path for this paper.