Method

There was a fundamental critique of my approach on Twitter:
“Librarian here. -Search strategy poor -Not explicitly defining “predatory journals” -Heavy reliance on convenience sampling -Clarify & test inclusion/exclusion criteria -Mechanism for resolving inter-rater disagreements? Talk. To. Your. Local. Librarians. NOW.”
That’s why I’d like to present my method in more detail, also to hear your opinion about it. The normal procedure for a systematic literature review is to find as much literature as possible via search strings in order to conduct an ”objective” study. This is usually done by using the large databases Web of Science or Scopus (or, rather rarely, googlescholar), or by using smaller subject-specific databases. I have been dealing with this form of procedure for quite some time, and have also worked with and published on it myself (https://link.springer.com/article/10.1007/s10734-016-0045-5 ). In my opinion, however, the methods have one major disadvantage: there is a lot of noise, i.e. articles are included that have nothing to do with the research question. Then the author(s) has to sort out the articles, which are not related to the research question - whereby the criteria often remain rather vague. That is why I started my search with other databases and started with a small number of articles. As I described on my blog:
“I started my search at the Directory of Open Science (DOAJ). “DOAJ is a community-curated online directory that indexes and provides access to high quality, open access, peer-reviewed journals.” (https://doaj.org/) Secondly, I used the Bielefeld Academic Search Engine (base). Base is operated by Bielefeld University Library and “one of the world’s most voluminous search engines especially for academic web resources” (base-search.net). Both platforms are non-commercial and focus on Open Access publications and thus differ from the commercial publication databases, such as Web of Science and Scopus. For this project, I deliberately decided against commercial providers and the restriction of search in indexed journals. Thus, because my explicit aim was to find articles that are open in the context of Open Science.”
In addition, it often happens that authors who are frequently quoted are included in the literature review without reflecting on why they are highly quoted (I have also done this myself). Reflecting on my own work, I then began to study bibliometric procedures and literature reviews. In the process, I encountered the method of bibliometric review. Bibliometric review combines bibliometric methods (citation analysis etc.) with a systematic overview of research fields. In my analysis, I found that very often high citation rates are equated with quality and that there is little reflexion on how algorithms work in bibliometric methods such as co-citation analysis. There is also little reflection on the sample, how high the noise is in the sample and maybe missing articles.
Based on these findings, I decided to choose a qualitative approach to the systematic review. That means to judge qualitatively whether something belongs in the sample or not. The criteria I have applied I describe in detail in the next section. In this way, I would like to point out connections that are present in the articles and that actually refer to aspects of Open Science. To illustrate the connections I created a figure with the free software yEd (https://www.yworks.com/products/yed ) and uploaded it to Zenodo (https://zenodo.org/record/3371415).
Now the following questions arise:
Do you agree to continue along this qualitative path? In addition, are there perhaps people who would like to try out the ”classical” way? Then we could compare which results we come to - but that would be an additional article :)
Kwok: of the definitions are well defined and clear for a qualitative then it could be fine to use, especially since it seems to be well justified. Thus, before committing to this, it would be useful to see what this approach is and how it is implemented. 
Max: I'm very much in for the qualitative approach. When reading through your procedure I was wondering if there are many papers that talk about teaching a specific open science practice (e.g. registered reports) without using the term open science. Would it make sense to assemble a corpus using the term "open science" first, then distill open science practices, and then search for those practices again to see if papers come up that do not refer to "open science"? I'm sure that in management studies there are many papers that talk about open access publishing without the term open science for example.
Tarandeep: I like the way you think, Max. 
Helen: I agree with Max re terminology. Open research / scholarship are used in the UK. Happy with the qualitative path for this paper.