Quality assessment of included studies
The agreement between the reviewers regarding the report and
methodological quality was 82.35% and 94.12%, respectively. All
discrepancies were removed by two experts. Regarding the methodological
quality, most of the studies (15) had a low risk of bias (Da Silva et
al., 2019; Guo et al., 2016; He et al., 2019; M.-J. Kim et al., 2019; Y.
S. Kim et al., 2016; Laksmitawati et al., 2016; H. A. Lee et al., 2017;
S. B. Lee et al., 2017; Lim, Kim, et al., 2015; Lim, Lee, et al., 2015;
Meram & Wu, 2017; Mohr et al., 2019; Pang et al., 2019; Sun et al.,
2017; Zhang et al., 2015) and two had a moderate risk of bias (Ghate et
al., 2018; Karatoprak et al., 2019) (Supplementary Material S8 - Table
4B, and S9). No study had a high risk of bias. Regarding the quality of
the reports, eight of the 17 studies had a moderate risk, while 9 had a
low risk of bias.
In all the SCIRAP evaluation criteria, there was an item that was not
covered by almost any of the studies, but one that we consider crucial
for the quality of the articles. For example, in the test compound and
controls session, only one author (H. A. Lee et al., 2017) described the
vehicle used to dilute the compounds. Similarly, in the test system
session, only two authors reported the number of cell passages used to
perform the experiments (M.-J. Kim et al., 2019; Mohr et al., 2019), and
no author provided information on the measures taken to track or avoid
contaminations by mycoplasma, bacteria, fungi or virus. Furthermore,
only two authors (Guo et al., 2016; H. A. Lee et al., 2017) provided
complete information regarding the toxicity of the test compound as well
as the phlogistic agent (LPS) (Supplementary Material S8 - Table 4A, and
S9).
In the methodological quality, the lack of information about the vehicle
used, as well as the lack of tests to verify the interference of
possible cytotoxicity of the compounds in the results, were the negative
aspects that were present in practically all the studies. In addition, 9
studies did not provide information on replicates or repetitions of the
experiments to verify the reproducibility/reliability of the results (Da
Silva et al., 2019; Ghate et al., 2018; Karatoprak et al., 2019; M.-J.
Kim et al., 2019; Y. S. Kim et al., 2016; S. B. Lee et al., 2017; Lim,
Kim, et al., 2015; Lim, Lee, et al., 2015; Sun et al., 2017). It is
important to note that all the studies provided complete information and
were extremely reliable in regard to the administration of the test
compound. This was to be expected, given the strictness of the
pre-established eligibility criteria (Supplementary Material S8 - Table
4B, and S9).
In the relation of the phlogistic agent (LPS), we considered the LPS
effects with indirect relevance for all the studies that did not provide
information on the manufacturers or sources of LPS strains used (Da
Silva et al., 2019; Ghate et al., 2018; Karatoprak et al., 2019; M.-J.
Kim et al., 2019; Laksmitawati et al., 2016) (Supplementary Material S8
- Table 4C, and S9). The other items (test systems, endpoint and
concentrations) were considered of direct relevance. For the tested
system, as expected, all the studies were of direct relevance, because
the experimental model in RAW cells induced by LPS was the most used to
mimic the inflammatory process in vitro , and was one of the
eligibility criteria used for the inclusion of studies. Likewise, in
terms of the endpoint and concentration, all the articles were also of
direct relevance, as the studies had to meet specific eligibility
criteria at the time of collection/treatment, and a particular
concentration of LPS. All the studies were considered reliable, with
restrictions (Supplementary Material S8 - Fig 9), as none completely met
with the guidelines in regard to the reporting and methodological
assessment. In summary, the studies presented some restrictions as to
the origin of the cell, a number of cell passages (15), and
microbiological control (All studies). Also, some studies did not report
toxicity (15), while others did not report experimental and biological
replicates (9).