Results
Our analysis of simulated datasets showed the rate of true positives
(probability of a core taxon assigned as such or signal) is close to one
in many cases and appears to provide support for the ability of those
methods to correctly assign core taxa (Figure 1a, denoted in blue).
Furthermore, the rate of false positives (probability of a non-core
taxon assigned as core member or noise) is close to zero in many cases
seemingly providing additional support for core assignment methods
(Figure 2b, denoted in blue). However, when examined individually these
two metrics only tell half the story, as we are concerned with the
ability of a given method to accurately identify the core taxa (i.e.
true positives), while not over inflating membership through inclusion
of non-core taxa (i.e. false positives); thus, being able to discern
signal from noise.
The net assignment scores for simulations revealed the inability of the
methods to accurately assign core membership (Figure 1c). The net
assignment value quantifies the absolute difference in true positives
(signal) and false positives (noise), with a net assignment value of 25
meaning the method assigned all of the correct taxa to the core with no
erroneous assignments and smaller values indicating poorer performance
in accurate core assignment. Our results show that rarely did the
methods accurately recover the correct number and identity of core taxa
(those simulated to be included in the core). In general, a large
difference in the abundance of core and non-core taxa
(πcore/πnon-core, with varying degrees
of precision), led to the greatest success in accurate identification of
the correct 25 core taxa (Figure 1c, right side x-axis, success denoted
by dark blue squares, white and red indicate poor performance). When
comparing results of the four core assignment criteria, the proportion
of sequence replicates and proportion of sequence reads and replicates
methods most often accurately assigned the 25 core taxa, with multiple
instances of a net assignment value >24 (Figure 1c). The
two methods tha0t utilized the proportion of replicates produced similar
results in our simulations. They were followed by the hard cutoff method
and then the cumulative proportion of sequence reads method (Figure 1c).
All methods, with the exception of the cumulative proportion of sequence
reads, were able to accurately recover the known core in some
circumstances (net assignment value >24). However, they did
so for different ranges of parameter combinations, suggesting each
method may better suited to different taxon distributions.
Even though core methods accurately assigned core membership in some
circumstances, the same methods produced negative net assignment values
in other situations, consistent with overestimation of core membership.
Core inclusion was most severely overestimated in the cumulative
proportion of sequence reads and hard cutoffs methods in simulations
with low πcore to πnon-core ratio and
high precision (parameterized by θ). This overestimation manifested as a
high false positive rate (noise) in certain simulated communities. In
general, the methods based on proportionality tended to assign the
smallest set of core taxa and possessed the best net assignment value
(i.e. correct assignment of known core taxa and limited erroneous
assignment of non-core taxa to the core) and as such could be considered
the most conservative.
For the two published datasets, the four core methods led to different
conclusions, with the inferred core corresponding to 1.21%-15.74% of
total taxa (Table 2). All methods assigned taxa with high abundance to
the core, though methods differed in their assignments with respect to
CV among replicates (Figure 2). More specifically, the cumulative
proportion of sequence reads method and the proportion of sequence
replicates method included highly abundant taxa regardless CV in both
datasets. The method based on proportionality of replicates and sequence
reads selected only abundant taxa with a relatively low CV in the human
microbiome dataset (Figure 2a) and selected abundant taxa regardless of
CV in the Arabidopsis dataset (Figure 2b). Finally, the core
method that uses both the proportion of reads and replicates appear to
arbitrarily exclude taxa with relatively high mean abundance and low CV,
taxa that fit multiple criteria for core membership. This is especially
evident in the Human Microbiome Project dataset. These exclusions
highlight problems associated with assigning continuously distributed
count data into core and non-core groups.
Examination of core assignments in the published datasets showed that
co-assignment (i.e. common core assignment by multiple methods) varied
depending on the dataset (Figure 3). The Human Microbiome Project
dataset yielded 176 core assignments that were assigned by all four
methods (9.5% of total unique core assignments). The Arabidopsisdataset produced 165 core assignments that were shared among all four
methods (8.1% of total unique core assignments). These common core
assignments equivalate to 1.49% and 1.1% of the total number of taxa
in each taxon table, respectively. For the Arabidopsis dataset,
758 taxa (37.2% of total unique core assignments) were assigned to the
core by two methods and 322 taxa (15.8% of total unique core
assignments) by three methods. As for the Human Microbiome Project
dataset, 404 taxa (21.8% of total unique core assignments) were
assigned to the core by three methods, and 530 taxa ( 28.6% of total
unique core assignments) were assigned by two methods.
Comparisons of differences in beta-diversity between assigned cores and
the full datasets, showed that in some cases the core datasets matched
the entire dataset, but this was not always true. The entireArabidopsis dataset showed both developmental stage and genotype
to be significant in structuring the community (p=0.001); this was true
for both the Bray-Curtis and Jaccard dissimilarity indices. The taxon
table including only taxa assigned by all four methods matched these
results (p=0.001) when using Bray-Curtis dissimilarity, but the Jaccard
index only resulted in a significant effect of developmental stage
(p=0.001) with genotype not significant predictor (p=0.132).
Beta-diversity analysis of the core communities based on each of the
four core-assignment methods separately mostly produced the same effects
on beta-diversity as observed for the entire dataset, except the hard
cutoff method. However, this method had comparable results to the taxon
table created from taxa co-assigned by all four methods, with
developmental stage being significant for both the Bray-Curtis and
Jaccard dissimilarity indices (p=0.001), and genotype being significant
for the Bray-Curtis index (p=0.001) but not Jaccard (p=0.148).
As for the Human Microbiome Project dataset, estimates of beta-diversity
were affected by the use of core taxa or all taxa, raising concern for
interpretation and the validity of core assignments. The full Human
Microbiome Project stool dataset showed both sex and sequencing center
to be significant (p<0.01), while visit number was shown to be
statistically insignificant (p>0.05). These results were
true for both Bray-Curtis and Jaccard dissimilarity indices. When
examining only taxa assigned by all four core assignment methods, visit
number, sex, and sequencing center were all significant
(p<0.05) with Bray-Curtis dissimilarity, but only sequencing
center was significant (p<0.001) with the Jaccard index.
Results of beta-diversity analysis based on the core communities
determined by each of the four core-assignment methods were similar to
the results from the full dataset for both dissimilarity indices, except
for the proportion of replicates reps and reads method. While the
proportion of replicates and reads agreed with the others on the
significance of sequencing center, this core assignment method showed
visit number to be significant (p<0.05) for Bray-Curtis
dissimilarity and insignificant for Jaccard dissimilarity
(p>0.05). As for sex, Jaccard dissimilarity was
insignificant (p>0.05) and Bray-Curtis dissimilarity was
significant (p< 0.01).
Comparison of core assignments to taxa deemed important by their degree
centrality revealed further disagreement. The Arabidopsis dataset
produced 2,258 taxa that were deemed important, either by any of the
four core assignment methods or by the cooccurrence network (Figure
4a,c). Of these 2,258 taxa, 1655 (73.3%) were uniquely assigned by the
core methodologies, while 222 (9.8%) were assigned by the network
alone. A small number of taxa, 381 (16.9%), was identified by both core
assignment methods and the network analysis. The average degree
centrality of taxa assigned as core by any method was 9.4, while the
average degree centrality of non-core taxa was 0.25. This dataset
produced large number of taxa deemed important by solely core
assignment, with 1655 core taxa possessing zero significant edges in the
network. The top 62 taxa, determined by degree centrality, were
identified by the core assignment methods as well. The taxa with the
highest degree centrality not picked up by core methods had a degree
centrality of 118. On the other hand, the human microbiome project
dataset produced very different results, with 3,181 taxa being
identified as important by either the any of the core assignment methods
or the network (Figure 4b,d), and almost half, 1586 (49.86%),
identified by both core assignment methods and the network analysis. Of
these 3,181 taxa, only a small portion of 264 (8.3%) were uniquely
assigned by the core methodologies, while 1331 (41.84%) were assigned
by the network alone. The average degree centrality of taxa assigned as
core by any method was 22.9, while the average degree centrality of
non-core taxa was 2.2. In the human microbiome network, the top 61 taxa,
in terms of degree centrality, were identified by both the cooccurrence
network and core assignment methods. The taxa with the highest degree
centrality not picked up by core methods had a degree centrality of 98.