The sterility of hybrids from interspecific hybridization limits speciation, which widely exists in interspecific hybrids of hermaphroditic plants and also been found in the F1 hybrids of hermaphroditic Argopecten scallops. However, the underlying mechanism of sterility remained unexplored in hermaphroditic animals. In this study, we firstly investigated the mechanism of sterility in the interspecific F1 hybrids of Argopecten scallops by examining the mutations in mitochondrial genes and expressions of nuclear genes. Our results showed that the ATP content in gonads of F1 hybrids was significantly lower than their parents, indicating that energy deficiency may be an immediate cause of sterility in F1 hybrids. The SNP variation types of transition in mitochondrial genes (CYTB, ND4, ND2, ATP6, and COX2) could change their hydrophobicity of amino acids and protein structures, which might contribute to sterility in F1 hybrids. The mutations and rearrangements of mitochondrial genes and abnormal expression of nuclear genes were found in F1 hybrids. Abnormal interaction between the mitochondrial and nuclear genes might contribute to sterility of the F1 hybrids through the following pathways: (1) inhibition of oocyte maturation by enhanced expression of Mfn2; (2) cell cycle arrest in G1 phase of oocytes by inhibited expression of CDK2; (3) cell apoptosis induced by mitochondrial apoptosis; (4) insufficient energy supply from abnormal mitochondria, and (5) mitochondrial dysfunction resulted from abnormal expression of other nuclear genes. The mechanism of sterility in interspecific hybrids of hermaphroditic animals would provide more information for understanding the reproductive isolation and exploitation of heterosis.