Patients with thyroid-associated opthalmopathy reportedly exhibit widespread morphological abnormalities in attention and default mode network areas, but alterations in coordinated patterns of brain structural networks underlying thyroid-associated opthalmopathy are still poorly understood. This study aims to investigate the disruptions in gray matter structural networks between thyroid-associated opthalmopathy patients in active and inactive phases. We constructed subject-wise structural networks based on gray matter volumetric features for 37 active and 35 inactive thyroid-associated opthalmopathy patients, and 23 healthy controls. Graph theory analysis was conducted to assess the alterations of global and regional network topological properties for thyroid-associated opthalmopathy patients and their correlation with clinical characteristics. Compared with inactive thyroid-associated opthalmopathy patients and controls, active thyroid-associated opthalmopathy patients showed significantly decreased global/local efficiency, and significant altered nodal betweenness centrality in the visual, attention, default mode network and sensorimotor areas. Intriguingly, the betweenness centrality of left lingual gyrus was significantly negatively correlated with bilateral intraocular pressure, and medial superior/opercular inferior frontal gyrus was significantly correlated with anxiety and depression parameters. Our results suggest that thyroid-associated opthalmopathy patients at different clinical stages have segregated disruptions in gray matter network topology, which may due to the visual function and psychological changes. These findings may provide new insights into the neurobiological mechanisms of thyroid-associated opthalmopathy.