In order to improve the accuracy of image-based transmission line defect detection, while reducing the computational complexity and the high demand on chip performance, an object detection framework is proposed, which aims to improve model performance without increasing the scale of the model and the amount of calculation. We first introduce an efficient feature fusion module to combine different-level semantic features in nonlinear transformations. It includes channel-level hierarchy features, linear projection and residual mappings to gather task-oriented features across different spatial locations and scales. Then a context information modelling module is proposed to extract features around the target objects, which further increases the detection accuracy. Finally, an Intersection-over-Union-based training examples sampling strategy is adopted to alleviate the class imbalance problem. Experiments on our dataset show that the proposed method, with a similar number of model parameters, has an accuracy improved by 8.1% compared to the baseline, and outperforms all the competitors in the area of transmission line defect detection.