Anomaly detection is widely used in manufacturing and medical imaging. We propose a self-supervised defect detection method based on multi-scale feature fusion, which can effectively improve the detection and localization accuracy. The method of pseudo-defect construction was used to enhance the training data. To make the pseudo-defects more realistic, the extreme point of feature heatmap was used as the anchor point of the defect area, and the defect image was fused with the original image to construct the pseudo-defect. A multi-scale feature fusion network was proposed that utilizes the self-attention mechanism and the interaction between multi-scale features to extract semantic features containing rich contextual information to improve detection and localization accuracy further. The proposed method achieved competitive experimental results on both the MVTec AD and Chest X-ray datasets. Compared with other pseudo-defect simulation methods, the heatmap-based pseudo-defect construction method improves by at least 2%. It achieves comparable results with other state-of-the-art defect detection methods.