Local adaptation to temperature is essential for woody plants to against changeable climate and safely survive the winter. To uncover the specific molecular mechanism of low temperature adaptation in woody plants, we performed selective-sweep analysis and genome-wide association study (GWAS) on a wild woody plant naturally distributed in different climate zones and latitudes. We sequenced a core collection of 134 accessions selected from 494 paper mulberry (Broussonetia papyrifera L.), phenotyped the accessions in high latitudes of 40º N for two overwintering traits. We further performed genome-phenotype and genome-environment associations, and genome-wide scans for temperature selection. The population structure analysis indicated that accessions showed forceful geographic distribution patterns because of the adaptation to local climate. We detected 75 selective regions possibly undergone temperature selection and identified 14 trait-associated SNPs corresponded to 16 candidate genes. Meanwhile, low temperature adaptation was also supported by other three SNPs with values lower than threshold but harboring different primary genotype among geographic groups. Overall, we propose a possible network of cold signal perception and responses in woody plants, some genes are considered unique to woody plants while others have been studied in herbs, which highlighting a key hit for studying the specific molecular mechanism of low temperature adaptation or overwintering in woody plants.