To explore the evolution law of pores/cracks and meso-structures in rocks under stress, sandstones were damaged by different stresses to be studied here. The damaged specimens can be divided into two types by macro-mechanical characteristics: Low Damaged Specimens (LDS) and High Damaged Specimens (HDS). For the LDS, pore/crack propagation mainly occurs between skeleton minerals and filler materials. For the HDS, besides pore/crack propagation, micro-pore initiation inside skeleton minerals is the main factor for damage. According to the meso-mechanics model, particle rotation leads to pore/crack propagation under low-stress conditions, but the modification of specimens’ skeleton structure is limited. As the stress increases, particles will be under higher stress because the rotation of particles will gradually weaken, leading to the formation of transgranular cracks, which transform the skeleton structure. Therefore, under different stresses, the meso-evolution behavior of minerals controls the evolution law of pores/cracks and changes meso-structure of rocks.