The effect of different microstructures (granular bainite and martensite) on the low cycle fatigue (LCF) behavior of SA508-IV steel was investigated. The LCF tests were performed with the strain amplitudes of ±0.45% and ±0.6% at 300 °C, respectively. The experimental results showed that the stress concentration between M/A islands and bainitic ferrite matrix was prone to facilitate cracks initiation, which would cause more crack initiation points in granular bainite during the fatigue. The fraction of the high-angle grain boundaries (HAGBs) of granular bainite and martensite were 32.01% and 42.34%, respectively. The HAGBs could effectively hinder crack propagation. In addition, the separation of fatigue striations in the martensite structure was narrow compared to granular bainite structure under the same conditions, suggesting the crack propagation rate of martensite was slower. Finally, the fatigue live of martensite was much longer at the same strain amplitude, which was attributed to less crack initiation, more HAGBs, more fine carbides and less crack propagation rate.