Automatic segmentation of multi-modal Cardiac Magnetic Resonance Imaging (CMRI) scans is challenging due to the variant intensity distribution and unclear boundaries between the neighbouring tissues and other organs. The deep convolutional neural networks have shown great potential in medical image segmentation tasks. In this paper, we present a deep convolutional neural network model named Multi-Modal Cardiac Network (MMC-Net) for segmenting three cardiac structures namely right ventricle (RV), left ventricle (LV), and left ventricular myocardium (LVM) from multi-modal CMRI’s. The proposed MMC-Net is designed using a densely connected backbone enabling feature reuse, an atrous convolution module for fusing multi-scale features, and a pixel-classification module for generating the segmentation result. This model was evaluated on a publicly available MS-CMRSeg-2019 challenge dataset in segmentation of RV, LV, and LVM from CMRI scans. The segmentation results from extensive experiments demonstrate our MMC-Net can achieve better segmentation performance compared to other state-of-the-art models, and the existing approaches. Additionally, the generalization ability of the proposed MMC-Net is validated on another publicly available ACDC dataset without fine-tuning. The results demonstrate that the proposed MMC-Net shows a powerful generalisation ability of segmenting RV, LV, and LVM with higher performance.