In the current state of the art of process industries/manufacturing technologies, computer-instrumented and computer-controlled autonomous techniques are necessary for damage diagnosis and prognosis in operating machinery. From this perspective, the paper addresses the issue of fatigue damage that is one of the most commonly encountered sources of degradation in polycrystalline-alloy structures of machinery components. It is possible to conduct in-situ detection & classification of damage as well as an assessment of the remaining service life through ultrasonic measurements of material degradation and their computer-based analysis. In this paper, tools of machine learning (e.g., convolutional neural networks (CNNs)) are applied to synergistic combinations of ultrasonic measurements and images from a confocal microscope (Alicona) to detect and evaluate the risk of fatigue damage. The database of the confocal microscope has been used to calibrate the ultrasonic database and to provide the ground truth for fatigue damage assessment. The results show that both the ultrasonic data and confocal microscope images are capable of classifying the fatigue damage into their respective classes with considerably high accuracy. However, the ultrasonic CNN model yields better accuracy than the confocal microscope CNN model by almost 9%.