River deltas commonly have a heterogeneous substratum of alternating peat, clay and sand deposits. This has important consequences for the river bed development and in particular for scour hole formation. When the substratum consists of an erosion resistant top layer, erosion is retarded. Upon breaking through a resistant top layer and reaching an underlying layer with higher erodibilty, deep scour holes may form within a short amount of time. The unpredictability and fast development of these scour holes makes them difficult to manage, particularly where the stability of dikes and infrastructure is at stake. In this paper we determine how subsurface lithology controls the bed elevation in net incising river branches, particularly focusing on scour hole initiation, growth rate, and direction. For this, the Rhine-Meuse Estuary forms an ideal study site, as over 100 scour holes have been identified in this area, and over 40 years of bed level data and thousands of core descriptions are available. It is shown that the subsurface lithology plays a crucial role in the emergence, shape, and evolution of scour holes. Although most scour holes follow the characteristic exponential development of fast initial growth and slower final growth, strong temporal variations are observed, with sudden growth rates of several meters per year in depth and tens of meters in extent. In addition, we relate the characteristic build-up of the subsurface lithology to specific geometric characteristics of scour holes, like large elongated expanding scour holes or confined scour holes with steep slopes. As river deltas commonly have a heterogeneous substratum and often face channel bed erosion, the observations likely apply to many delta rivers. These findings call for thorough knowledge of the subsurface lithology, as without it, scour hole development is hard to predict and can lead to sudden failures of nearby infrastructure and flood defence works.