The Horn of Africa drylands (HAD) are among the most vulnerable regions to hydroclimatic extremes. The two rainfall seasons — long and short rains — exhibit high intraseasonal and interannual variability. Accurately simulating the long and short rains has proven to be a significant challenge for the current generation of weather forecast and climate models, revealing key gaps in our understanding of the drivers of rainfall in the region. In contrast to existing climate modelling and observation-based studies, here we analyze the HAD rainfall from an observationally-constrained Lagrangian perspective. We quantify and map the major oceanic and terrestrial sources of moisture driving the variability in the long and short rains. Specifically, our results show that the Arabian Sea (through its influence on the northeast monsoon circulation) and the southern Indian Ocean (via the Somali low level jet) contribute ~80% of the HAD rainfall. We see that moisture contributions from land sources are very low at the beginning of each season, but supply up to ~20% from the second month onwards, i.e., when the oceanic-origin rainfall has already increased water availability over land. Further, our findings suggest that the interannual variability in the long and short rains is driven by changes in circulation patterns and regional thermodynamic processes rather than changes in ocean evaporation. Our results can be used to better evaluate, and potentially improve, numerical weather prediction and climate models, which has important implications for (sub-)seasonal forecasts and long-term projections of the HAD rainfall.