The interplay between environmental suitability, dispersal and biotic interactions induces spatial patterns of species’ co-abundance. Existing statistical frameworks that infer the underlying interactions from these patterns either ignore the species response to the environment or they fail to account for the asymmetric nature of interactions. Here, we propose a framework that (a) models pair-wise associations as directed influences from a source to a target species, parameterized with two species-specific latent variables: the response of the target species to the community, and the effect of the source species on the community; and (b) jointly fits these associations with a habitat suitability model through a conditional abundance model. Using both simulated and empirical data, we demonstrate the ability of the framework to recover known associations and highlight the properties of the learned association networks. Our framework should now pave the way for getting more accurate pictures of interspecific dependencies from empirical data.