Regional cooperation among urban water utilities is a powerful mechanism for improving supply reliability and financial stability in urban water supply systems. Through coordinated drought mitigation and joint infrastructure investment, urban water utilities can efficiently exploit existing water supplies and reduce or delay the need for new supply infrastructure. However, cooperative water management brings new challenges for planning and implementation. Rather than accounting for the interests of a single actor, cooperative policies must balance potentially competing interests between cooperating partners. Structural imbalances within a regional system can lead to conflict between cooperating partners that destabilize otherwise robust planning alternatives. This work contributes a new exploratory modeling centered framework for assessing cooperative stability and mapping power relationships in cooperative infrastructure investment and water supply management policies. Our framework uses multi-objective optimization as an exploratory tool to discover how cooperating partners may be incentivized to defect from robust regional water supply partnership opportunities and identifies how the actions of each regional partner shape the vulnerability of its cooperating partners. Our methodology is demonstrated on the Sedento Valley, a highly challenging regional urban water supply benchmarking problem. Our results reveal complex regional power relationships between the region's cooperating partners and suggest ways to improve cooperative stability.