Background: Over 650 million people worldwide lack access to safe water supplies, and even among those who have gained access to ‘improved’ sources, water may be seasonally unreliable, far from homes, expensive, and provide insufficient quantity. Measurement of water access at the level of communities and households remains crude, and better measures of household water insecurity are urgently needed to inform needs assessments and monitoring and evaluation. We set out to assess the validity of a quantitative scale of household water insecurity, and to investigate (1) whether improvements to community water supply reduce water insecurity, (2) whether water interventions affect women’s psychological distress, and (3) the impacts of water insecurity on psychological distress, independent of socio-economic status, food security, and harvest quality. Methods and Findings: Measures were taken before and one to six months after a community water supply improvement in three villages in rural northern Ethiopia. Villages similar in size and access to water sources and other amenities did not receive interventions, and served as controls. Household water insecurity was assessed using a 21-item scale based on prior qualitative work in Ethiopia. Women’s psychological distress was assessed using the WHO Self-Reporting Questionnaire (SRQ-20). Respondents were either female heads of household or wives of the heads of household (n = 247 at baseline, n = 223 at endline); 123 households provided data at both rounds. The intervention was associated with a decline of approximately 2 points on the water insecurity scale between baseline and endline compared to the control (beta -1.99; 95% CI’s -3.15, -0.84). We did not find evidence of impact of the intervention on women’s psychological distress. Water insecurity was, however, predictive of psychological distress (p <0.01), independent of household food security and the quality of the previous year’s harvest. Conclusion: These results contribute to the construct validity of our water insecurity scale, and establish our approach to measuring water insecurity as a plausible means of evaluating water interventions. Improvements to community water supplies were effective in reducing household water insecurity, but not psychological distress, in this population. Water insecurity was an important predictor of psychological distress. This study contributes to an emerging literature on quantitative assessment of household water insecurity, and draws attention to the potential impact of improved access to water on women’s mental well-being.