A thematic approach to strategic evaluations
In 2017 WaterAid ran its first global thematic evaluation, on WASH in the health sector and WASH access for people with disabilities. Helen Parker, Strategic Analyst at WaterAid international, shares what we learned, and why it’s useful.
Why thematic evaluations?
Evaluations are a crucial part of accountability. They allow us to review our work, identify what is going well and assess challenges and failures. Evaluations should also be embedded in broader processes of organisational learning, where findings feed into best practice, drive better implementation and provide a baseline for future assessments.
After we published our Global Strategy in 2015, we revised our strategic assessment framework. This required careful consideration and consultation, to ensure that strategic indicators and targets did not create perverse incentives or distract from our mission of transformative change in the WASH sector.
We now have a range of global-level monitoring tools we use to inform effective governance, including annual reporting on user numbers, spending, policy change and programme performance ratings. To complement this, we have moved away from single country evaluations, to focus on thematic evaluations of specific areas of work. This enables us to examine a subject deeply, look at a cross-section of examples and identify successes and challenges.
With the combination of rolling thematic evaluations and annual review we aim to achieve breadth and depth in the assessment of our work.
The first thematic evaluation
From May to December 2017, colleagues from WaterAid Sweden, WaterAid international and WaterAid UK managed our first major global thematic evaluation for this strategy. The evaluation focused on the themes of WASH in the health sector and WASH access for people with disabilities, which fall under our strategic aims of ‘equality’ and ‘integration’ respectively.
An external team conducted the evaluation, which included: desk-based document review; focus group discussions and interviews; field-based case study visits to Madagascar, Tanzania, Nepal and India; and desk-based case studies of Cambodia, Mali and Uganda. The method was based on a revised version of the DAC criteria to measure relevance, efficiency, effectiveness and sustainability. Impact was not considered because of the burden of proof needed to link cause and effects, but there were qualitative, anecdotal examples of potential impacts.
WASH in health
WaterAid’s WASH in health work integrates service delivery, capacity-building and advocacy, in partnership with Ministries of Health and other health actors. We focus on healthcare facilities, nutrition, neglected tropical diseases and hygiene. The evaluation identified areas of success including active global advocacy for the relationships between WASH, health, nutrition and maternal and child health (e.g. through Healthy Start); national-level capacity building and improved coordination (e.g. supporting cross-sector working groups and joint sector reviews); and good examples of service delivery in healthcare settings.
In health clinics that the evaluation team visited in Tanzania and India, for example, staff and local health officials were very positive about WaterAid projects. They said that initiatives to provide a water supply, hygiene training, rehabilitated toilets and new waste management facilities (incinerators and placenta pits) had contributed to fewer sepsis-related deaths, reduced infections and lower incidence of diarrhoea (although data were limited). Importantly, staff also felt happier and more motivated by the cleaner environment and increased patient satisfaction.
In Nepal, the evaluation team met with Government partners and watched an interactive session by female community health volunteers as part of our hygiene promotion through immunisation project. The team were impressed that the initiative successfully leveraged Government engagement and financial support in a highly resource-constrained environment.
Of course, the evaluation also identified challenges. For example, in some health clinics, water supply was intermittent, soap was not available, or toilets were broken. Government partners requested that WaterAid provide more funds for further projects, as they were unable to replicate the models. Recommendations focused on: sustaining services in healthcare facilities; building stronger partnerships to facilitate maintenance, upscaling and replicating programmes; and improved monitoring of our work to address sustainability issues and inform evidence-based decision making.
WASH access for people with disabilities
Equity and inclusion is critical to meet our strategic vision and Sustainable Development Goal 6 on universal access to water and sanitation. The evaluation found that WaterAid has made significant investments in our inclusion agenda, leading to a high level of internal awareness and commitment. The evaluators also suggested WaterAid was a leader in promoting disability inclusion in the WASH sector.
The evaluation was positive about WaterAid’s effective partnerships, engaged advocacy, inclusive service delivery models and successful conversion of research into practical tools. For example, WaterAid initiatives have strengthened partner capacities, developed disability-inclusive replicable tools, manuals and guidelines (such the Compendium of accessible WASH technologies), built inclusive public WASH facilities (e.g. in schools in Madagascar and public toilets in Bangladesh) and contributed to cutting-edge research, for example on disability and menstrual hygiene management.
WaterAid’s commitment to working in partnership with disabled people’s organisations to claim their rights to water and sanitation was particularly commended, reflecting the disability empowerment mantra of ‘nothing about us, without us’. For example, in India, WaterAid is partnered with the Odisha Disabled People’s Forum. The group were very clear about the benefits of the partnership: awareness around WASH; training to conduct accessibility and safety audits on public taps and toilets; and, most importantly, ‘collective empowerment’ to organise around advocating rights beyond WASH.
However, one of the main challenges identified both internally and among partners was a lack of full understanding (and addressing) of the barriers that affect disabled people, particularly around attitudes and stigma. The evaluation also identified the need for improved standardisation of and adherence to our own best practice guidelines, as different parts of WaterAid have different levels of understanding around inclusive programming. It was recommended that extra effort and cross-organisational learning is needed to ensure universal best practice.
What next?
Thematic evaluations do not replace the need for regular monitoring or for country-level reporting according to countries’ own strategic priorities. However, they provide a touchstone and full-colour picture of our work in a specific area, shining a spotlight on the challenges we need to address. The first evaluation provided useful insights and valuable recommendations, which now act as a mandate for us to address challenges more systematically and with ongoing follow up.
We will continue to conduct thematic evaluations regularly and the next is well underway, with an ambitious scope of ‘sector strengthening for sustainable services’ – another one of our four strategic aims. Watch this space for the results.
Read the recent evaluation executive summary here.
