Introduction
Zenex focuses on supporting the development of effective models that can be implemented at a systemic level in partnership with the Department of Basic Education (DBE). Zenex also manages a range of projects to benefit the early grade education system using structured pedagogical approaches.
Since 2015, the Programme to Improve Learning Outcomes (PILO) in KwaZulu-Natal has been implementing such a project. PILO is an NGO that designs and tests systems to improve learning in public schools, specifically by building district-level capacity. In 2020, PILO embarked on an embedding intervention to institutionalise new practices in provincial education departments.
Projects such as this have the potential to be integrated successfully into the system. However, not enough is known about the processes of embedding these in the system, how to measure the success of embedding, and what the enablers and inhibitors of embedding are.
Zenex is thus supporting a thorough evaluation of this intervention to document the process and determine what works within the system. JET Education Services will evaluate the programme.
Overview
The evaluation of PILO’s programme is taking place in 2021 and 2022 and it employs a case study design to understand behavioural and cultural changes in context. Contribution analysis is being used to identify the impact of the intervention and the factors involved in promoting and preventing change.
Qualitative and quantitative data is being collected from monitoring documentation; interviews and focus groups with stakeholders, including officials, principals and teachers; observations of training, meetings and practices at different levels in the system; and surveys with teachers and school principals. The data is derived from rural and urban education settings in KwaZulu-Natal.
The sample is large and the research involves about 16 different focus group discussions, more than 30 individual interviews, numerous contextual observations, and two surveys focusing on different role players.
The researchers are using this rich data to piece together a performance story or evidence-based narrative of the programme’s achievements and how these have been accomplished. The credibility and sufficiency of the evidence will be assessed. The study will also involve reflective exercises considering whether its approach is the most appropriate for measuring the programme’s success and the level of embeddedness.
Objectives
The primary objective of the evaluation is to determine how effective PILO has been in embedding changes at the school and district level by the end of 2022. To meet this objective, the evaluation of the success of PILO’s embedding exercise will focus on answering the following questions:
- Have the desired qualitative changes to the targeted provincial, district and school knowledge, protocols, processes and procedures occurred?
- If changes are observed, are they a result of PILO’s embedding programme, or are there alternative explanations for the changes observed?
- Which factors enable or constrain PILO’s embedding activities, and are these activities appropriate and sufficient?
Outcomes
The evaluation will produce the following:
- Formative and final summative reports incorporating feedback from the draft report.
- Presentation to key stakeholders, including findings, conclusions and recommendations.
Conclusion
Building on knowledge gained about what works at scale from its previous projects, Zenex is currently seeking to understand how to embed initiatives within the system and thus make lasting changes in the education system. The evaluation will be essential to understanding embedding and ensuring the traction of Zenex’s and other donors’ systemic initiatives.