Khulisa completed an External Evaluation of the Allan Gray Orbis Foundation’s Programs in South Africa in 2018-2019. The evaluation was conducted by a team from Khulisa that included two youth entrepreneurship specialists. During the inception phase, we conducted an evaluability assessment to determine the extent to which programs could be evaluated, after which we were able to propose a suitable design to the evaluation. The evaluation included: 1) design, including a review/revision of the Theory of Change 2) process and 3) outcomes/impact evaluation. To determine outcome/impact, Khulisa used a quasi-experimental design where we retrospectively followed program beneficiaries on their entrepreneurial development path to assess how they were supported by the programs. They were compared with a cohort of applicants who almost were accepted into the program.
Khulisa followed a multi-stakeholder participatory approach. We conducted interviews with key Foundation stakeholders and used a combination of quantitative and qualitative data collection methods such as surveys, group discussions, and case studies to gather data from past and current program beneficiaries (Scholars, Candidate Fellows and Association members) and a comparison groups. The evaluation provided a number of strategic and programmatic recommendations to be taken forward by the Foundation in strengthening their programs.
Since 2016, Khulisa designed and has delivered a three-week Monitoring & Evaluation (M&E) for International Development course for graduate students at the Johns Hopkins University (JHU) School of Advanced International Studies (SAIS). The course leads students through a comprehensive overview and application of M&E and learning, including best practices, current challenges and trends in the field, and real-life examples. Through a mixture of lecture and practical application, students work in groups to design and present M&E plans based on case studies of past and current Khulisa M&E projects, and also performance indicators for these case studies. Luke Connolly, Associate Director, Internships & Experiential Learning at JHU SAIS, says this of Khulisa: “[They have] outstanding knowledge in their field; excellent and timely communications; personable employees; services are highly rated by students who consume their course.
USAID Southern Africa contracted Khulisa to assist with planning, designing, monitoring, evaluating and learning from interventions focused on improving the reading skills of primary grade learners as part of the Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate (PERFORMANCE) activity.
Under this contract, Khulisa conducted the Design and Implementation Evaluation of the Foundation for Professional Development (FPD) run Reading Support Project (RSP). Comprising 263 schools and over 1500 teachers, the RSP is focused on improving the reading teaching practices of Grade 1 to 3 teachers in Setswana and English First Additional Language (EFAL) in two districts in the North West Province of South Africa, with the ultimate outcome to improve learners’ reading skills.
The first step in the evaluation was to assesses the feasibility of the RSP reaching its intended outcomes by looking at the program design, through a design evaluation. A key lesson from this design evaluation is that through developing process maps for this intervention, we (the evaluators) were able to identify potential barriers, challenges, and pathways to enable the achievement of the outcomes. It led to a number of recommendations for strengthening the design of the RSP. The report is available here.
Khulisa is currently conducting the implementation evaluation of the RSP, guided by findings from the design evaluation. Activities for this part of the evaluation include a review of RSP monitoring data and Data Quality Assessment (DQA), a fidelity review, a cost and time analysis, coaching case studies, and interviews and surveys with coaches, teachers, principals, School Management Team (SMT) members, district and provincial stakeholders, and other actors.
To support the Government of South Africa, USAID Southern Africa is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) program. The overall goal of PERFORM is to improve the reading skills of primary grade learners. Khulisa was awarded the Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate (PERFORMANCE) Indefinite Delivery Indefinite Quantity (IDIQ) contract to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from the PERFORM interventions. Under this IDIQ contract, Khulisa has performed data collection for the Early Grade Reading Study (EGRS) Phase 1 & Phase 2 in North West province. This Task Order entailed large scale data collection, collation, analysis, and reporting services to USAID and the South African Department of Basic Education (DBE).
Khulisa conducted oral and written early grade reading assessments with 9,656 learners. Khulisa also collected survey and observation data from 215 school principals, 631 teachers, and 3,462 parents. Khulisa collated, cleaned, and analyzed the data and supported the development of a technical report and learning brief. One of the unique features of the Task Order was the development and administration of a School Functionality Tool, which was used to collect 217 observations of school functionality. Full reports are available here: Early Grade Reading Study
Khulisa was awarded the five-year Scaling Up Nutrition Learning and Evaluation (SUN LE) project from USAID Zambia. Zambia’s SUN program is a cross-ministry, multi-sectoral initiative aimed at reducing the country’s very high rate of childhood stunting. SUN LE provides a wide range of research, evaluation, and learning services/support to Zambian SUN and implementation partners to facilitate learning for improved program management and to support SUN program accountability through tracking outcomes.
Khulisa conducted a performance evaluation of three biodiversity and ecotourism activities in Mozambique: the Arco Norte, Lake Niassa and Gorongosa restoration projects.
The evaluation assessed the projects’ effectiveness, impact and sustainability to inform future USAID and/or Government of Mozambique biodiversity and tourism activities. Read report
Khulisa evaluated the education support provided to Orphans and Vulnerable Children (OVC) in KwaZulu-Natal. The project was implemented by Research Triangle Institute (RTI International) in partnership with the KZN Department of Education, MiET Africa and JET Education Services. Read report
USAID, in partnership with the Western Cape Education Department in South Africa, commissioned Khulisa to conduct an impact evaluation of the provincial Emergent Literacy(ELIT) project in Grade R under USAID’s School capacity & innovation program (SCIP). The project provided training and resources in emergent literacy, including an orientation to an early literacy programme called Stellar (Strengthening Teaching of Early Language and Literacy in Grade R) to all Grade R teachers in the Western Cape. Khulisa deployed a quasi-experimental research design in this provincial-wide study. The evaluation demonstrates our ability to design, execute and analyse robust impact evaluations. Read report
The Zenex Foundation commissioned Khulisa to evaluate the effectiveness of a Community of Practice (CoP) model and methodology for a Mathematics and Science learner support community through the BRIDGE project. The evaluation aimed to explore whether BRIDGE’s model and methodology provided an innovative approach for developing sustainable CoPs, and whether participation in CoPs results in meaningful practice and policy change through to government level. Khulisa found that BRIDGE had maintained a high level of participation in their CoPs. Support from a wide range of stakeholders (NGOs, businesses, Foundations, CSIs, government, academia and individuals) attested to the value of participating in CoPs. This included the quality of dialogue and presentations, networking and sharing, quality information products as well as knowledge generation and management.
Khulisa’s Jennifer Bisgard facilitated sessions on evaluation design for the foundation course and evaluability assessments for the advanced M&E course at the World Bank Center for Learning on Evaluation and Results (CLEAR-Anglophone Africa) as part of the Development Evaluation Training Programme in Africa, training officials from 22 African countries in August, 2017.
Khulisa conducted an external evaluation on USAID| South Africa’s “Systems Strengthening for Better HIV/TB Outcomes project” that was implemented by 7 District Support Partners (DSPs) working in 8 provinces across 21 districts of South Africa. The evaluation assessed the progress that the project made towards achieving pre-determined technical assistance objectives, anticipated outputs, and outcomes. It assessed the quality of the DSPs’ project implementation and determined which technical assistance approaches and activities worked and why. Findings of the evaluation were presented USAID| South Africa to inform the next phase of program design. Read report.
The Managing Democratic Elections in Africa (MDEA) Program is a regional elections capacity building project that trains African election officials on electoral management principles. Implemented by the University of South Africa (UNISA) in partnership with the Independent Electoral Commission, Khulisa’s evaluation found that the project was successful in bringing African election officials together and creating informal networks for learning. Read more
For Earthlife Africa, Khulisa conducted a baseline study on the sustainable use of natural resources to improve climate change resilience amongst 350 women in eight South African communities. The project aims to uplift women in poverty by empowering them to manage and use natural resources, contribute to sustainable livelihoods, and improve their resilience to climate change as well as support training and capacity building of selected women in communities. The assessment covered the type of energy, water sources and farming solutions the women deployed in the Gauteng, Limpopo and Western Cape provinces.Khulisa employed a mixed methods approach eliciting information from a variety of different sources, which formed the basis for the impact assessment of the project at mid-term review and at the end of the project evaluation.
Khulisa evaluated the 3-year “Empowering Local Radio with ICTs” project in seven African countries for the United Nations Educational, Scientific, and Cultural Organization (UNESCO). The project aims to increase the quality of local radio broadcasting by integrating the use of ICTs, strengthening local reporting and engaging as beneficiaries, poor people’s participation in debates related to their social needs and livelihoods. Khulisa evaluated the project progress towards attainment of results at 32 radio stations across all seven countries, using a qualitative evaluation approach incorporating the OECD DAC criteria. For reporting, Khulisa has employed a unique evaluation approach – the Meta Story – to provide a birds-eye-view of the project’s performance.
Khulisa conducted a data quality assessment (DQA) for the Glaser Paediatric AIDS Foundation (EGPAF) on 25 health indicators in 66 sites. The sites spanned health facilities in four Mozambican provinces. The DQAs helped verify the accuracy of data reported by the EGPAF sites. Khulisa provided debriefings with EGPAF and its local partner, Ariel and also assessed the respective data management systems in Maputo, Gaza, Nampula and Cabo Delgado provinces for the quality of data recorded on source documents as well as the reports generated and submitted to third parties.
Between 2010 and 2017, as a subcontractor to MSH under the Grants Management Solutions (GMS) project, Khulisa carried out 33 assignments to provide technical support to Global Fund Recipients and Country Coordination Mechanisms (CCMs) in 16 countries. The technical support services provided by Khulisa involved strengthening and building the capacity of grantees and CCMs in grant M&E and oversight, programme management, and budget development.
In 2016, Khulisa conducted two scoping studies for GALVmed. The first scoping study led to a field study to collect data on household dynamics relating to livestock in Ethiopia, Tanzania and India. Khulisa used rapid group and household surveys to collect data from over 2800 male and female livestock-owning farmers. At minimal cost, the study helped our client better understand its beneficiary base. The second scoping study provided GALVmed with examples of businesses in the agri-input sector that are successfully selling their products to large numbers of smallholder farmers in Africa.
In 2017, Khulisa conducted a scoping study that produced a proposal to collect quantitative data on vaccinators/sales agents, smallholder farmers and retailers in Uganda. The scoping study led to a field study that collected data from 670 male and female livestock-owning farmers, 57 retailers, and 53 vaccinators/sales agents. Khulisa was commissioned in late 2017 to provide support to GALVmed’s monitoring and evaluation activities by conducting various ad hoc writing assignments for data sharing and dissemination of findings.
Khulisa carried out a national survey of public libraries in South Africa to inform the NLSA, the Department of Arts and Culture, Provincial and Municipal Library Services, as well as the Library and Information Services sector about the status of public access computing (PAC) and equipment, current capabilities, and future training needs in public libraries. The study is being used to prioritise investment funding, and ICT and training interventions, for public libraries in each province.
The USAID-funded South Africa Supplier Diversity Council (SASDC) had a pattern of under-reporting by its corporate members. As a result, USAID could not measure the impact of its support to SASDC. Khulisa conducted a two part independent evaluation of SASDC to verify results reported to USAID, to explore drivers of under-reporting by corporate members, and to offer data management and programme value recommendations. The assessment recommended how SASDC and its partners could improve data management and data accuracy. The evaluation revealed a more positive story around SASDC’s results and highlighted several success stories around corporate partnerships with small and medium suppliers from disadvantaged backgrounds. This left SASDC (i.e. post-USAID funding) in a stronger position, in terms of their programme focus and data system strength.
Khulisa was appointed by GTAC to provide strategic project management of a group of technical advisors to support the turnaround of the Safety and Security Sector Education and Training Authority (SASSETA) through the South African Government Technical Advisory Centre (GTAC). Reporting directly to both GTAC and the SASSETA Administrator, Khulisa managed and coordinated the activities of a team of technical advisors who were appointed by GTAC to develop and institutionalize organizational systems, processes and procedures and create a staff performance management system that would enable SASSETA achieve its goals and mandate. The project supported the SASSETA’s shift towards unqualified audits
The Partnership for Aflatoxin Control in Africa (PACA) is an African Union initiative to protect crops, livestock, and people from the effects of aflatoxins. Khulisa’s Theory of Change clarified the network’s implementation strategy.
Khulisa moved the South African Department of Science and Technology (DST) from a paper-based reporting system to a web-based reporting system. This initiative included work for all DST-supported science entities such as the South African Space Agency and the Southern African Large Telescope (SALT).
Khulisa conducted a performance evaluation for USAID/Malawi Early Grade Reading Activity (EGRA). The evaluation identified capacity gaps to improve the reading skills of Malawian students in Standards 1-3 and identified the necessary course corrections to support the EGRA projects in meeting its goals.
Khulisa used a participatory approach incorporating a Theory of Change model to evaluate USAID South Africa’s Tuberculosis (TB) Program. The analysed the programme’s performance at all project spheres, from South African government to community beneficiary level in nine provinces of South Africa with the aim of launching a follow-on project to support South Africa’s National TB Programme (NTP).
Khulisa conducted a mid-term performance evaluation for USAID’s Sexual HIV Prevention Program (SHIPP). The evaluation assisted the funder and its partners to improve the project’s future implementation.
Khulisa built an M&E framework and conducted an intense process evaluation of the Farmer Voice Radio (FVR) project using local researchers and data collectors. The project convinced farmers to change their agriculture practices to achieve greater food security and improved livelihoods. Khulisa also developed 46 training tools and 15 supplementary training materials to support the implementation of the project.
In 2014, Khulisa conducted a comprehensive baseline survey on South African health promoters across five districts in Gauteng for the South African Department of Health. The project recommended capacity building of current and incoming health promoters through orientation tools and knowledge promotion workshops. The 2014 project led to a second assignment in 2015, in which Khulisa developed materials and trained health promotion coordinators who adopted the Khulisa-designed training material across Gauteng
Khulisa Management Services conducted a diagnostic evaluation of the government-supported smallholder farmer sector using theory-driven methods. The South African Presidency’s Department of Planning, Monitoring and Evaluation (DPME) is using this evaluation to inform policy around smallholder farmers.