Terms of Reference (ToR) for Final Evaluation
1. General Information
Post/Title:
Final Evaluation of the Education Sector Program Implementation Grant (ESPIG)
Program
GPE Funded ESPIG
Reporting to:
Program Manager- Education Sector Program Improvement Grant
Duration
40 working days
Closing date
23nd May 2022
2. Background Information
2.1. About Education Sector Program Implementation Grant (ESPIG)
The Global Partnership for Education (GPE)-funded Education Sector Program Implementation Grant (ESPIG) is aligned with the priorities of the Federal Government of Somalia’s Education Sector Strategic Plan (ESSP 2018-2020). The overall objective is to increase access to quality education for more than 50,000 out-of-school children; enhance the quality of primary education; and improve the capacity of the Ministry of Education, Culture and Higher Education (MOECHE) at Federal/Member States to regulate and manage the education sector. The Federal Government of Somalia Ministry of Education, Culture and Higher Education (MoECHE) is implementing the grant in collaboration with the Federal Member States Ministries of Education in Jubaland State, South West State, Galmudug State, Hirshabelle State and Benadir regional administration with CARE as the grant agent and Concern Worldwide as sub-contractor.
The main objective of the ESPIG is to improve equitable access to and quality of education outcomes for all Somali primary school children through strengthened system capacity to design evidence-driven Education Sector Strategic Plan (ESSP) reforms, and collaborate effectively with partners in their implementation. ESPIG was envisioned to maximize the potential of development investments in education in Somalia through the combination of four key principles:
- Use of evidence-based approaches to address key gaps in access, learning and retention, and inform tailored efforts to reach marginalized groups;
- Creation of conditions to implement solutions that address the intersection of multiple barriers to access, retention and learning;
- Generation of synergies with privately managed schools/networks, development actors and the private sector to ensure complementarity of efforts and leverage existing capacity/ investment;
- Capacity building of federal, state, regional and district-level actors for a cohesive, efficient and dynamic approach to the design and implementation of solutions.
The program’s five components are described below:
- Equitable access to quality education will increase through changes in three intermediate outcomes: Increased access to education for out of school children; strengthened capacity of community education committees; and community actions towards improved school safety.
- Enhanced quality of education, leading to grade-appropriate improved learning outcomes, will be achieved through the following intermediate outcomes: effective monitoring of teacher education and management policy, strengthened and harmonized efforts in teacher pre-service training; improved access to teaching and learning materials; and a strengthened assessment framework.
- Enhanced system capacity to regulate and manage the education sector will emerge from the combination of the following intermediate outcomes: (i) strengthening regulatory and monitoring system for private, community and government schools and (ii) improved capacity of education officers for planning, budgeting, policy implementation, coordination and progress tracking.
- Build a strong framework for program monitoring, accountability and communication, generating evidence to track progress, inform adaptive management and support planning and management processes.
- Effective and efficient program management, embedded within government systems at local level and leveraging the grant agent’s capacity and experience in country and globally
2.2. Program beneficiaries
Direct program beneficiaries includes: (i) about 50,000 out-of-school poor or marginalized primary children who will benefit from school inclusion grants; (ii) about 600 Community Education Committees who will benefit from training to improve inclusion, security and learning at their schools; (iii) 6,570 teachers who will benefit from teaching guides associated to the new curriculum; (iv) 240 primary pre-service student teachers who will benefit from new teacher training institutes; (v) 2,000 teachers benefitting from tailored in-service teacher training; (vi) 297,168 grade 1-8 pupils who will receive textbooks linked to the new curriculum; (vii) 150 education officers will benefit from professional development to improve skills in planning, coordination and monitoring.
3. Objective of the Final Evaluation
The Education Sector Program Implementation Grant (ESPIG) is seeking to procure the services of a consultancy company to conduct a final evaluation to assess the extent to which the stated objectives and ESPIG components were achieved (or not) in the project i.e. the extent to which the expected results of the ESPIG intervention have been achieved. The evaluation should generate relevant findings, lessons learned and recommendations to guide and inform on the best practices and strategies/approaches for improving other future phases and education programs. The final evaluation will have full participation of the project staff and key stakeholders.
3.1 Specific Objectives of the Final Evaluation
The consultant will critically evaluate the outputs and outcomes achieved under each component and how those have influenced the situation of target groups and beneficiaries. This will require critical examination of the individual activities to understand their contribution towards achievement of the results. The consultant is expected to reference changes in target groups, final beneficiaries and school conditions at the end of the project vis a vis the endline and baseline. The specific objective of the evaluation is to: “evaluate the outputs and outcomes achieved under each component and the project’s overall relevance, effectiveness, efficiency, validity of design, sustainability, factors affecting performance, alternative strategies and its strengths and weaknesses.
Overarching evaluation questions: The final evaluation will be guided by the overarching questions below;
Table 3 : Overarching questions
1
Effectiveness
- To what extent were the overall objectives and specific objectives achieved?
- What were the major factors influencing the achievement or non-achievement of the objectives?
- Were the assumptions underlying the overall program ToC valid?
2
Efficiency
- Which interventions were most effective in contributing to change relative to the personnel and associated costs?
- What have been the most and least efficient interventions?
3
Relevance
- To what extent are the objectives of the program ToCs still relevant for the targeted groups and beneficiaries?
- Which changes in the external context or lessons in its implementation, have influenced the relevance of the ToC and how?
4
Sustainability
- How sustainable is the ESPIG as a program and what results have been achieved? And how sustainable are these results?
- What is being done to ensure the sustainability of program results and how successful are those efforts?
5
Impact
- What has happened as a result of the programme?
- What the major positive changes are as identified in response to the effectiveness of the interventions?
- Did the program have any unintended negative outcomes?
- Did the program intended outcomes lead to unintended positive or negative changes?
6
Lessons and recommendations
- What are the main lessons learnt during the program design and implementation?
- Which adjustments to the ToC should be made in the next phase of the program?
- What are the main recommendations of this component and the program in general?
Specific evaluation questions: The assessment of the program components will be based on specific evaluation questions guided by the overarching questions above. The Specific evaluation questions under each component are listed below;
Component 1: Equitable access to quality education
Main activities:
- Provide capitation grants to target schools to increase access to education for out of school children
- Strengthen capacity of community education committees
- Community actions towards improved school safety[1]
Evaluation questions
- What was the impact of the intervention on children’s access to school? How did the impact vary per location and gender? To what extent did the intervention result in the enrolment of children from marginalized groups?
- To what extent did the provision of capitation grants mitigate the impact of COVID-19 crisis and prolonged drought on the most vulnerable children’s education?
- To what extent did the COVID-19 crisis and drought affect the ability of schools to ensure the attendance of students benefitting from capitation grants?
- What was the impact of the school grants component on school infrastructure?
- Are community education committees better prepared to manage schools? Have they developed skills and capacities to deliver needed services?
- What was the impact of the intervention on education outcomes in targeted schools, particularly in terms of enrolment, attendance and retention?
- To what extent did the federal member states and communities in underserved areas benefit from the intervention?
- Were capitation grants implemented in an efficient and transparent manner? To what extent are community education committees and local education officials involved in their oversight? What were the main uses of capitation grants?
- What activities are being implemented by community education committees? Has the intervention contributed to changes in CEC activities (types of activities, support to subgroups of students and/or how those activities are being implemented)?
Component 2: Enhanced quality of education, leading to grade-appropriate improved learning outcomes
Main activities
- Monitor teacher education and management policy
- Construct and equip one teacher training institute[2]
- Distribution of primary school textbooks
- Early grade assessments
- Tailored in-service teacher training[3]
Evaluation questions
- What was the impact of the provision of primary school textbooks on learning environments?
- Are learners benefiting from increased access to teaching/learning resources? How?
- Are teachers benefitting from increased access to teaching/ learning resources? How?
- To what extent was the implementation of this component affected by the combined negative impact of the COVID-19 crisis, prolonged drought and security issues in 2020-22?
Component 3: Enhanced system capacity to regulate and manage the education sector
Main activities
- Strengthen regulatory and monitoring system for private, community and government schools
- Strengthen capacities of education officers for planning, budgeting, policy implementation, coordination and progress tracking.
- EMIS system development and support
- Conduct a joint review of education sector
- Support ESA/ESSP processes
Evaluation Questions
- In what ways has the program enhanced fiduciary management and utilization of resources at the MOECHE and FMS MOEs?
- To what extent are the policies developed under the ESPIG, such as the private school policy and CEC training harmonization, being implemented?
- How have the various trainings, infrastructural support enabled MOECHE and respective state authorities to handle activities within their mandate effectively?
- To what extent is the new EMIS operational, compared to the previous version?
- How is the newly developed EMIS different from the previous versions? Does the system enable the MOECHE to better assess the status of the education system?
- To what extent have planning processes, such as the JRES, been supported by the ESPIG?
Component 4: Build a strong framework for program monitoring, accountability and communication
Main Activities
- Conduct evaluation studies – baseline, midterm study, and final evaluation
- Support the MoECHE and Federal Member States (FMS) to carry out monitoring activities to verify progress against ESPIG / ESSP indicators
- Learning and communication
- Feedback and Complaints Response Mechanism (FCRM)
Evaluation Questions
- How have the recommendations of the baseline/midterm survey informed adaptation and reprogramming efforts?
- Are the MoECHE and FMS Ministry of Education undertaking monitoring activities? Has support to monitoring activities enabled accountability and adaptive management?
- Are there platforms to share learning and/or communication? How is this working?
- How is the Feedback and Complaints Response Mechanism working? How can it be improved?
Cross-cutting Issues: Particular attention will be given to cross-cutting issues such as special needs and inclusion, gender, vulnerable and marginalized groups, human rights, child protection and safeguarding, and health and hygiene promotion in the project. A conflict-aware, do-no-harm approach has been used to implement activities.
4. Technical Approach and Methodology
The consultancy company is expected to propose a relevant methodology to assess the intervention’s contribution to the Somali education system, responding to the evaluation questions above. In addition, the evaluation will assess the program’s relevance, efficiency, sustainability, coordination, and lessons learned**. The selected methodology should be able to show the extent to which the program achieved its intended results thus far and reached the intended target populations. The proposed methodology should also allow the program to identify and quantify the impact of contextual factors on the expected results.
4.1 Document Review
The consultant will be expected to review all relevant documentation to understand the assignment and the context beforehand. This will enable the consultant to gather secondary data that will complement the primary data that will be collected. CARE will provide other relevant documentation on implementation once services are contracted.
4.2. Primary Data Collection
The primary data will be collected from various qualitative and quantitative data collection methods namely:
- School survey (longitudinally tracking the schools sampled at the ESPIG baseline and midline studies);
- Analysis of school enrolment records, retention and transition records, test scores and attendance records, and observation of students’ attendance through headcounts, where possible.
- Interviews with CECs, teachers, and headteachers;
- Key informant interviews with the regional and district officials, other MOE and MOECHE staff, and partner agencies;
- Case studies with beneficiaries, including but not limited to students (particularly girls) who have enrolled/ re-enrolled through capitation grants.
Other methods may be added as appropriate.
4.3. Data Collection, Processing, and Analysis
The consultant is expected to train enumerators on data collection methodologies and tools; quality assurance’; confidentiality; ethics and informed consent. CARE Somalia will provide training on child protection and prevention of sexual harassment, exploitation, and abuse (PSHEA) and require all enumerators to comply with its PSHEA and child protection policy. In addition, the consultant will be expected to supervise data collection in the field, translation, and transcription of qualitative data, ensuring quality is maintained throughout the entire process. The data collection tools will be prepared and shared with CARE before the training and thereafter piloted and revised accordingly after the pre-test exercise to ensure they are acceptable and match the program’s needs. The Consultant will then analyze the data collected, prepare the final evaluation report and present the findings, conclusions, and recommendations to CARE who will then share it with consortium partners, the donor, and other key stakeholders.
5. Deliverables
Deliverables under this consultancy will include the following:
- Inception report: Including the proposed methodology, data collection tools, analysis framework, and a detailed work plan.
- Data collection report
- Complete datasets and syntax files used for analysis.
- Complete transcriptions of qualitative data and respective audio files.
- Draft report: Will be presented to CARE for input and feedback. CARE will give feedback within seven days of receipt of the draft report.
- Final report inclusive of CARE’s feedback.
- Summary of findings and presentation.
6. Duration and Time
The exercise is expected to commence immediately after the contract is signed by all parties. The estimated deadline for completion is July 30, 2022.
#
Tasks
Duration (Man days)
Literature review/Desk Review
2
Preparation of inception report, data collection tools, translation into Somali, and testing/revision of tools
5
Travel to and from target locations
2
Enumerator training
3
Field data and information collection
15
Data and information analysis
6
Report writing and presentations
7
Total
40
7. Desirable Qualifications:
This assignment requires previous experience in evaluating large-scale system development programs and in particular, large-scale education programs, preferably in the region and/or in Somalia. In addition, the consultancy company should have experience in working in fragile, conflict-affected contexts. The consultant will be required to devise appropriate strategies to generate as much information as possible within the allocated period, considering the ongoing crisis and potential challenges to reach some locations.
Education and background required:
- Extensive experience in assessing large scale education programs, particularly system development/ strengthening initiatives;
- Demonstrated expertise in quantitative analysis and use of mixed methods;
- Excellent report writing skills in English;
- Adequate knowledge of the context, including the Somalia education system, policies, and guidelines.
- Previous experience in conducting community-based and school surveys in Somalia, provide contracts with previous NGOs and evidence of previous reports.
- Ability to roll out large-scale data collection and ensure data quality in the South Somalia context.
8. Core Values & Critical Key Competencies
- Strong analytical skills with a strong ability to do editing and proofreading
- Ability to think critically and strategically in difficult conditions
- Excellent interpersonal communication, relationship building, and networking skills
- Ability to work under pressure and meet tight deadlines in stressful conditions
9. Administrative/ Logistical Support
9.1 Budget
The consultancy company will submit to CARE a financial proposal covering consultancy fees, operational costs and reimbursables, in line with the methodology described in the technical proposal.
9.2 Schedule of payment
The following payments will be paid to the consultant based on an agreed upon mode of payment.
- After inception report: 30%
- After data collection is completed and datasets are received: 40% (totaling 70% of the payment)
- After Final Report: 30% (final payment)
The payments will only be made when the deliverables have been assessed by CARE to meet quality standards.
9.3 Logistics
CARE will provide the following support towards the successful execution of this consultancy:
- Provide all relevant internal documents needed during the desk review.
- Pay for all flight costs.
- Provide transport and accommodation in the field during the activity
- Provide stationeries and supplies required during the activity
- Provide linkages with key stakeholders and actors at all levels to facilitate key informant interviews.
- Pay consultant’s fees upon satisfactory completion of the assignment.
10. Conditions of Work
The ESPIG MEAL Lead will coordinate activities related to this consultancy in Somalia. CARE USA’s Senior Research and Learning Advisor will provide technical assistance to the evaluation process. The MEAL Lead and the Senior Research and Learning Advisor will review and approve the deliverables. The consultant will be required to abide by the organization’s applicable rules and regulations, code of conduct, and child safeguarding policy. CARE upholds the principle of working independently of political, commercial, military, or religious objectives. CARE has a zero-tolerance approach toward sexual harassment, exploitation and abuse, and child abuse.
11. How to apply
All interested consultants/firms are requested to write an expression of interest following the attached EOI format ONLY by email to: SOM.Consultant@care.org. Please indicate ‘Final Evaluation – Education Sector Program Implementation Grant’’ as the subject heading. Application deadline 23
May 2022. Any canvassing will lead to automatic disqualification.
[1] Noting the modifications made to this subcomponent as part of the revisions agreed with the GPE
[2] The original agreement with the GPE required the construction of two teacher training institutes and was later revised to include the construction of only one TTI. The revised agreement is currently in effect.
[3] Due to a protracted approval process, this subcomponent’s implementation started in April 2022 and will not be included in the final evaluation.
How to apply
All interested consultants/firms are requested to write an expression of interest following the attached EOI format ONLY by email to: SOM.Consultant@care.org. Please indicate ‘Final Evaluation – Education Sector Program Implementation Grant’’ as the subject heading. Application deadline 23 May 2022. Any canvassing will lead to automatic disqualification.