CBM CO Pakistan’s Programme
Evaluation for the Year
2021

Terms of Reference for CBM CO Pakistan’s Programme Evaluation for the Year 2021

Evaluation Summary

Project/ Project Numbers	CBM Country Programme in Pakistan
Partner Organisation	All Partner organisations in 2021
Project  start and end dates	All project active in 2021
Evaluation Purpose	Assessing country programme implementation in year 2021
Evaluation Type	Process Evaluation
Commissioning organisation/contact person	CBM Country Office Pakistan
Evaluation Team members (if known)	t.b.d.
Primary Methodology	Mixed Methods approaches including quantitative information derived from reports and from field Qualitative Methods through Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs) Field Visits ( randomly selected projects) Observations Literature review of projects documents including proposals, PPRs, Narrative Reports.
Proposed Evaluation Start and End Dates	TBD
Anticipated Evaluation Report Release Date	TBD
Principal Recipients of Final Evaluation Report	CBM (Country Office Pakistan, CBM MA’s, Asia Regional Hub) Project Partners;
Secondary Recipients of Final Evaluation Report	Ministry of Interior; Economic Affairs Division

1.          Background  

CBM is an international development organisation, committed to improving the quality of life of people with disabilities in the poorest communities of the world irrespective of race, gender or religious belief.

Based on its core values and over 100 years of professional expertise, CBM addresses poverty both as a cause and as a consequence of disability, and works in partnership with local and national civil society organisations to create an inclusive society for all.

CBM has been working in Pakistan for the last many decades through implementing partner organizations in different geographical locations of the country. The partner includes public sector organizations, hospitals and national non-governmental organization.

CBM Programme in Pakistan aims to improve living condition of persons with disabilities in far flung areas through provision of necessary service delivery in eye health, community based inclusive development, system strengthening and coordinating with Disable Peoples’ Organisations (DPOs) to include their inputs and expertise to best serve persons with disabilities.  

CBM Country Office has well thought out country strategy plan 2018-2022 which defines key programmatic priorities and objectives for the country office and act as a guiding document for the CBM in country programming.

Key focus area for CBM Programme in Pakistan are prevention of avoidable blindness, provision of inclusive eye care services at primary level, training of mid-level allied health personnel, innovations in eye health programmes, SAFE implementation, Community Based Inclusive Development (CBID) programme comprising of community empowerment, inclusive health, inclusive livelihood and Disability Inclusive Disaster Risk Reduction (DiDRR) and improving accessibility. In addition to regular programme and projects CBM provides ad-hoc support to DPOs initiatives at national and sub-national level.

2.          Evaluation Objective, Scope and Intended Use

Objectives:

• To assess quality and quantity of overall progress/achievements against agreed annual target for the year 2021
• To assess perception of final beneficiaries including persons with disabilities about the services and participation in project activities
• To document 3/3 6, success stories/case studies about different interventions and also capture high resolution pictures of the project/programme participants / activities  
• To document lessons learnt and recommendations for improving project/programme implementation in 2021

2.1.     Scope of the evaluation

This evaluation is focused on the year 2021 programme implementation which will determine how well the programme was implemented and how well it remains successful in achieving the annual targets. This process evaluation will assess perception of targeted population and activities contributing to the intended changes. The findings will guide decisions for further improving the quality of the programme and recommendations will be incorporated in 2021.

In addition to probing typical evaluation questions this study will also assess the projects/program implemented in a manner to ensure dignity, voice and choice of persons with disabilities in specific and other groups in general.

2.2 Suggest areas to be considered in Evaluation

Area of Enquiry	Suggested guiding questions
Relevance
To assess the programmes/projects currently in implementation meets the needs of the targeted population and are according to the local context	Do different projects’ objectives reflect the needs of the targeted population/groups/persons with disabilities? Does local context and realities factor in design and implementation of the projects? Does IEH projects interventions are relevant to the local needs of the communities / country. Does the implementing partner have capacity in terms of staffing, local/thematic knowledge and experience to implement projects? How much the projects are sensitive to the emerging needs of the targeted groups? To what extent project objectives, outputs and activities are aligned/comply with Country Strategic Plan 2018-2022 and contribute to the achievement of the key objectives of the plan?
Effectiveness
To assess the progress towards achieving annual targets agreed for 2021 Annual Action Plan	To what extent targets, objectives and out puts of annual action plan 2021 has been achieved? What are key contributing factors in achieving and non-achievement of targets agreed in annual action plan for 2021? How effective are the projects in reaching out to the deserving persons and to persons with disabilities? How effectively IEH projects interventions are beneficial to the people living around Are SOPs and standards applied in the work of each partner as per CBM standards?
Efficiency
How well resources  (financial, human and material) utilised efficiently and effectively while serving the target groups	To what extent project targets are delivered in a timely and economic manner? Are the objectives being achieved economically? Cost vs benefits)?, particularly to innovations in eye health programmes. What mechanisms have been put in place by the partner(s) for efficient utilization of resources?
Sustainability
	How well activities/ Projects related to the sustainability of the projects are implemented? What efforts partner are taking to ensure sustainability of the project within public systems (IEH) ? Are the positive effects or impacts sustainable? How is the sustainability or permanence of the intervention and its effects to be determined? Are IEH Projects aligned with IPEC 2020-2025 in terms of suitability plan
CROSSCUTTING ISSUES
Disability Inclusion
	To what extent the projects interventions benefits men, women, boys and girls with and without disabilities? Are appropriate measures taken to ensure persons with disabilities are included in the programme? How many persons with disabilities have been served by CBM Programme in 2021? Do the activities contribute in creating an enabling environment for persons with disabilities where their rights are respected and their voices are heard in fora formulating policies? Have the infrastructure made accessible in the planed IEH projects? Does Staff of particular projects have been provided trainings on WGQs and Disability inclusion in eye health programmes
Awareness Raising
	Do the projects contribute in sensitization of partner staff towards disability inclusion? Has awareness on disability inclusion increased among staff, and other relevant stakeholders?
Participation
	To what extent the project activities ensure involvement of persons with disabilities and their families in intervention? Are / Were men and women with disabilities involved in project implementation or considered as the main target group of the intervention?
Gender/Equity
	Do the partners promote gender equality during implementation of the projects? Does the project equally benefit different genders and were principles of equity followed where needed?
Coordination	How well partners are coordinating with other stakeholders specially Government counter parts and likeminded organisation including DPOs?

2.2.     Target audience and intended use

The findings and recommendations from this evaluation will be used to further improve projects/programme implementation in 2021. The information gathered will be shared with all relevant stakeholders specially partners to improve projects outcomes which can contribute to the ultimate change each project aims for.

The evaluation report shall inform CBM and its project partners about the success rate achieved during 2021. What could be improved in terms of the project design and implementation?

The report will also be submitted to Ministry of Interior and Economic Affairs Division in compliance with the requirement of annual third party evaluation given in the MoU.

3.          Methodology

The study design and methodology/tools will be developed by the selected consultant and be shared along with inception report. It is expected that consultant will use participatory methods to conduct this assignment, with taking care of including disadvantageous groups by providing appropriate means to ensure their meaningful participation. The consultant will use appropriate data collecting methods which includes a check list for FDGs and indicative questions for KIIs.

Field Visits would be made part of the evaluation.

The methodology must be inclusive for persons with disabilities to fully reflect their opinion and voices.

a.             Target population

For this assignment the targeted population is (5 IEH Partners implementing 08 Projects, Districts Chakwal, Matiari, Layyah, Rahimyarkhan, Kambar Shahdadkot and Peshawar, moreover LRBT Mandra in Districts Jhelum , Mayo Hospital Lahore, PICO Peshawar, MALC Karachi and SIOVS Hyderabad were provided with equipment)  District Jhelum )

 4 partners implementing 5 five projects on Inclusive eye health in Talagang, Multi Sector Community Based Inclusive Development Programme (DIDRR, Inclusive Health, Physical accessibility, Community empowerment, Inclusive livelihoods) in Muzaffargarh in eight Union Councils and Allied Health Human Resource Development in Karachi and Lahore. The consultant should come up with sample covering service receivers, key staff of partner for KII, and other stakeholders relevant to the project 

Note: The projects for which external end-project evaluation conducted in recent will not be part of this evaluation however findings from recent evaluation will be used for this assignment as secondary information

b.             Desk Review

The consultant will review all project documents (MYP documents, Project Annual Action Plans, Project Progress Reports, Narrative Reports, End of Year Reports etc.) including Country Strategic Plan, and other relevant CBM documents on inclusion and programme quality. Further the consultant should consult secondary data sources if required for triangulation or other purposes

c.              Qualitative data:

Qualitative data collection (FGD, KII) will validate quantitative data (data derived from reports/collected in field) through these methods beneficiaries’ perception regarding the services and project activities will be assessed in the selected geographical locations where projects are implemented

Sampling:

Sampling size and strategy will be identified by consultant, it should be representative and scientifically valid. The sample size should be reflective of all target groups, gender balance, having adequate representation of persons with disabilities, age consideration etc. The consultant will select open ended questions/check lists to be shared with CBM

Ethical Consideration:

The consultant should adhere to all ethical guidelines related to research, In addition to the generic research ethics CBM Safeguarding policy and consent form for photographs and case studies should be strictly adhered to.

Data Analysis/interpretations:

Consultant will present the findings both quantitative and qualitative findings and will brief CBM on lessons learned, recommendations during the final de-briefing session.

3.1.     Commissioning responsibility

The evaluation is commissioned by CBM, represented by its, Country Director or any other persons assigned by the CBM Country Office Pakistan.

3.2.     Evaluation Team

M&E specialists or any other professional having expertise in evaluation of projects involved in development and health especially eye health related services delivery. The team should ideally be gender balanced and include persons with disabilities.

Safeguarding Policy:

As a condition of entering into a consultancy agreement the evaluators must sign the CBM Child Safeguarding Policy and abide by the terms and conditions thereof.

3.3.     Management of the evaluation and logistics

The members of the evaluation team are responsible for all logistical arrangements related to the field research phase, including transport, accommodation and arrangement of meetings with local stakeholder. The evaluation team shall not be accompanied by staff of CBM or the project partners during the field visits. However, partners and CBM can support in identifying people to be interviewed and setting up meetings for the consultant.

CBM Country Office can support the evaluation team with making logistical arrangements and schedule meetings with project staff of partner organisation, beneficiaries and other local stakeholders.

The evaluation team is responsible to accommodate the needs of persons with disabilities during stakeholder meetings (e.g. workshop, focus group discussions). CBM Country Office can advise and support the evaluation team in making the necessary arrangements.

The members of the evaluation team are responsible to acquire the necessary insurance (including health, accident, and liability) and permits (including visa, NOC etc.). The CBM Country Office will support the evaluation team in acquiring the necessary travel permits.

6. Expected Results

An evaluation report in English, based on the standard evaluation report format/template of CBM, shall outline the findings of the evaluation and the recommendations for CBM and the project partners for future intervention or improvement in the current intervention.

The draft evaluation report shall be shared with CBM within two weeks after the completion of the field exercise. The final evaluation report shall be shared with CBM within one week after receiving feedback from CBM on the draft report.

The findings and recommendations of the evaluation shall be presented by a member of the evaluation team to CBM and a debriefing will be conducted focus on key findings and recommendations.

7.  Duration and Phasing

Tasks	Location	Number of Days	Expected Dates
Review of available project documentation	N/A	03
Preparation of evaluation methodology/tools and planning of field research	N/A	02
Inception meeting	Islamabad	01
Field research	Punjab/Sindh	05
Data analysis/compilation and writing of draft report	N/A	05
Writing of final report	N/A	01
Presentation of findings		0.5
Total number of days		17.5

8.          Costs and payments

CBM will provide HR cost as per mutual agreement. All payments to the consultant will be made on the basis of terms and conditions of the agreement. A mobilization cash advance can be provided by CBM (not more than 30% of the total agreed budget) at the request of the consultant.

9.          Attachments

The following documents should be known to the evaluation team in advance of the field research phase:

• Project documents including budget, Annual Action Plans, Narrative reports, Project Year end reports, CBM annual reports etc.
• CBM safeguarding policy
• CBM evaluation report format

Appendix: Stakeholder Analysis for Evaluation.

You could use a matrix like this to examine stakeholders involved in the project, and determine whether and how they could contribute to the evaluation, through interviews, surveys and meetings.

Please list all current and potential external and internal stakeholders including beneficiaries that contribute or influence the success of the proposed project(s) being evaluated.

Stakeholders	What is their interest and contribution in the proposed project?	What is their power and influence in the project (1-5 rating, 1=low, 5=high)	Will the project involve / these stakeholders in the evaluation? How?
Primary Stakeholders
Partners organizations.
Government Officials
National and Provincial coordinators ( Sindh, Punjab, KP).
e.g. Women and men with disabilities
e.g. Girls and boys  with disabilities
Secondary stakeholders
e.g. Mothers, fathers, care givers
e.g. Volunteers
e.g. Local Government
e.g. Board of partner organisation
e.g. Central Government
e.g. Community IEH - CBID
e.g. NGOs Peer INGOS
e.g. Project staff
e.g. CBM staff