9. Monitoring and Evaluation
Monitoring and Evaluation have a range of purposes in humanitarin programming, but the critical one is this: better outcomes for crisis-affected populations from CARE’s humanitarian programming through accountability and learning.
Monitoring and evaluation help in understanding how the assistance and support that CARE provides to disaster-affected communities affects them. Monitoring and evaluation is therefore a critical part of CARE’s Humanitarian Accountability Framework (HAF). It allows us to compare the results of our humanitarian actions with our strategic intent (e.g. CARE Program Strategy, Humanitarian & Emergency Strategy, Core Humanitarian Standards) with technical standards (such as SPHERE and companions) and with expected outcomes and benchmarks for the response (from response strategy, proposal frameworks etc.). Efficient decision making and evidence based learning heavily depend on the quality and timeliness of monitoring & evaluation.
1.1 Roles and responsibilities of monitoring and evaluation
1.2 Role of the Monitoring and Evaluation Coordinator in emergency team
1.3 Definition of key terms relating to monitoring and evaluation
- Assess CO capacity for monitoring and accountability.
- Establish monitoring and evaluation systems from the very outset of the emergency response.
- Use CARE’s Humanitarian Accountability Framework to inform the development of monitoring and evaluation systems.
- Establish appropriate objectives and indicators at the individual project level as well as the overall emergency programme level during the design phase of the response.
- Ensure that the monitoring and accountability system in place is capable of delivering real-time information on what is happening in emergency response conditions.
- Plan for data collection and analysis. Double-check that the information to be gathered is going to give a realistic picture of what is actually happening.
- Plan for reporting, feedback and use of results.
- Ensure that the Monitoring and Evaluation Coordinator coordinates data collection and analysis for monitoring purposes across the programme.
- Employ a range of appropriate and participatory data collection methods.
- Confirm that all monitoring data collected is analysed and presented in a timely and user-friendly way.
- Ensure that appropriate managers review and act on monitoring data.
- Include resources for monitoring and evaluation activities in project budgets.
- Ensure monitoring includes feedback to communities.
- Design an appropriate monitoring and evaluation system for the emergency response.
- Ensure monitoring and evaluation systems consider all aspects of the response management.
- Establish appropriate objectives and indicators at the design phase.
3.1 Putting a monitoring and evaluation system in place
3.1.1 Steps in designing a monitoring and evaluation system
3.2 Aspects of the response to consider
3.2.1 What to look for when monitoring an emergency response programme
3.3 Objectives and indicators
3.3.1 Checklist for indicators
- Coordinate data collection and analysis responsibilities across the programme.
- Select a range of appropriate and participatory data collection methods.
- Conduct timely data analysis.
- Ensure timely management review of monitoring results and correct any issues arising.
4.1 Data collection and analysis responsibilities
4.2 Data collection methods
4.2.1 Participatory data collection methods
4.3 Data analysis
4.3.1 Quality of information
4.4 Management review of monitoring results
4.5 Proposal tracking and documentation
The term ‘accountability monitoring’ is used to mean the monitoring of our performance on accountability as described by the thrid pillar of the CARE Humanitarian Accountability Framework (HAF). In Chapter 32 of the CET you’ll find a more detailed description of CARE’s Quality and Accountability commitments for humanitarian programming.
Accountability monitoring can help CARE to:
- Check that the accountability systems that have been set up are working effectively.
- Focus our monitoring on approach, processes, relationships and behaviours, quality of work, satisfaction as well as outputs and activities.
- Priortise listening to the views of disaster affected people to assess our impact and identify improvements.
- Provide a feedback opportunity for staff, communities and other key stakeholders to comment on our response and how we are complying with our standards and benchmarks.
Accountability monitoring contributes to CARE’s overall monitoring and evaluation activities. Aspects can be integrated into other project monitoring tools, or carried out as a specific activity e.g. a beneficiary satisfaction survey or FGD (Focus Group Discussion) to solicit feedback and complaints from specific groups amongst the crisis affected population, with specific vulnerabilities or in isolated communities as part of a formal complaints mechanism. Ideally, accountability mechanisms and the monitoring of their effectiveness should be built into project proposals from the outset.
Accountability data (including complaints data) needs to be incorporated into monitoring reporting, alongside monitoring of project progress.
Rapid Accountability Review (RAR)
The Rapid Accountability Review (RAR) is the central tool for accountability monitoring in CARE’s humanitarian programmes.
What is a Rapid Accountability Review?
A RAR is a rapid performance assessment of emergency response against CARE’s HAF that takes place within the first few months of an emergency response. It generates findings and recommendations that are used to make immediate adjustments to the response. It is also a key source for any response review and performance management process. It usually entails interviews with CARE management, staff, communities and other key external stakeholders, and is led by an independent team leader.
What is the purpose of a RAR?
The overall goal of the RAR is to improve the quality of CARE’s response by assessing its compliance with established good accountability practice. More specifically, the RAR:
- Provides a real time assessment of HAF compliance early during a humanitarian response
- Ensures that the views of our key stakeholders are taken into account in making adjustments to our response and in drawing lessons learned.
- Identifies good practices, highlight gaps (including gaps in capacity) gaps and areas for improvement
- Makes recommendations to CARE management (CO, CI and CARE Members) for immediate action related to the ongoing response
When does it take place?
- Ideally, a RAR is conducted within 2 months of the start of an emergency event, and feeds into the general response review and performance management process
- A similar process can also be repeated at later stages of response in order to take stock of HAF compliance and improvements made, or to feed into a particular event such as a response evaluation, an emergency strategy review, or EPP event
How to conduct a RAR
Detailed information about how to conduct a RAR can be found in the Annex 9.6 RAR-Guidance.
In summary a Rapid Accountability Review should ideally include:
- A self-assessment by CARE staff and partners against relevant indicators (see Annex 9.7: staff engagement)
- Focus group discussions with affected populations (see annexes 9.8a, 9.8b, 9.9,)
- key informant interviews
- an synthesis and analysis meeting / workshop to review results of the above and prepare lessons and recommendations for the ongoing response and for an After Action Review (AAR – see section 8. Learning and Evaluation activities) below
Other examples for accountability monitoring can be found at Annex 9.9a Sample of accountability monitoring tools, including:
- Simple questionnaires.
- Focus group discussion tools.
- Staff review tools.
- Monitoring tool to help research into local communities’ views.
The RAR Summary (annex 9.6a) provides a tool that facilitates the synthesis of information collected during the accountability monitoring in an organised way against the 9 commitments of the HAF. It also identifies which key performance criteria are relevant at what stage of the response or for what level of accountability review (light, basic, comprehensive).
Depending on the methodology and format of the RAR there can be different reporting formats. Here are few examples:
- Annex 9.8c – RAR findings Ethiopia 2017
- Annex 9.8d – RAR report Haiti 2017 (french)
- Annex 9.10d – RAR/AAR report Papua New Guinea 2018
Most staff will have experiences of meeting people who are not fully happy with the work or behaviour of CARE or partners in their community or region. Most of this feedback or complaint is received informally e.g. people approach staff who are visiting the community, or visit CARE’s office in search of assistance or resolution to their problems or grievances. It is also not unusual for staff of one agency to receive a complaint about another agency. Receiving feedback, suggestions and complaints about our work is normal, important and should be welcomed.
Whilst there are occasions where complaints are handled well by field staff, there are many examples when they are not. At times, staff, already overwhelmed with day to day emergency activities, may find it difficult to manage the informal feedback and complaint they receive, might not prioritise complaints, or might forget or lose complaints. Tensions can also arise when a complaint is received about a member of staff and it is not clear how this complaint will be dealt with and by whom.
To improve this, CARE offices should put in place a more formalised system of soliciting, receiving, processing and responding to the feedback and complaints we receive. These systems should aim to provide a safe, non-threatening and easily accessible mechanism that enables even the most powerless to make a suggestion or complaint. On the part of CARE, this requires us to address and respond to all complaints, and to be timely and transparent in our decisions and actions.
Experience shows that complaints mechanisms can have enormous benefits for both communities and for CARE staff. It can help to establish a relationship of trust between staff and communities and improve the impact of our response. It can help save time and money that would otherwise be wasted. It can help build a safer organisation and safer environment for our staff, and for our beneficiaries, especially the most vulnerable amongst them. On the other hand, setting up a mechanism that does not function well (for example if complaints are not followed up) may contribute to frustration and worsening relationships with communities and local stakeholders.
Complaints procedures can be simple, although they need to be carefully planned and follow certain key principles. A badly designed or managed complaints procedure can be harmful. Here are 10 discussion points and suggestions for good practice to help establish a complaints mechanism that is:
- well understood
- promotes transparency
- accessible to all
Checklist to help establish a complaints mechanism
- Plan and budget for a complaints mechanism from the beginning of an emergency
- Build staff awareness and commitment to a complaints mechanism
- Provide a range of ways people can complain
- Make sure it can handle extreme cases of fraud and abuse
- Be clear about the scope of the complaints mechanism and communicate this clearly
- Develop a complaints mechanism procedure document and always follow the established procedure
- Clearly communicate the complaints mechanism to all key stakeholders as part of overall information sharing systems
- Complete the feedback loop: use the complaints data to improve overall performance and to provide feedback to communities (two way communication and feedback)
- Be clear on roles and responsibilities in managing complaints, and provide adequate training and support to staff
- Monitor the complaints mechanism to verify that it is effective
See Annex 9.18 Feedback, complaints and response mechanisms for further detail.
The IASC Task Team on AAP and REACH have developed a Menu of AAP related questions (May 2018) that should be included in needs assessments and/or reviews to support the design or adaptation of AAP systems especially with regards to communications strategies, inter-agency feedback mechanisms and participation strategies.
Feedback to communities on our monitoring and evaluation results (including complaints data and results from monitoring of our accountability) should be part of CARE’s overall information sharing to communities. An important part of this is to make reports reader friendly and share them as much as possible with all staff. Provide key complaints data in public places e.g. on websites and community noticeboards.
Providing such information to affected communities in an accurate and timely way is a fundamental ingredient of building trust. Trust is in turn a fundamental ingredient of participation. People will only engage meaningfully with individuals or institutions that they believe they can trust.
The following chart demonstrates how a flow of communication should work in this environment.
See Annex 9.19 – Information provision to affected communities for further detail.
- Organise an After Action Review.
- Conduct an evaluation when required.
CARE’s Policy on Evaluations is available at Annex 9.1. This policy highlights CARE’s commitment to learning from humanitarian response with a view to improve our practices and policies for future responses. All CARE COs are required to comply with this learning policy. Support and advice can be provided by CEG for learning activities.
8.1 Organising an After Action Review
8.2 Commissioning and managing an evaluation
Include monitoring and evaluation line items in project budgets.
9.1 Monitoring and evaluation costs
Lessons learned from previous CARE emergency operations have found that CARE COs often lack the capacity to design and implement monitoring and evaluation systems during emergency responses. In particular, COs have difficulty in adapting their regular monitoring and evaluation systems (used for longer-term programming) to more unpredictable emergency situations that are changing rapidly.
As both a relief and development agency, CARE has determined that programme and project standards should apply to all CARE programming, including emergencies, post-conflict rehabilitation and development, whether CARE is directly providing assistance, working with or through partners, or conducting advocacy campaigns.
CARE’s Quality & Accountability policies & standards should be used to inform the development of monitoring and evaluation systems in order to be consistent with the Core Humanitarian Standards (CHS) as well as the Code of Conduct for the International Red Cross and Red Crescent Movement and NGOs in Disaster Relief.
CARE’s Evaluation Policy describes CARE’s commitments to using evaluations to promote systematic reflective practice and organisational learning, and accountability to help contribute to significant and sustainable changes in the lives of people we serve.