Evaluation findings

AuthorDirectorate-General for European Civil Protection and Humanitarian Aid Operations (ECHO) (European Commission), Landell Mills International
Final Evaluation Report: Part A- Afghanistan Country Programme Evaluation
3. Evaluation findings
3.1. Relevance
EQ1: To what extent did the design and implementation of EU-funded actions take into account the needs
of the most vulnerable population in Afghanistan, in particular women, children, elderly and disabled
people? To what extent were beneficiaries consulted in the design and implementation of DG ECHO-
funded projects?
3.1.1. Needs assessment
Tools and processes
Participation in and use of joint needs assessments coordinated by OCHA were common among
partners of ERM which is DG ECHO’s key programming instrument for delivery of emergency response
for the conflict-affected areas in Afghanistan and has been in existence since 2011.40 KII with partners in
all regions visited by the ET and documents (FichOps) research indicated that needs assessments often
involved multiple agencies, including the Directorate of Refugees and Repatriation (DoRR), OCHA and
World Food Programme (WFP), and sometimes could be very “crowded” with a large number of
40 DG ECHO (2017). ERM Emergency Response Mechanism Common Rationale 7, Afghanistan2017 (revision 2 October 2017)
Key findings:
DG ECHO and partners use multiple methods and joint needs assessments to identify the
vulnerable and gaps; however, the evaluation found weak evidence of beneficiary
consultation, participation of women and incorporating age and disability factors in response.
Data sharing on needs assessment is weak in general in the Afghanistan humanitarian
system and OCHA is now working out protocols in this regard.
As part of accountability to affected population (AAP), the complaints mechanism was weak
for most of the evaluation period, though this is now being strengthened by DG ECHO
Final Evaluation Report: Part A- Afghanistan Country Programme Evaluation
participants. During the drought response in 2018, OCHA coordinated needs assessments in Herat and
Badghis. Beneficiary selection was based on the Household Emergency Assessment Tool (HEAT)41
which is now used by all agencies in joint assessments. Though the standard HEAT tool has 100
questions, this was compressed to 20 in order to suit the 2018 emergency context. The duration of
household assessment depended on the size and type of caseloads, varying from one day (2018 drought)
to several weeks. The HEAT assessment is usually followed by a rapid market analysis to ensure that
the cash assistance (by DG ECHO) provided will be usable in a functioning and accessible market.
The joint needs assessment does not follow a standard procedure and is designed by partners each
time it is undertaken, partly because the procedure requires negotiation with local authorities, coordinated
by OCHA. Key informants (KIs) described that the process normally involved 5 partners, sometimes with
one partner taking the lead in collecting data (with others as observers), sometimes one partner would
use the HEAT tool while others used their own questionnaire, and on other occasions partners divided
up the area and collected data independently which then were shared at the end. This leads to a situation
where households are asked multiple times the same or slightly different questions by different NGOs.
ERM partners have been very vocal in advocating for a change to the current procedure, recognising that
it not only costs time and efficiency but may impinge upon humanitarian space and dignity of beneficiaries.
There is supposed to be at least one female staff member in assessment teams to ensure that
information was collected from female-headed households, girls and women. That this rarely happens in
practice was observed in DG ECHO monitoring missions, but no actions were taken.42 It may be that it is
not always possible to have female staff on assessment teams, or for female members to access women,
without the explicit authorisation by the Shuras which are composed of men only. Interviews with female
beneficiaries (drought response 2018) in Herat and Mazar showed that there was no inquiry during needs’
assessment of specific needs of women as most often needs assessments were carried out with the head
of the family (primarily men) and elders. Though needs assessment processes may not have included
women, interviews with women beneficiary groups confirmed that non-food items provided by ERM
partners did take care of specific needs of women (sanitary napkins, underwear, separate toilets for
women, etc). Women also reported that water distribution points were located close to their homes, as
otherwise they had to travel some distances to fetch water.
In addition to HEAT assessments, DG ECHO partners utilised data from secondary sources such
as Humanitarian Needs Overview (HNO) and Humanitarian Response Plan (HRP),43 findings from post-
distribution monitoring (PDM) and information from other humanitarian actors, including reports and
analysis from OCHA and United Nations High Commission for Refugees (UNHCR) on IDPs and
returnees. Overall, the needs assessments were well structured, and most acute humanitarian needs of
the affected population were taken into consideration.44 The DG ECHO monitoring missions considered
the overall needs assessment to be of good quality, demonstrating sound justification for the proposed
action, as evidenced in their observations in a number of project follow up documents.
41 HEAT, originally developed by ERM partners, was adopted by HCT in 2016 for joint needs assessments
42 As noted in a DG ECHO project follow up document.
43 DACAAR (2016). HAA--Humanitarian Aid Actions Final Evaluation Report, Afghanistan. (Date of submission: 28/07/2016)
44 DG ECHO (2018). ERM Emergency Response Mechanism Interim Report, Afghanistan2018 (date of final comment 22/12/2018)
Final Evaluation Report: Part A- Afghanistan Country Programme Evaluation
Needs assessment during the drought of 2018 proved problematic. First, partners and DG ECHO
(and the Government and humanitarian community in general) failed to recognise the early warnings of
the impending drought that started to flow from about late 2017. Several FichOps noted the delay in
drought response; partner and DG ECHO staff interviews in Herat and Kabul confirmed that initially there
was reluctance to respond as ERM was geared more for responding to conflict and rapid onset disasters,
such as floods. One partner-NGO in Badghis
warned of displacement in February 2018,
but this was ignored. By March, there was an
influx of drought IDPs in Herat. Over a dozen
KIs stated that the Government was initially
opposed to any service delivery to the
drought IDPs due to the fear of pulling more
people into the cities in search of assistance.
DG ECHO awaited the OCHA response
strategy that was drafted at the end of June
and launched its response (blanket cash
distribution) in July-August, at the request of OCHA. DG ECHO states that part of the reason for the delay
lay in the fact that until it was fully established that the drought crisis had moved into a phase 4 level of
International Food Security Phase Classification (IPC),45 it was difficult to justify use of emergency funds
for any response.
KIs from most of the non-ERM partner agencies, external agencies (not funded by DG ECHO), as well
as a few ERM partners complained that partners do not share their needs assessment data with
other agencies or the cluster, something ERM partners recognise, but justify on the ground that cluster
leadership is weak across the board. Data sharing protocols between humanitarian agencies are currently
being discussed (led by OCHA). Many of the clusters are co-led by government agencies, which means
that sharing beneficiary-specific data with clusters, in a conflict situation, is problematic. As ERM partners
are usually the first on the scene, they may be asked to lead the interagency assessment, working with
OCHA and DoRR. ERM partners collect the data using KoBo toolbox46 and use this for analysis. However,
the assessment reports that ERM partners bring out only focuses on the ERM caseload, and does not
provide any data on the needs/gaps other agencies need to meet.47
Screening of beneficiaries and ensuring that the most vulnerable are targeted is a recurring problem in
all humanitarian actions. In Herat and Qal-e-naw, as soon as needs assessments were conducted during
the 2018 drought, more families arrived and the numbers swelled, according to all KIs who were directly
involved in the response. It was often impossible to know whether new arrivals were from the drought-
affected areas or were arriving for other reasons such as conflict or were simply local people living in
conditions of great hardship who considered their own needs on a par with those of outsiders.
ERM partners have raised doubts from time to time about the effectiveness of the ERM criteria in targeting
the most vulnerable, as these criteria and method of selection are moderated by negotiations with local
authorities which may sometimes lead to high exclusion and inclusion errors, for example: (a) those too
poor to come to the capital find it hard to get vetted by the DoRR which is required to assess each
45 The Integrated Food Security Phase Classification (IPC) is a set of standardised tools that aims at providing a "common currency" for
classifying the severity and magnitude of food insecurity. The IPC-Acute scale categorises the severity of acute food insecurity into Five
Phases: Phase 1 minimal; Phase 2 stress; Phase 3 crisis; Phase 4 emergency; and Phase 5 famine.
46 KoBo Toolbox is an open source software which has tools for field data collection, on and offline, for use in challenging environments.
This software has become popular with humanitarian organisations.
47 The ERM focus is on verification and beneficiary selection, using the HEAT) the HEAT data is not systematically analysed or used
in order to identify other needs, but this doesn’t mean the data is not there.
In both Kandahar and Balkh, interviews with partners and
communities showed that though the 2018 drought affected
these regions as well, the humanitarian response was
concentrated mostly in Herat and Qala-e-Naw (Badghis).
Site visits and community discussions in IDP settlements in
these regions indicated that a very minimalist response was
launched by a few organisations (including those supported
by DG ECHO) in late summer of 2018, and as IDPs kept
moving into towns (Kandahar and Mazar) even as late as
December 2018, the humanitarian system failed to respond
adequately to the growing needs.

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT