Evaluation findings

• Author: Directorate-General for European Civil Protection and Humanitarian Aid Operations (ECHO) (European Commission), Landell Mills International
• Pages: 44-78

Final Evaluation Report: Part A- Afghanistan Country Programme Evaluation

44

3. Evaluation findings

3.1. Relevance

EQ1: To what extent did the design and implementation of EU-funded actions take into account the needs

of the most vulnerable population in Afghanistan, in particular women, children, elderly and disabled

people? To what extent were beneficiaries consulted in the design and implementation of DG ECHO-

funded projects?

3.1.1. Needs assessment

Tools and processes

Participation in and use of joint needs assessments coordinated by OCHA were common among

partners of ERM which is DG ECHO’s key programming instrument for delivery of emergency response

for the conflict-affected areas in Afghanistan and has been in existence since 2011.40 KII with partners in

all regions visited by the ET and documents (FichOps) research indicated that needs assessments often

involved multiple agencies, including the Directorate of Refugees and Repatriation (DoRR), OCHA and

World Food Programme (WFP), and sometimes could be very “crowded” with a large number of

40 DG ECHO (2017). ERM — Emergency Response Mechanism Common Rationale 7, Afghanistan—2017 (revision 2 October 2017)

Key findings:

•DG ECHO and partners use multiple methods and joint needs assessments to identify the

vulnerable and gaps; however, the evaluation found weak evidence of beneficiary

consultation, participation of women and incorporating age and disability factors in response.

•Data sharing on needs assessment is weak in general in the Afghanistan humanitarian

system and OCHA is now working out protocols in this regard.

•As part of accountability to affected population (AAP), the complaints mechanism was weak

for most of the evaluation period, though this is now being strengthened by DG ECHO

Final Evaluation Report: Part A- Afghanistan Country Programme Evaluation

45

participants. During the drought response in 2018, OCHA coordinated needs assessments in Herat and

Badghis. Beneficiary selection was based on the Household Emergency Assessment Tool (HEAT)41

which is now used by all agencies in joint assessments. Though the standard HEAT tool has 100

questions, this was compressed to 20 in order to suit the 2018 emergency context. The duration of

household assessment depended on the size and type of caseloads, varying from one day (2018 drought)

to several weeks. The HEAT assessment is usually followed by a rapid market analysis to ensure that

the cash assistance (by DG ECHO) provided will be usable in a functioning and accessible market.

The joint needs assessment does not follow a standard procedure and is designed by partners each

time it is undertaken, partly because the procedure requires negotiation with local authorities, coordinated

by OCHA. Key informants (KIs) described that the process normally involved 5 partners, sometimes with

one partner taking the lead in collecting data (with others as observers), sometimes one partner would

use the HEAT tool while others used their own questionnaire, and on other occasions partners divided

up the area and collected data independently which then were shared at the end. This leads to a situation

where households are asked multiple times the same or slightly different questions by different NGOs.

ERM partners have been very vocal in advocating for a change to the current procedure, recognising that

it not only costs time and efficiency but may impinge upon humanitarian space and dignity of beneficiaries.

There is supposed to be at least one female staff member in assessment teams to ensure that

information was collected from female-headed households, girls and women. That this rarely happens in

practice was observed in DG ECHO monitoring missions, but no actions were taken.42 It may be that it is

not always possible to have female staff on assessment teams, or for female members to access women,

without the explicit authorisation by the Shuras which are composed of men only. Interviews with female

beneficiaries (drought response 2018) in Herat and Mazar showed that there was no inquiry during needs’

assessment of specific needs of women as most often needs assessments were carried out with the head

of the family (primarily men) and elders. Though needs assessment processes may not have included

women, interviews with women beneficiary groups confirmed that non-food items provided by ERM

partners did take care of specific needs of women (sanitary napkins, underwear, separate toilets for

women, etc). Women also reported that water distribution points were located close to their homes, as

otherwise they had to travel some distances to fetch water.

In addition to HEAT assessments, DG ECHO partners utilised data from secondary sources such

as Humanitarian Needs Overview (HNO) and Humanitarian Response Plan (HRP),43 findings from post-

distribution monitoring (PDM) and information from other humanitarian actors, including reports and

analysis from OCHA and United Nations High Commission for Refugees (UNHCR) on IDPs and

returnees. Overall, the needs assessments were well structured, and most acute humanitarian needs of

the affected population were taken into consideration.44 The DG ECHO monitoring missions considered

the overall needs assessment to be of good quality, demonstrating sound justification for the proposed

action, as evidenced in their observations in a number of project follow up documents.

41 HEAT, originally developed by ERM partners, was adopted by HCT in 2016 for joint needs assessments

42 As noted in a DG ECHO project follow up document.

43 DACAAR (2016). HAA--Humanitarian Aid Actions Final Evaluation Report, Afghanistan. (Date of submission: 28/07/2016)

44 DG ECHO (2018). ERM — Emergency Response Mechanism Interim Report, Afghanistan—2018 (date of final comment 22/12/2018)

Final Evaluation Report: Part A- Afghanistan Country Programme Evaluation

46

Needs assessment during the drought of 2018 proved problematic. First, partners and DG ECHO

(and the Government and humanitarian community in general) failed to recognise the early warnings of

the impending drought that started to flow from about late 2017. Several FichOps noted the delay in

drought response; partner and DG ECHO staff interviews in Herat and Kabul confirmed that initially there

was reluctance to respond as ERM was geared more for responding to conflict and rapid onset disasters,

such as floods. One partner-NGO in Badghis

warned of displacement in February 2018,

but this was ignored. By March, there was an

influx of drought IDPs in Herat. Over a dozen

KIs stated that the Government was initially

opposed to any service delivery to the

drought IDPs due to the fear of pulling more

people into the cities in search of assistance.

DG ECHO awaited the OCHA response

strategy that was drafted at the end of June

and launched its response (blanket cash

distribution) in July-August, at the request of OCHA. DG ECHO states that part of the reason for the delay

lay in the fact that until it was fully established that the drought crisis had moved into a phase 4 level of

International Food Security Phase Classification (IPC),45 it was difficult to justify use of emergency funds

for any response.

KIs from most of the non-ERM partner agencies, external agencies (not funded by DG ECHO), as well

as a few ERM partners complained that partners do not share their needs assessment data with

other agencies or the cluster, something ERM partners recognise, but justify on the ground that cluster

leadership is weak across the board. Data sharing protocols between humanitarian agencies are currently

being discussed (led by OCHA). Many of the clusters are co-led by government agencies, which means

that sharing beneficiary-specific data with clusters, in a conflict situation, is problematic. As ERM partners

are usually the first on the scene, they may be asked to lead the interagency assessment, working with

OCHA and DoRR. ERM partners collect the data using KoBo toolbox46 and use this for analysis. However,

the assessment reports that ERM partners bring out only focuses on the ERM caseload, and does not

provide any data on the needs/gaps other agencies need to meet.47

Targeting

Screening of beneficiaries and ensuring that the most vulnerable are targeted is a recurring problem in

all humanitarian actions. In Herat and Qal-e-naw, as soon as needs assessments were conducted during

the 2018 drought, more families arrived and the numbers swelled, according to all KIs who were directly

involved in the response. It was often impossible to know whether new arrivals were from the drought-

affected areas or were arriving for other reasons such as conflict or were simply local people living in

conditions of great hardship who considered their own needs on a par with those of outsiders.

ERM partners have raised doubts from time to time about the effectiveness of the ERM criteria in targeting

the most vulnerable, as these criteria and method of selection are moderated by negotiations with local

authorities which may sometimes lead to high exclusion and inclusion errors, for example: (a) those too

poor to come to the capital find it hard to get vetted by the DoRR which is required to assess each

45 The Integrated Food Security Phase Classification (IPC) is a set of standardised tools that aims at providing a "common currency" for

classifying the severity and magnitude of food insecurity. The IPC-Acute scale categorises the severity of acute food insecurity into Five

Phases: Phase 1 – minimal; Phase 2 – stress; Phase 3 – crisis; Phase 4 – emergency; and Phase 5 – famine.

46 KoBo Toolbox is an open source software which has tools for field data collection, on and offline, for use in challenging environments.

This software has become popular with humanitarian organisations.

47 The ERM focus is on verification and beneficiary selection, using the HEAT) – the HEAT data is not systematically analysed or used

in order to identify other needs, but this doesn’t mean the data is not there.

In both Kandahar and Balkh, interviews with partners and

communities showed that though the 2018 drought affected

these regions as well, the humanitarian response was

concentrated mostly in Herat and Qala-e-Naw (Badghis).

Site visits and community discussions in IDP settlements in

these regions indicated that a very minimalist response was

launched by a few organisations (including those supported

by DG ECHO) in late summer of 2018, and as IDPs kept

moving into towns (Kandahar and Mazar) even as late as

December 2018, the humanitarian system failed to respond

adequately to the growing needs.