Evaluation of UNHCR’s Level 3 Response in Ukraine

EVO/2024/11

Following the escalation of the international armed conflict between Ukraine and the Russian Federation in February 2022, UNHCR declared a Level 3 (L3) emergency for Ukraine and initiated a large-scale response to meet the humanitarian and protection needs inside Ukraine. In line with UNHCR's Policy on Emergency Preparedness and Response, the UNHCR Evaluation Office (EVO) commissioned an evaluation of UNHCR’s L3 Response to the crisis in Ukraine.

The evaluation analyzed the extent to which UNHCR provided a relevant, timely and effective response to the crisis in Ukraine, taking into consideration the complex enabling and constraining factors since the escalation of the international armed conflict between Ukraine and the Russian Federation in February 2022.

The evaluation had three objectives:

• Inform the development of implementation plans for the 2025–2027 multi-year strategy
• Contribute to better plan for achievement of results for internally displaced people (IDP), returnees (refugee and IDP) and war-affected people – through an analysis of the engagement, partnerships and results of UNHCR’s activities as part of a broad, inter-agency effort
• Help UNHCR plan for and support under the leadership of OCHA, the transition towards medium and longer-term solutions for IDPs, and the potential return of large numbers of refugees from neighbouring countries of asylum, under the leadership of the Government of Ukraine.

The evaluation also documents the achievements, challenges, lessons learned, and future positions and adaptation required to further strengthen UNHCR’s programming, response and advocacy in Ukraine.  Given the response’s scale in budgetary terms, the evaluation additionally provides external accountability to partners who have funded UNHCR’s response inside Ukraine.

The evaluation adopted a non-experimental, mixed-method approach. The external independent evaluation team conducting the evaluation used a combination of primary and secondary, quantitative and qualitative data to derive their findings and conclusions including a literature review from a repository of over 1,200 documents, 147 key informant interviews (KIIs) with UNHCR staff, implementing partners, government representatives and cluster staff, 17 focus group discussions, and surveys with UNHCR staff and its partners.

The period under evaluation spanned from the start of the international armed conflict in February 2022 to March 2024, with a focus on informing UNHCR’s operation in 2024 in addition to its 2025–2027 strategy.

The evaluation was framed around four overarching evaluation questions that reference the Organization for Economic Cooperation and Development/ Development Assistance Committee (OECD-DAC) criteria:

• EQ1: Relevance of UNHCR’s implementation
• EQ2: Effectiveness of UNHCR’s approach to delivering protection and solutions
• EQ3: Coherence of UNHCR’s contribution to the government’s delivery of social protection and recovery
• EQ4: Coherence and effectiveness of UNHCR’s engagement in inter-agency coordination.

Effectiveness of UNHCR’s approach to delivering protection and solutions

Areas of strength

1. UNHCR has been effective in delivering a range of humanitarian assistance and protection to some of the most vulnerable people in Ukraine, at scale. Key factors that contributed to this success have included flexible funding, capable staff, strong partnerships and UNHCR's long experience working in Ukraine. This is particularly visible in the effective and timely scale-up of its multi-purpose cash assistance, delivered by UNHCR.
2. UNHCR’s law and policy work, legal aid and community-based protection have been strengths of UNHCR’s response. They have provided an important ‘bridge’ that has facilitated its engagement in recovery and solutions. UNHCR has played a pivotal role in piloting approaches and advocating for action in support of sustainable recovery solutions.
3. Partnerships have been an essential part of UNHCR's response in delivering a complex set of activities, often in highly insecure areas. UNHCR's partners have shown courage, competence and ability in delivering assistance and protection to IDPs, returnees, war-affected people and within these, other vulnerable groups. UNHCR has provided a supportive environment for these partnerships.

Areas requiring renewed focus/attention

1. While UNHCR strategy in Ukraine places greater emphasis on national non-governmental organizations and civil society partners, UNHCR’s multi-year strategic plan could set a clearer vision for how UNHCR will seek to strengthen these partnerships in a way that is consistent with its Grand Bargain and Global Refugee Forum commitments.
2. Early and sustained engagement to supporting Area-Based Approaches (ABA) has been an important part of UNHCR’s approach, but it has lacked coherence from the collective IASC system in Ukraine which has resulted in an approach which is yet to achieve significant scale. The implementation of UNHCR’s multi-year strategic plan offers an opportunity for it to review its engagement and determine its priorities which is particularly important given the context of funding reductions alongside continuing high levels of humanitarian need.

Relevance of UNHCR’s implementation

Areas of strength

1. UNHCR's response effectively adapted to the changing needs of the population inside Ukraine. The agency met many of its age, gender and diversity commitments and has sought to ensure that people have access to information and feedback channels.
2. UNHCR has performed well at shifting its focus and adapting its programmes to the changing context. This has been undertaken in response to both large-scale changes in the situation and more specific information from partners.
3. UNHCR's programmes have been designed to be sensitive to the conflict in Ukraine and its humanitarian assistance is consistent with its humanitarian principles. The agency has done well to balance the need to work with the government with a need to maintain operational independence.

Areas requiring renewed focus/attention

1. Territories under the Temporary Occupation of the Russian Federation continue to have the greatest unmet needs and UNHCR has supported collective efforts to advocate for humanitarian access to these areas. Despite the escalation of the situation to the Inter-Agency Standing Committee (IASC) Principles at the global level, efforts to gain principled access have ultimately failed and require ongoing efforts to ensure comprehensive humanitarian access.
2. There is scope for UNHCR’s to improve how it identifies and targets the most vulnerable people, as part of strengthening its intersectional analysis. There are also gaps in the coherence of UNHCR’s feedback mechanisms that compromise efforts to analyse, respond to and act on the feedback it receives or that is reported to its partners.
3. Refining needs analysis will continue to be important as UNHCR seeks to find long-term solutions for people affected by the war. Cash assistance has been a valuable resource for people affected by the crisis, and UNHCR has played a lead role in its provision. However, the evaluation found that there is still scope for it to ensure that cash assistance is routinely provided as opposed to goods in-kind.

Coherence of UNHCR’s contribution to the Government's delivery of social protection and recovery

Areas of strength

1. UNHCR has adopted a collaborative approach in Ukraine that has supported and engaged the government at all administrative levels despite the challenges posed by the war.  The organization has been particularly effective in its engagement with the Ministry of Social Planning and its long-standing support to PeReHid.
2. UNHCR’s engagement across government at both a strategic and operational level has strengthened the coherence of UNHCR’s response in addition to permitting it opportunities to advocate for policy changes on behalf of those receiving assistance and protection.

Areas requiring renewed focus/attention

While UNHCR's approach of working collaboratively with the Government in Ukraine has yielded positive outcomes, there is scope for improvement in how the collective IASC system coordinates its support to the government on issues of internal displacement and solutions.

Coherence and effectiveness of UNHCR’s engagement in inter-agency coordination

1. UNHCR has worked hard to navigate the challenges of a rapidly changing context and to scale-up its coordination leadership capacity since the international armed conflict. This has permitted its clusters (Protection, Shelter and Non-Food Items (SNFIs), and Camp Coordination and Camp Management) to achieve many of their core functions in addition to making efforts to leverage the benefits of tri-cluster coherence, which has been a strength of the response. All three clusters have improved their engagement with local actors over time.

Areas requiring renewed focus/attention

There continues to be gaps in planning for the future collective coordination system in Ukraine. It should be considered a priority for the government-led and locally rooted coordination mechanism to move from principle to practice.

Good practices

• Multi-functional teams encompass diverse expertise, from different sectoral or functional areas
  (e.g. community based protection, CBI, legal protection etc.).The Ukraine operations highlights two effective ways of deploying MFT to enhance delivery and cross sectoral collaboration (1) Instigating a Multifunctional team approach for field visits in support of protection mainstreaming. MFT visits and relationship-building at the operational level (e.g. for collective sites and area-based programmes) currently act as a cornerstone for protection mainstreaming. These have clearly been a key element of UNHCR’s integrated approach, encouraging synergy, adaptation and dialogue between shelter, CCCM and protection teams at a tactical operational level that is critical to protection mainstreaming and supporting better results. (2) MFTs deployed in Programme Monitoring: Monitoring visits are conducted biannually by teams, aligning with the cross-sectoral nature of UNHCR’s programming, particularly for protection. These teams, comprising up to 20 people, either visit together or split based on technical specializations. Joint monitoring visits and meetings with authorities are always multifunctional. For example, in social housing projects, the shelter team leads, but protection staff provide significant input to ensure inclusivity and support any relevant referrals. Multifunctional knowledge is critical for mainstreaming protection in shelter and CCCM, effectively reducing risks across sectors
• Support to IDP Councils as a form of democratic expression and accountability
  IDP Councils in Ukraine are consultative bodies, composed of local authorities, IDPs and civil society representatives. They exist at oblast and local levels to serve as a link between IDPs and local authorities to ensure that IDPs can actively participate in public decision-making. They are tasked with developing recommendations to ensure that IDPs are included in local programming and policy decisions affecting them – thereby facilitating IDPs’ full-inclusion and participation in their community. UNHCR supports approximately 100 out of the 1000 IDP Councils in Ukraine. Given localization trends and EU accession discussions, IDP Councils have the potential to emerge as a powerful voice and agent for IDPs. However, as noted above, many are currently frustrated by their lack of capacity in terms of both resources and meaningful access to people in power and voice. They have yet to find their place in the ecosystem. UNHCR’s early decision to support IDP Councils has played an important role in strengthening their capacity in this respect and this is a model that could be relevant to other IDP operation.
• Incorporation of Principled Programming and AAP in UNHCR Ukraine’s Risk Management
  The Risk Register explicitly outlines a series of actions (treatments) that need to be put in place to mitigate the existing level of risk around AAP and community ownership. While the actions had not yet been taken by the time of the evaluation, inclusion of a risk requiring formal Senior Management attention, underscores the significance of AAP/ Community Engagement and Accountability (CEA) for UNHCR in facilitating relevant and effective protection and assistance programmes in Ukraine. In specifically recognizing community engagement and AAP as a key potential risk and outlining treatment actions, AAP and community engagement is routinely considered by management as part of delivery of protection and assistance. Additionally, UNHCR's risk matrix included the risk of politicization, requiring the risk manager to maintain a continuous assessment of UNHCR’s positioning vis-à-vis the Government. The impartiality of UNHCR’s assistance and perceptions thereof are also monitored.
• Standardized general protection case management information system
  UNHCR has initiated a joint project with the International Rescue Committee to develop a Primero-based Protection Case Management Information Management System that will align with the child protection (CPMIS+) and GBV (GBVIMS+) case management systems. This will be tested over 2024/5. At present, general case management information is agency specific, paper-based, subject to data protection breaches and incompatible with area of responsibility (AoR) case management systems. The system and tools should benefit not just the Ukrainian response but other humanitarian responses in the future.
• Addressing Staffing Challenges during the L3 Scale-Up: "The Accompanied Journey":
  The rapid increase in staffing during the L3 scale up brought a number of challenges, including the need to rapidly onboard a large number of new staff into the organization. Several efforts were made to streamline and strengthen the process, including through the "Accompanied Journey Approach". Despite some criticism of the quality of emergency response training and learning programmes, the approach was found to have provided an integrated and coordinated learning and development package for staff induction. This was led by UNHCR’s Global Learning and Development Centre and addressed a range of learning needs from basic inductions to more advanced team and managerial support. It drew on a range of services that included coaching support, webinars, playlists, English language skills and mindfulness sessions.
• The approach taken by the SNFI cluster to promote longer-term shelter solutions
  The SNFI produced a paper on Adequate Housing which aims to identify opportunities that can be used to promote a longer-term approach to addressing the shelter needs of affected populations. It examines the boundaries of the impact of Shelter cluster activities and how stakeholders involved in sustainable solutions and development initiatives can commence and complement their efforts. In the absence of an operational transitional coordination structure in Ukraine, it moves away from the concept of clusters as being “humanitarian” and seeks to pave the way for a longer-term approach to coordination in the shelter sector. The Evaluation team see the approach taken by the SNFI cluster in Ukraine and the approach outlined in this paper as potentially of value in other contexts where formal transition frameworks are not yet established.
• Monitoring by the CCCM cluster against Resolution #930 compliance
  Resolution #930 on collective sites is a comprehensive policy addressing crucial aspects related to the well-being and living conditions of IDPs. Collective sites in Ukraine are established primarily by the authorities in schools, dormitories, sanatoriums, and other public buildings across the country and provide temporary accommodation to those forced to flee their homes who have no other shelter options or means to rent an apartment or house, especially as costs of living have increased due to the war. The Collective Site Monitoring (CSM) is an initiative of the CCCM cluster in Ukraine implemented by REACH and supported by Cluster partner organizations and the Government of Ukraine Ombudsman Office. It aims to provide a range of stakeholders, including humanitarian agencies and Ukrainian authorities, with essential information regarding the situation in collective sites hosting IDPs. The tenth round of monitoring was modified to add or improve indicators to better assess compliance with minimum standards set out in Resolution #930. The inclusion of an annex in the collective monitoring site reporting that focuses on Resolution 930 compliance provides a yardstick for measuring adherence to standards, while enhancing accountability to government at a time when the transition of the clusters to government leadership has become ever more important.

Lessons learned

• Shifting UNHCR’s involvement in livelihoods and economic inclusion to strategic rather than operational level
  Until recently, UNHCR and partners provided support for small-scale businesses and livelihoods initiatives across Ukraine. The programme proved resource-intensive and a stretch for UNHCR’s skill set and funding. It also took place in a context where others, such as GIZ, IOM and UNDP, were already working at scale. As a result, UNHCR has been rethinking its approach more as a facilitator and convenor with a pilot on career counselling in Poltava, support to job fairs in the centre/west and job/skills matching for people in collective sites. A livelihoods and economic inclusion expert has recently joined the Durable Solutions team and should assist UNHCR Ukraine to further refine its approach on the basis of its comparative added value. This approach aligns with recommendations made in a 2018 Evaluation of UNHCR’s Livelihood Strategies and Approaches and further reiterated in the 2023 Evaluation of UNHCR’s Engagement in Situations of Internal Displacement.
• Need for early establishment of output monitoring
  It is essential that systems are established, at the earliest opportunity in a response, to verify the accuracy of data input into “Activity Info” on the number of beneficiaries reached, particularly from partners, for example by triangulating these data against actual distribution data. Since they are used to report against set targets and indicators, inaccuracies are misleading and can have negative impacts on the management of decision-making.
• Tri-cluster coordination
  In the earliest stages of the L3 response, UNHCR endeavoured to facilitate tri-cluster synergies through the deployment of an Inter-agency Coordinator to promote this. Interviews revealed that in practice, at that time, the clusters were still seeking to establish themselves, so it was ambitious to expect them to have the capacity to focus on synergies. However, feedback from interviewees within the cluster was positive, and there is a perception that it added value in contributing to the development of some joint strategies and advocacy papers. The inter-agency post was curtailed in 2023. The use of an inter-agency coordinator to promote tri-cluster synergies offers scope for lesson-learning for UNHCR, although it is not without its challenges. The key issue is to determine how, and under what conditions, such a post can achieve the greatest added value. There was a feeling in Ukraine that an opportunity was missed to work more broadly on other areas of inter-agency coordination (clusters, working groups). While the concept appeared to be sound, the timing – early in the response – was imperfect; there are also questions about whether a senior management post might be best placed to promote tri-cluster synergies, as it has the benefit of seniority.

Strategy

1. As part of the implementation of its 2025–2027 Multi Year Strategic Plan, UNHCR Ukraine should more clearly define the scope of its support to:

• Area-based approaches: leverage its catalytic role to promote a sustainable approach to allow UNHCR to reduce its level of engagement.
• Collective sites: outline a set of conditions for UNHCR to responsibly phase out of its operational support.
• Community-based approaches: develop an over-arching community-based protection strategy.
• Community-based organization (CBO): develop benchmarks to permit UNHCR to monitor and support CBO graduation objectives.

Operations

2. Internally and across the clusters under its leadership, UNHCR should ensure that the use of Cash Based Interventions is the default modality and that the use of in-kind is clearly justified (e.g., demonstrated lack of market functionality or lack of financial service providers).

3. UNHCR should develop a coherent Accountability to Affected Populations strategy that integrates a “whole-of-house” approach as part of its wider community-based protection strategy.

Government and partnerships

4. UNHCR Ukraine should advocate within the HCT and UNCT for a more coherent approach to Government System-strengthening and Government ownership of the response across the collective IASC system. If called upon to do so, it should be prepared to actively contribute to a process of review and mapping to determine the best means of coordinating collective system-strengthening efforts in the future.

5. UNHCR should further advance localization by investing in the capacity of national NGO partners and Community-based Organizations. The 2025-2027 Multi-Year Strategic Plan offers an opportunity for UNHCR to set out a more predictable basis to support its partners.

Collective coordination and clusters

6. UNHCR should support the transition of the clusters to government-led coordination through the following actions:

• In the context of the HCT-mandated Area-Based Coordination consultancy, advocate for a transition of the clusters to a coordination model that places government in leadership roles.
• For the clusters UNHCR leads, UNHCR should identify potential government principals who can assume a leadership role.
• UNHCR should actively contribute within the UNCT to the development of a coordination model for the UN’s work in the areas of recovery, development and durable solutions that has a foundation in government leadership and enables a seamless transitioning of the clusters and sustainable, nexus programming.

Global Analysis and Guidance

7. UNHCR globally should review its application of area-based approaches and amend guidance to ensure that area-based approaches incorporate collective objectives and outcomes for both programming and advocacy. This should clarify UNHCR’s contribution as part of a wider collaborative multi-stakeholder effort in which UNHCR has a distinct competence consistent with its mandate.

8. Drawing on lessons from previous L3 responses, including in Ukraine, UNHCR globally should consider developing a “post L3” roadmap to support Country Operations in the timely planning for scale-down in a way that protects and sustains dividends from the response. It should include practical guidance on how and when to start prioritizing and planning for responsible drawdown both geographically and sectorally. The guidance should also include aspects of human resource restructuring and partner management in the context of resource reductions.

The management response is due on 3 April 2025 and will be added to this page when available.