Spoiler alert: There isn’t one.
Closing the feedback loop is notoriously challenging. To correct and adapt our programming, we need to understand what doesn’t work, who it doesn’t work for, and what improvements or alternatives are needed. Based on this information, we need to take action when possible, and demonstrate this at a collective or individual level. Even if we do understand the challenges being raised, we might not be in a position to take action or affect change. Refugees face a range of complex issues, and while we can demonstrate we’ve listened, we cannot always provide a resolution. The lack of a durable solution for millions of those who have been forcibly displaced, ultimately means we have an inherently dissatisfied constituent.
So how do we do it?
There is no panacea to this complex issue. Quick-fix solutions are often over-simplistic and fail to address the key challenges operations face. Protection is not a single transaction; it isn’t discretely linked to one product or service, nor to one producer or provider. So feedback solutions need to reflect this complexity, and there is no one-size fits all or silver bullet solution. However, we do know there are some essential features that support the closing of the feedback loop.
We need to:
- Listen to everybody who needs individual support, and provide a timely ‘answer’
- Understand and listen to diverse communities, and their idiosyncratic needs
- Collect up-to-date, actionable information that highlights challenges and capacities
From measuring perception to understanding satisfaction
There has been an emerging trend1 in the application of third-party perception surveys in humanitarian response, in a bid to improve the accountability of programming and to close the feedback loop. These perception surveys aim to provide a snapshot of a community’s satisfaction, against a set of defined indicators linked to the response. Adopting techniques which draw on customer satisfaction experience to quickly generate and neatly visualise data, it is clear why these
packaged, third-party surveys appeal to many agencies. However before an operation opts for a third-party satisfaction survey as an accountability tool, the considerations above should be taken into account.
Listen to everybody who needs individual support, and provide a timely response
Anonymous perception surveys do not allow for individual follow-up and resolution. Respondents will receive collective feedback at best – but this is not guaranteed. If a third-party enumerator is asked questions by the community during their data collection or receives urgent feedback and complaints they have no responsibility or accountability to take action. They cannot immediately provide information, make a referral or adapt an intervention – in short, they cannot be accountable for service providers. Therefore, there is a risk that perception surveys are inherently extractive, and may only exacerbate the echo-chamber felt by many community members.
As UNHCR, we have processes and tools that recognise and help us track unique records – these need to be optimized to ensure we are able to close the loop. Our primary focus and prioritisation of resources should be on accountability solutions which enable personalization and for us to provide individualized follow-up if needed.
Understand and listen to diverse communities, and their idiosyncratic needs
Many perception surveys use cluster sampling. While this may be statistically representative, there are some inherent challenges with this survey design. Perhaps the most significant for UNHCR is that cluster sampling may not represent the diversity of a population. From different types of probability sampling, cluster sampling is considered the least representative of the population – especially when the population characteristics are unknown.
As a protection agency, we should identify where and how needs and capacities most significantly diverge. Purposeful sampling, as often adopted through the participatory assessment approach, may, therefore, be preferable. Although championed for providing feedback at scale, perception methodologies are often dwarfed by our own existing community engagement practices. As an example, the one-time survey in Bidibidi (Uganda) sampled 300 respondents from the refugee community, with an estimated total of 270,000. In the same settlement, UNHCR has regular contact (at least monthly on a formal basis) with nearly 570 elected Refugee Welfare Council representatives. Members of these councils were selected by the community to represent the diversity of each village. The challenge is how we capitalise on these existing mechanisms to strengthen our feedback and accountability. What can we leverage from our existing tools and processes to ensure that data is collected and analysed to better inform programme design and adaptation?
Collect up-to-date, actionable information that highlights challenges and capacities
Perception surveys provide a topline ‘snapshot’ of information which can be one-off or routine, dependent on methodology – rarely are they real-time. The surveys tend to quantify responses by applying a Likert (1-5) scale, or restrict answers to multiple choices. Example questions include: Are your main problems being addressed?2 or Do relief workers treat you with respect?3 While an operation may know that a response was ‘negative’ – only scoring a low 2 – they are not sure why, for who, or how to remedy it. In many perception surveys, they are not linked to a specific service on intervention. Therefore, findings can remain abstract and ‘unactionable’ for many practitioners.
Unpacking the complex problems faced by refugees certainly involves a fuller picture than a one-time snapshot can provide. UNHCR’s community-based protection approach aims to jointly determine challenges and how they can be met, by identifying local capacities and developing protection solutions with communities. Through individual dialogues, participatory assessment approaches and community dialogues, we collect a wealth of qualitative information. Our focus should be to strengthen how this data drives decision making, response design, and course correction. Perception surveys may be complementary to these, but it’s important to first leverage and utilize the data we do have.
Why are perception surveys getting so much traction then?
Because we have an ‘actionable’ information gap, we need to enhance our data-informed decision-making.
Our community-based protection approaches aim to be individualized, inclusive and to work with communities to identify challenges, capacities, and solutions. As an organisation, we engage with thousands of refugees on a daily basis, but there is much more that we can do to capitalise on these interactions. With more investment in data collection, analysis and information sharing we can be more effective as an organisation, we can better adapt our activities and tailor our services. By understanding the value of the information shared with us through this engagement – and by acting on it – we can close the feedback loop. Examples include data collection methodologies for community interactions (i.e.voice recording, online data collection) but also data analysis: including natural language processing or the application of the tetraclass model4 to analyse refugee’s satisfaction of specific services. We also need enhanced data visualisation techniques, and tools to transparently communicate decision-making and hold decision-makers to account.
In Bidibidi, Uganda, UNHCR is supporting the American Refugee Council to test the ‘Kuja Kuja’ project which engages outreach volunteers to digitally collect feedback and recommendations on protection services in Zone 5. The methodology builds on customer service experience and applies machine learning to analyse data which is then visualised and mapped. In Lebanon, the UNHCR team are trying to develop a solution to manage and analyse information shared through a Facebook group with over 160,000 members. These are just two examples – if you have more, we’d love to hear from you.
—–
1: For example: Perception Surveys were used in Ebola Response (2014-2015), Nepal (2015) and Uganda (2017).
2: Ground Truth, Nepal 2015
3: Ground Truth, Antigua 2018
4: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4139752/
We’re always looking for great stories, ideas, and opinions on innovations that are led by or create impact for refugees. If you have one to share with us send us an email at [email protected]
If you’d like to repost this article on your website, please see our reposting policy.